Following on from our previous post on the risks of privilege escalation in Kubernetes via the node/proxy resource, we’re going to take a look at how users who have rights to the certificate signing request (CSR) API in Kubernetes might be able to use them to escalate their privileges in a cluster. In addition to highlighting a potential security risk to be aware of, we’ll also explain how this Kubernetes feature works.
Introducing the CSR API
Kubernetes uses client certificate authentication in several locations, including the communication between the kubelet and the API server, so it needs some way to issue and sign new certificates. This is where the CSR API comes in.
For client certificate authentication to be used, the generated certificates must be signed by a trusted certificate authority. This is similar to what happens if you look at the certificates on websites you visit, where they will have been signed by one of the certificate authorities embedded into your browser. If they haven’t been signed, you’ll get a warning about them being untrusted. The Kubernetes CSR API provides a mechanism for that signing to take place without the need of direct access to the certificate authority key files.
Essentially, the API allows users who have the appropriate role-based access control (RBAC) permissions to send a CSR to Kubernetes, where it can be approved by another user who has the appropriate rights.
The API has several signers, which can sign certificates for different purposes. Some of these like kubernetes.io/kube-apiserver-client-kubelet provide certificates used by internal components of Kubernetes. These signers tend to be quite restricted in what can be in the requests that they sign, so they’re not that interesting from a security perspective.
The signer that is specifically of interest here is kubernetes.io/kube-apiserver-client. It can sign certificates that are used for authentication to the Kubernetes API server and has very few restrictions on what can be included.
Privilege escalation with the CSR API
So how would a user who only has rights to create CSRs and then approve certificates escalate their rights in the cluster? The first thought that would likely occur to an attacker is to create a new certificate in the system:masters group. This is a hard-coded cluster-admin group that can’t have its rights revoked, so it’s a great target for an attacker.
However, by default in Kubernetes, it shouldn’t be possible to issue a certificate with this group. That’s because there’s an admission controller in the default set called CertificateSubjectRestriction, the sole purpose of which is to block this use case.
With that blocked, what are the other options an attacker can take? One good one is to look at the usernames already defined in the cluster to see which have a high level of privileges, and then mint a new certificate that matches one of them. Kubernetes has no mechanism to say which specific credentials are valid for a given user. So there’s no restriction on creating a new client certificate for a system account and then using that to authenticate to the cluster, because Kubernetes will then give you the rights of that system user.
Exactly which users and groups will exist in a cluster depends on the distribution in use. However, looking at the setup in a kubeadm cluster, we can see some likely targets. Although it’s not possible to get a definitive list of users in a cluster, we can see ones that already have RBAC permissions defined, which is what’s needed for this purpose.
Looking down that list, system:kube-controller-manager springs out as a good potential target. This is the account used by the Kubernetes controller manager component, which is core to the cluster’s operation, so it likely has decent privileges. Looking at the permissions provided to that user, we can see the rights provided to secret resources:
Having rights to get secrets at a cluster level is pretty much game over for that environment, so it’s clear that a user with those rights would be handy to an attacker. If the attacker wanted to get all the way to the cluster-admin level, one way would be to use the rights to get the service account token secret for the clusterrole-aggregation-controller. This particular service account has the “escalate” right, which allows it to effectively increase its own permissions to cluster-admin on demand.
Detecting and mitigating CSR API privilege escalation
So, knowing that privilege escalation via the CSR API is possible, what are the best ways to detect and mitigate it?
- Kubernetes auditing should be enabled on all clusters, and all events related to the CSR API should be monitored carefully. While there’s a level of use by system services, it should be possible to notice unexpected certificates being created. An important point to note is that if an attacker can create a client certificate with the same name as a system account, Kubernetes auditing may not be very helpful because it won’t log the source of authentication, just the user name.
- RBAC privileges to this API should be restricted, including privileges for users and administrators of the cluster and permissions given to any software (e.g., operators) that’s installed into the cluster. If specific access is requested to CSR objects in the cluster, or general wild-card access that would include access to CSR objects, the request should be reviewed carefully to reduce the risk that those rights could be misused.
- Another layer of defense is to use admission control. For example, if access to the CSR API is needed, admission control could be used to stop request to use the io/kube-apiserver-client signer or to block the issuing of certificates for accounts like system:kube-controller-manager.
Conclusion
This is another example of the complexity of Kubernetes RBAC and the risks of excessive privilege grants in Kubernetes clusters. It’s important to ensure that when you’re using Kubernetes as part of your environment, rights are carefully controlled and additional detective and preventative controls are deployed to manage any risks from unauthorized access to Kubernetes APIs.