When you’re operating Kubernetes clusters, an important area of focus is in ensuring your authorization model is correct and provides users with the least privileges needed for them to carry out their roles. As such, blanket cluster-admin privileges should never be used and in particular the in-built system:masters group should be avoided.
What is system:masters?
Most Kubernetes distributions include a number of default cluster roles and roles which provide general sets of permissions for users of the system, and also permissions used by the controllers and other control plane components.
However, in addition to these defaults, there are some built-in groups which are used by Kubernetes. The first two are relatively straightforward. system:unauthenticated is used to provide anonymous rights to the Kubernetes API and system:authenticated is used to provide rights to all authenticated users.
This brings us to the third in-built group, system:masters, which is handled a bit differently. System:masters is a group which is hardcoded into the Kubernetes API server source code as having unrestricted rights to the Kubernetes API server.
Why should system:masters be avoided?
Any user who is a member of this group has full cluster-admin rights to the cluster. Even if every cluster role and role is deleted from the cluster, users who are members of this group retain full access to the cluster.
We can demonstrate this access with a kind test cluster, as the default user for these clusters is a member of system:masters (N.B. Do not do this to anything other than a test cluster you don’t mind breaking!). First, we can delete all the cluster roles with this command:
kubectl delete clusterroles.rbac.authorization.k8s.io --all
Then we can delete all the roles in the kube-system namespace for good measure using this command:
kubectl -n kube-system delete roles --all
kubectl auth can-i --list| Resources | Non-Resource URLs | Resource Names | Verbs |
| *.* | [] | [] | [*] |
| [*] | [] | [*] |
Also, it’s important to note that this is the same even if you’re using webhook authorization instead of Kubernetes in-built RBAC system, as requests for access by members of this group will not be sent to the webhook for evaluation.
Membership of system:masters is particularly dangerous when combined with Kubernetes client certificate authentication model, as Kubernetes does not currently provide any mechanism for client certificates to be revoked. So this means that a client certificate based credential issued with a group name of system:masters has full access to the Kubernetes API which cannot be easily removed. If an attacker gains access to this credential they will have long lived cluster admin level access to the environment.
What should you do about system:masters?
The first step for actions you should take in your clusters is to avoid making any users a member of this group. In the case where you do need to grant someone cluster-admin rights (and this should be used sparingly), you can create a clusterrolebinding to the in-built cluster-admin clusterrole. This will have the same effect as using system:masters, but would allow those rights to be removed if necessary, by removing their user from the clusterrolebinding.
The second area to look at is, how the Kubernetes distributions you use make use of system:masters. In a number of distributions, a bootstrap user will be created as part of cluster setup, which will be made a member of this group.
For example, in kubeadm or distributions based on it, a file called admin.conf will be created as part of installation. This is a client certificate based credential which belongs to the system:masters group. It is very important that access to that file is carefully controlled, as it is effectively a generic backdoor user with a default lifetime of one year. As noted in the kubeadm documentation, this file is intended for break glass use only and should not be used for general cluster administration.
With the large number of Kubernetes distributions available, creating a full list of places where system:masters is used would be challenging. However, other locations I’ve noticed it in use include :
–RKE (as of v1.2.5) and in Azure AKS.
Conclusion
Kubernetes authentication and authorization models have a number of edge cases which are important to understand when designing and operating your clusters. It’s important to carefully manage high privileged credentials, such as those created as members of the system:masters group to reduce the risk of unauthorized access to your clusters.