One of the challenges of managing containerized environments is how to store sensitive information that’s needed for the operation of the applications running in those environments. Kubernetes provides a built-in secrets object type, but a common comment about them is that, from a technical standpoint, they’re just the same as other object types like Config Maps. So why should we use secrets specifically?
This is something I’ve seen mentioned by cluster operators and also distribution providers, who sometimes make use of config maps for storing sensitive information.
When looking at how secrets and config maps work at a technical level there are a lot of similarities. Both object types are accessed via the Kubernetes API, access to them is controlled via RBAC or other authorization systems, and the data is ultimately stored in etcd.
So why should we use a dedicated secret type and not just place all our configuration information in more general-purpose objects? I’d argue it’s a matter of showing the intent of the data, so that it’s clear to everyone that this information should be handled carefully, and there are a couple of areas where this becomes evident.
Controlling access to information with RBAC
A key difference between secret data and general application configuration data, is in how safe it is to allow read access to that information. In general, read access to application configuration data should be safe enough to provide to users of the cluster and internal systems that monitor or manage cluster application state.
On the other hand, secret data should not be shared widely as read access to it will often allow for privilege escalation attacks, and generally should be confined on a “need to know” basis. So the RBAC requirements are very different.
A good example of where placing secrets in config maps can cause problems is where the generic Kubernetes cluster roles are used. Kubernetes provides a cluster role called “view” which users may use as a way to provide read-only access to cluster resources. The view cluster role is, in my experience, relatively commonly used as a way to provide read-only access to cluster resources to developers and other users who might need to interact with the cluster.
Looking at the rights provided we can see it gives read access to config maps (amongst others) but no access to secrets.
rules:
- apiGroups:
- ""
resources:
- configmaps
- endpoints
- persistentvolumeclaims
- persistentvolumeclaims/status
- pods
- replicationcontrollers
- replicationcontrollers/scale
- serviceaccounts
- services
- services/status
verbs:
- get
- list
- watch
So this is the first good reason to keep secret information in secrets, whether you’re a cluster distribution developer, add-on developer or cluster user. It avoids the risk of inadvertent access to your secret information, and your intent that the information should be handled carefully, is clear.
Secrets handling by the platform
A second good reason to ensure that you use secrets for sensitive information is around communicating your intent to the underlying platform. It’s a general goal for most systems to avoid inadvertently placing sensitive information in diagnostic logs or other forms out output that could be seen by a wider audience than they should be. Especially when many companies use external SaaS based logging and monitoring systems, making sure your secrets don’t end up exposed is important.
In-line with this approach, Kubernetes tries to avoid logging secrets and credentials in general. There are multiple KEPs on avoiding logging secrets, for example using static analysis to defend against logging secrets and also looking at log sanitization to avoid logging sensitive information.
The important part to note here, is that Kubernetes can only take steps to mitigate these risks *if* it knows the information is intended to be sensitive, and this is where intent is important again.
If config maps, or other object types, are used to store sensitive information, Kubernetes has no way to understand the sensitivity of the data and to make sure it’s handled appropriately. So if config maps are used, the data may end up in logs or crash dumps.
Conclusion
Maintaining the good security around cluster secrets is an important part of overall cluster security. If you’re a cluster operator, there’s a couple of key points to note from this:
- One option is to use an external system for protecting Kubernetes secrets, as they will have more functionality for handling them such as integration to existing enterprise secrets stores.
- If you’re not using an external secrets system, make sure all your sensitive information uses Kubernetes secret objects and not any other object type.
- Review how your Kubernetes distribution and any supporting software you install in the cluster handle sensitive information.
- If you are using Kubernetes secrets, ensure that you enable Kubernetes optional secrets encryption. This is not the default, and is risky in case your etcd database gets compromised or exposed. There is good documentation on how to encrypt secret data on the Kubernetes site.