In our many conversations with customers, Docker secrets management has come up as a particularly thorny issue that seemed to lack an elegant, cross-platform solution for container environments. Not a new issue in the enterprise space, especially as pertains to large-scale DevOps environments, the challenges of managing secrets become amplified in container environments.
Verboten Shortcuts
All too often, users take non-secure “shortcuts” to using secrets with containers that may make their lives easier, but put their organization at risk.
One such shortcut is placing the secret inside the container image. From a security standpoint, it’s akin to having your bank login credentials on a post-it note attached to the bottom of your keyboard. It’s a no no. If the secret is embedded into the container image, it is accessible to anyone who handles the image throughout the pipeline, and worse, it will be there whenever a container is instantiated from the image – in environments that are impossible to predict and control.
Another, slightly better shortcut, is using Docker environment variables to provide the secret to the container upon running it. This is a better approach because the secret only “lives” in the specific runtime environment, but it still leaves the secret exposed to users with access to the host, since environment variables can be viewed in docker inspect <container id> output and inside /proc/<pid>/environ. At Aqua we plugged this particular security gap back in June 2016 when we introduced environment variable encryption. However, using environment variables to pass secrets is still not a scalable, effective way to manage secrets, and it still leaves traces of the unencrypted secret values in the orchestration tools.
Containing Secrets in Containers
There are already proven ways of Docker secrets management in enterprise environments, for example using HashiCorp Vault. Obviously the central repository for secrets should be encrypted and protected with user access controls. But when it comes to containers, there are 4 unique challenges that must also be addressed:
- Managing which containers have access to which secrets:
Secrets should only be accessible to the containers that actually need them – this has to be done at the container level. You need a mechanism that allows to map secrets to relevant containers, and then indicates which container is using which secret. - When do you actually retrieve the secret from the vault?
As noted above, putting the secret in the container image exposes it to many more users and processes than is needed, and puts it in jeopardy of being misused. The right time to make the connection between the container and the secret it needs is when the container runs, and not before. - How is the secret stored in the container once retrieved?
The container must have access to the secret when it’s running, but the secret should not be stored on disk or exposed at the host level. The secret should be retrieved from memory, and in a way that’s only accessible to the relevant container. Once the container is stopped, the secret disappears. - How to handle secret rotations?
Secrets can change over time. A database password, for example, might change after some time to reduce risk of exposure. In this case you would need a system that is smart enough to update all containers that are currently using that secret with the new value.
Managing Secrets with Aqua
With version 2.0 of our container security platform, we introduced a complete Docker secrets management solution for containers, with integration to HashiCorp Vault.
Our solution allows you to centrally manage secrets and container access to secrets, and view which containers are using those secrets in real time. It injects secrets into the container at runtime, and ensures that the secrets is stored in memory and only accessible to the designated container.
Scanning for Secrets in Images
The image vulnerability scanner uncovers secrets that may be stored in images – secrets such as AWS tokens, SSH keys, or passwords in cleartext. Now, whenever an image scan runs (usually as part of your CI/CD build), you’ll be notified if your images contain potential secrets.
Read about managing and securing Kubernetes secrets