A recently discovered CVE in containerd allows attackers who can run a custom image in a cluster to break out to the underlying node and, in some cases, escalate privileges to cluster-admin level.
This CVE is interesting for several reasons. First, the vulnerability shows up in the container image, not in the Kubernetes manifests, so it’s harder to spot with infrastructure-as-code (IaC) or admission control solutions. Second, it uses standard container image commands, so it’s not easily detected by malware scanning agents.
In addition to talking about the details of this CVE and possible mitigations, it’s important to note the role of the Kubernetes security community in highlighting these issues. After noticing the information from the researcher who found the issue, several members of the community worked to reproduce it and understand the likely impacts and possible mitigations. These discussions can generally be found on the #Kubernetes-Security and #SIG-Security channels on Kubernetes Slack.
Technical details
This issue was found and reported in November 2021, and a patch for containerd was released on March 2, 2022. The issue lies in containerd’s container runtime interface (CRI) plugin, a component that generally interfaces between Kubernetes clusters and containerd. The issue was that by specifying a volume in a container image that references paths on the host (e.g., /../../../../../../../../var/lib/kubelet/pki/), arbitrary files from the host would be copied into the running container as it launches.
This effectively allows an attacker who can create a custom image to pull things such as credentials from the underlying host, which makes it possible to escalate privileges in that cluster node—and likely to the whole cluster.
Here’s a basic example of a container specification that exploits this issue:
|
FROM registry.access.redhat.com/ubi8/ubi |
Building this image and then deploying it to a cluster will copy the kubelet’s private key into the running container, as shown in this video:
In this example, these credentials can then be used to carry out actions via the Kubernetes API with the rights of the kubelet itself.
Additionally, there’s a GitHub repository with the files for this example and a more advanced option that can be used to show privilege escalation in a cluster.
This is just one example of how this attack could be used, by changing the volume statement to point to other directories on the host that the attacker is interested in.
Because this vulnerability is in the CRI component of containerd, it’s not likely to be exploitable in clusters where Docker is used as the container runtime, because it uses Dockershim instead of containerd’s CRI.
Detections and mitigations
Depending on how this issue is used, it could be moderately difficult to detect. The copying process will be done by containerd itself, not by a user running in the container, so detections will rely more on monitoring sensitive locations on the cluster node filesystem. For example, in the basic example above, there’s access to the kubelet’s private key file, which should be monitored carefully.
As ever with mitigation, the correct approach is to ensure that the patched version is deployed to all cluster nodes. Another key security control for this issue is to ensure that only trusted images are deployed to the cluster.
Where those mitigations aren’t practical, or as a temporary fix, you can use admission control to look at images being deployed. This approach would rely on the admission control software in use being able to reference external data as part of its validation process, which is possible with Kyverno and OPA. Specifically, Kyverno has a policy that can be used to block any images that have volume statements. If this functionality is required, a safe option would be to block it entirely.
Conclusion
This is another CVE in the long line of container breakout vulnerabilities that have come out in 2022. These vulnerabilities have affected several layers of common Kubernetes stacks. In each case, the vectors and possible mitigations are somewhat different, but there are some common themes that cluster operators should be looking at.
First, these vulnerabilities underscore the importance of regular patch cycles on all cluster nodes. This should be a routine that is practiced often to minimize the risks and ensure that patching can be done quickly and safely when needed.
Another key point is that Kubernetes environments require a defense-in-depth strategy. A single preventative control can’t keep a cluster safe, so a layered set of both preventative and defensive controls is absolutely required to account for potential weaknesses.