A new vulnerability was found in containerd, located in the container image-pulling process. The new CVE includes manipulation of the image manifest, allowing attackers to craft an image that can leak the host’s registry or cloud credentials when pulled from a registry. This leak occurs even before the image is running any code on your server. The CVE was found by Brad Geesaman who presented it in his “ContainerDrip” write-up, this blog post is based on that and summarises it, you can find the original write-up in the links below.
Containerd is an industry-standard container runtime that is available as a daemon for Linux and Windows. It can manage the complete container lifecycle of its host system: image transfer and storage, container execution and supervision, low-level storage, and network attachments, etc.
A container image is a combination of a manifest file and some individual layer files. The manifest file contains a JSON-represented description of a named/tagged image, layers, sizes, and digest. The manifest can contain a ‘foreign layer’ which is pulled from a remote registry. When using containerd, if the remote registry responds with an HTTP 401 status code, along with specific HTTP headers, the host will send an authentication token that can be stolen. Adversaries can exploit this vulnerability and build dedicated container images designed to leak the host’s token, then use it to take over a cloud project.
Taking a deeper dive into this vulnerability, the manifest is Docker Image Manifest Version 2, Schema 2, which is the standard format for containerd. The manifest describes the file system layers and how to run them, after the container runtime verifies the layers, it can continue to download them and run the image. Sometimes the manifest will be a manifest list that represents support for multiple platform/architecture combinations, nowadays, most images on ‘DockerHub’ are manifest lists.
Looking at the Docker Image Manifest Version 2, Schema 2 documentation, here is an example of a manifest:
This manifest JSON provides configuration and a set of layers for an image container, and its last layer is a “foreign layer” with a URL field. You can find the complete manifest specification here: Image Manifest V2, Schema 2 Specification.
If we read deeper into the layer description, we see that the manifest supports an optional field for an external URL from which content may be fetched, and it can be any registry or domain.
This feature holds the vulnerable part of the image-pulling process in containerd, so to initiate containerd to pull a layer from an external registry, you have to edit its image manifest. Recalling that any image is basically a manifest, this means anyone can upload an image to trigger this feature.
Now, everyone pulling this image will be redirected to a remote registry to download that layer. Next, knowing that clients pulling images from a remote registry might need to be authenticated enables us to ask for an authentication OAuth token, according to the authentication process. As it turns out, with the right response from the remote external registry, we can make that happen by just responding with HTTP 401 and the correct Realm when asked for authentication, as seen in this ‘nginx’ config:
To simulate this attack, we need to push our image to a GCR registry, then pull it from a GKE cluster running vulnerable containerd (at the time the vulnerability was found). Once executed, looking at our ‘nginx’ weblog, which is simulating a remote external registry from which the containerd was instructed to download a foreign layer, we can see a ‘Basic Auth’ header. If we decode it (‘base64’), it turns out to be an authentication token. In our simulation, it is a GCP Service Account OAuth token.
In this instance, the token has ‘project editor’ permissions over cloud resources but is scoped down so it can’t edit permissions. Given another configuration, this service account could change permissions and lead to account take-over. Of course, GKE is just one of many examples where this issue can occur.
Conclusion
It’s always good practice to periodically verify that you’re using the latest version of software, as is the case here. To remediate against this CVE you should ensure your systems are running the latest version of their container runtimes. This vulnerability was patched in containerd 1.2.14, and containerd 1.3.x was also tested and validated as not vulnerable.