Don't Leave Your Keys Exposed: Lessons from IBM Privilege Escalation Flaw

Would you ever give your keys to a stranger? That’s exactly what someone at IBM did: they left private keys to the Docker host environment in IBM’s Data Science Experience service accessible to the outside world. Wayne Chang, security consultant who found this, explains in his original report:

The container's access to the Docker Swarm host API was secured via TLS, however, the keys to access said API were readily available *in the container*. As is commonly known, access to the Docker host API may be trivially leveraged into root access on the host machine.

On the flip side, IBM should be commended for fixing the vulnerability in just 2 weeks, which is quick compared to industry averages.

What Happened?

To exploit the API keys, all one needed was a Web browser connected to the internet.

Anyone could sign up for a free trial of the Data Science Experience service, and then take the following steps to exploit, according to Chang:

• Enter RStudio Web Environment, and focus the cursor on the command line. That practically gives access to command line running within the underlying container.
• Download the Docker client into underlying container. To achieve that, Chang simply downloaded and extracted Docker:
  system("wget https://test.docker.com/builds/Linux/x86_64/docker-1.13.1-rc1.tgz")
  system("tar -xvzf docker*.tgz")
• Now, and this is the punch line: The private key and the certificate were mistakenly left inside the container image. So the final step was to use the Docker client and the keys left behind to connect to Docker daemon on the host, and run an arbitrary container:
  system("DOCKER_API_VERSION=1.22 ./docker/docker -H 172.17.0.1 \
  --tlscacert /certs/ca.pem --tlscert /certs/cert.pem \
  --tlskey /certs/key.pem \
  run -v /:/host debian cat /host/etc/shadow")
  Note that mounting the host’s root directory (‘/’) gives you root-equivalent access.

In summary, the privilege elevation was very simple: Install Docker inside the container (just a Docker client, not the daemon) and run it against the host's Docker daemon while authenticating with the certificates inside the container.

Use your imagination and Chang’s blog to think of the potential implications of such a mistake. Besides obvious reputation and breach of customer trust in the service’s ability to maintain its data securely, the damage could result in the need to completely reimage all potentially affected hosts, as an attacker could have already installed a rootkit on them.

The beauty of container orchestration is its ability to very rapidly scale up and deploy numerous nodes running containers. Unfortunately this very mechanism might also propagate a damaged container equally fast.

So What Can We Conclude?

It is important to note that this is not a vulnerability in Docker itself. On the contrary, Docker Swarm’s TLS is a security feature which, when used correctly, greatly enhances the security of a containerized application. The problem was, of course, that it was not used correctly. As in the Vine incident we wrote about a few months ago, someone was sloppy with applying basic security practices. You can give a person a locked door and a key, but if they leave the key under the doormat they might as well leave the door unlocked.

Security advisories must sometimes resort to clichés, and this one is no exception: Human errors are natural and inevitable, so consider deploying controls that minimize the risk of such mistakes going unnoticed.

Recommended Security Measures

There are several measures anyone can take that would secure their environment against these types of errors:

• Lock down ‘docker pull’ command on the host. Normally there's no reason why it should be enabled - it's a command used to pull images from DockerHub and should only be available in dev environments.
• Protect your image registry with user authentication
• Protect the host with tools like SELinux or AppArmor to harden access control and prevent containers from easily gaining root access.

For Aqua customers, such an incident would be prevented at several points in the process:

• The Aqua image scanner looks for sensitive data including private keys and certificates. It would have alerted on the presence of the TLS keys.
• Aqua’s runtime controls would have prevented the container from having new executables placed in it (so the user would not have been able to install Docker).
• Aqua’s network nano-segmentation controls would have prevented the container from gaining privileges on the underlying host through its IP address.