Kubernetes Metadata and Manifesto

Kubernetes Metadata and Manifesto

We recently released the first prototype of Manifesto, a tool for storing metadata alongside a container image. Metadata can hold information about software that can change even if the software itself doesn’t, and we can use this information to create powerful automation tools.

Here’s a simple example use case that we’ll demonstrate in this article. Suppose that a fictional organization - we’ll call it CompanyX - has a number of different components owned by different teams. If one component starts behaving badly, for example generating error logs, the ops team needs to know who to contact.

CompanyX can store the contact information in the form of container image metadata, query it, and keep it up-to-date using Manifesto.

Storing Contact Information as Metadata

Alex Developer works at Company X as the lead on Component A. She adds her contact details, along with a webhook to the team’s Slack channel, into a text file.

> cat contact.file
email: a.developer@companyx.com
phone: 0123 456 789
slackwebhook: hooks.slack.com/services/abc/def/123

This is associated with the image for Component A using manifesto:

> ./manifesto put companyx/componenta contact contact.file
Storing metadata 'contact' for image 'companyx/componenta:latest'
Username: alexd
Metadata 'contact' for image 'companyx/componenta:latest' stored at sha256:6a39686306556ab512ec1e0842dc5134f8e5e879080e9cc9e21fecf78d95106e

Adding 'contact' metadata to manifesto for

If Alex moves to a different team, the contact file can be updated with the new team lead’s email and phone number, and stored against the container image with another manifesto put (without having to re-publish the container image for Component A).

This step can be automated as part of a CI/CD pipeline that updates runs manifesto put whenever contact files are changed.

We now have the metadata stored alongside the container image. The next step is to apply it to Kubernetes resources.

Adding Metadata to a Kubernetes Resource

Kubernetes resources can have metadata associated with them through Labels and Annotations. Both are key-value pairs, but they are used for different purposes:

  • Labels are typically used to indicate some attribute of the resource that might be used to identify it, or to select is as a member of some logical grouping of resources - for example, mark whether it’s part of a production, staging or development.
  • Annotations are more free-form and are intended for arbitrary metadata.

For the contact information in our example, Annotations are more appropriate.

The kubectl annotate command takes one or more KEY=VALUE parameters to define the annotations being added. Let’s apply the contact information to the deployment for Component A with the latest contact data for the container.

> kubectl annotate deployment componenta contact="$(./manifesto get companyx/componenta contact)"
deployment "componenta" annotated

We can take this a step further by first querying what the container image is for the deployment.

> IMAGE=$(kubectl get deployment componenta -o=custom-columns=:.spec.template.spec.containers[0].image --no-headers)

> kubectl annotate deployment componenta contact="$(./manifesto get $IMAGE contact)"
deployment "componenta" annotated

In this way we can easily add contact information (or any other metadata) in the form of annotations, to all the running software in our cluster.  If the contact information changes, this can be updated simply by overwriting the annotation.

Querying the Annotation

When something goes wrong and we need to find out who to contact about Component A, we can simply read the information out of the annotation.

$ kubectl get deployment componenta -o=custom-columns=:.metadata.annotations.contact --no-headers
email: alex.developer@companyx.com
phone: "0123 456 789"
slackwebhook: hooks.slack.com/services/abc/def/123

With this information we know who to contact about this component. This could easily be further automated, for example to send a message to the team’s Slack channel.

Other Use Cases for Metadata

There are many more scenarios where container metadata could be applied. Puppet have shown how to use Lumogon and Manifesto to store and query information about container image contents, and we have a list of use cases on the Manifesto Readme. Thought of another example? We welcome pull requests!

Picture of Liz Rice

Liz Rice

Liz Rice is the VP of Open Source Engineering at cloud native security specialists Aqua Security, where she works on container-related open source projects including kube-bench and kube-hunter. She chairs the CNCF’s Technical Oversight Committee, and in 2018 was Co-Chair of the CNCF’s KubeCon + CloudNativeCon events in Copenhagen, Shanghai and Seattle. She has a wealth of software development, team, and product management experience from working on network protocols and distributed systems, and in digital technology sectors such as VOD, music, and VoIP. When not writing code, or talking about it, Liz loves riding bikes in places with better weather than her native London, and competing in virtual races on Zwift.

Kubernetes Security, Image Vulnerability Scanning