Attack of the mutant tags! Or why tag mutability is a real security threat

Tag mutability can introduce multiple functional and security issues. In container land, tags are a volatile reference to a concrete image version in a specific point in time. Tags can change unexpectedly, and at any moment. In this article, we’ll learn how we can prevent them.

Tag mutability causes, among other things, the Time-of-check vs. Time-of-use (TOCTOU) issue. The image that is verified during the CI/CD pipeline or the Kubernetes admission phase differs from the image that is deployed in the cluster, bypassing image scanning security checks.

For example, a developer with direct access to the registry can exploit this fact to mutate a tag after the image has been scanned, and get a malicious version of your microservice running inside the production cluster, compromising your company security by leaking confidential information or providing a backdoor.

Using immutable tags would prevent these problems, but tag mutability is very convenient in many scenarios, and immutable tags are not widely supported in the registries.

In this article, we’ll discuss the implications of tag mutability, benefits and drawbacks, as well as some practices and techniques to prevent unexpected problems. We’ll also cover how to use Sysdig image scanner and admission controller tools to secure your Kubernetes deployments.

Exploring tag mutability

Basics of registries, images and tags

Before explaining what is tag mutability, let’s define some critical concepts:

• Registry is the service that we use to store and retrieve (push/pull) our container images. It can be a public SaaS service like docker.io, quay.io, GCR, ECR, etc., or a private registry hosted On-Prem.
• Images inside a registry are grouped into repositories, and repositories can have additional sub-groupings like organizations, projects, etc.
• A repository contains multiple tags.
• When we create a container, we pull an image from the registry, selecting a specific tag from a repository. This tag corresponds to an image and launches the isolated execution of a process in the file system defined by that image.

So we run a container using a specific image with a command like the following:

docker run my-registry.com/some-organization/a-repository:tag1

But what exactly is under the hood of an image? An image is just a JSON document, called the image manifest that describes:

• The image configuration (startup, environment and other parameters).
• A set of filesystem layers that, together, compose the root filesystem for containers using this image.

Manifests, image layers and image configurations are stored as binary blobs inside the registry. To address them, we use their SHA256 hash, for example, “20feefdbe22e555a5922072161f0bf42fdee25f38c026997c0f4c025e3edbe00”, although in this article we’ll only use the first digits for clarity: 20feefdb…

These hash digests aren’t handy, so we can define tags to refer to them (to the manifest ones at least). In the above scenario, tag1 is a link to the manifest file with the digest 1ab50232… If we want to refer to that manifest, for example, to download that image, we can use myregistry/myimage:tag1.

The manifest contains a reference to the image configuration, and the digests of the layers composing the image filesystem (e74c2290…, 71bb4d21…, etc.).

What is tag mutability

As digests are created based on an object content, a given digest will always identify the same object. They are immutable.

This is great to avoid duplicate content. For example, two layers with identical content on two different images, those layers will have the same digest. We can reuse that layer, saving space and bandwidth.

To avoid the limitations of digests, we use tags, which are just a pointer to a manifest file. And, as a pointer, they can change over time.

Mutant tags (or mutable tags) are tags that reference one specific image via its manifest, but it can reference a different image at a separate point in time.

Inmutable tags are those tied to a specific image and cannot be changed afterwards.

Creating a mutant tag

Creating a mutant tag is a pretty common and straightforward process:

Note that the old manifests are still available in the registry, so you can retrieve them using the manifest digest instead of the image tag. We could use:

• myimage@sha256:1ab50232… to pull the first image.
• myimage@sha256:17b345ca… to pull the second one.
• and either myimage:tag1 or myimage@sha256:612bc919… to pull the third one.

Also, multiple tags can point to the same manifest. In the example, you can see that myimage:tag1 and myimage:othertag refer to the same image at this point, although they could mutate!

Tag mutability use cases

Obviously, mutable / mutant tags are useful, as they have multiple use cases. For example:

Use “latest” (the default tag if not specified) to always point to the latest version of an image, or image variants like “alpine” or “slim”, that always point to the latest corresponding versions of those variants.

It’s also quite common to use mutant tags to track versions in different environments, like “dev”, “prod”, “qa” and more. These tags are updated when a new version is deployed to one of these environments.

For versioning simplicity, tags like :1 or :2.0 can be aliases to a minor or patch version. Using these tags, users can rely on pulling the latest major version or an image, with no unexpected behavioural or configuration changes, but making sure they are getting the latest minor version with included bug and security fixes.

Deployment issues with tag mutability

Some typical problems that developers or operations teams face relate to tag mutation and container deployments. Any place where you deploy a container using an image tag, you are performing a non-deterministic deployment, as there is no guarantee that the tag corresponds to the expected image. This applies to a simple docker run, or for orchestration tools like docker-compose, Kubernetes, etc.

Let’s see an example:

• Peter, a developer, executes a container in his laptop using myimage:latest. He pulls version 1.1 of the application and runs it.
• A couple of hours later, Laura pulls and runs the same myimage:latest in his computer. But is it the same image? No, the image maintainer has published version 1.2 in between, and latest is now referencing version 1.2.
• Next week, they deploy their application in the production cluster, but myimage:latest is now pointing to version 2.0, which introduces breaking changes in the configuration file format.

As a result, the application crashes in production.

For Kubernetes, you can end up running different images for containers that should be identical depending on the scheduling and volatility of pods, or the status of the host cache.

The following screencast reproduces the issue by delaying the schedule of a pod in one of the nodes. We use cordon to disable scheduling in one of the nodes, mutate the tag and then uncordon the node, but the same problem appears if a node is later added to the cluster, or a pod is expelled and rescheduled in a different node.

Repeatable deployments

In order to get deterministic and repeatable deployments, Peter and Laura can use digests instead of tags. By using myimage@sha256:<digestValue> instead of myimage:tag, they can ensure that they’ll deploy the same manifest in their computers and in production, avoiding this kind of unexpected version changes.

For example: use myimage@sha256:1ab50232… instead of myimage@sha256:latest.

One note on Kubernetes behaviour: Kubernetes resolves the tags to the image digest after the pod is scheduled and the image pulled. Had it been resolved before creating the pod resource, it would have saved us from some issues. Also, note that Kubernetes has different ImagePull policies, so Always will always download from the registry instead of using cached images.

The TOCTOU problem

These issues so far are related to bad timing, unexpected updates, etc., but not to malicious activity or security threats. TOCTOU stands for Time-of-check Time-of-use, so a TOCTOU bug can happen if checks are done in a different moment than usage, compromising the security of your system.

Let’s analyze a case where an attacker could exploit a TOCTOU bug to run malicious software in your cluster, even when using a Kubernetes Admission Controller that scans every image before deploying into the cluster. The Admission Controller should detect bad practices or vulnerabilities in the containers, so you can forbid images that don’t comply with your security policies.

Here’s an example workflow of an attacker exploiting a bug like this:

1. A pod creation/update request is sent to Kubernetes API.
2. The admission controller intercepts the request and triggers an image scan.
3. The request contains myimage:latest as the image name, so the image scanner requests that image to the registry…
4. … and it pulls the manifest and the corresponding layers, composing the image to scan it.
5. The image scanner processes the image, generates a report and sends it back to the admission controller. The admission controller then evaluates the scan report and admits the pod, as there are no vulnerabilities.
6. At this moment, after the scan finishes, an attacker pushes a malicious image to the registry using the same myimage:latest tag.
7. Kubernetes schedules the pod and the container runtime pulls the image from the registry in the node.
8. The pulled image differs from the scanned image, causing a potentially malicious (or at least unscanned) image to be deployed in the cluster.

How easy is it to push the malicious image between the scan and the pod schedule?

Since scanning images can take anywhere from a few seconds to a few minutes, depending on the image size, and scheduling can also take a while, there is plenty of time to perform the push in between.

You can prepare the malicious image in advance by building it, pushing it to a repository and retagging locally to myimage:latest:

$ docker build -t myregistry.com/maliciousimage .
Uploading context 18.829 MB
Step 1/2 : FROM busybox
 ---> 769b9341d937
…
Successfully built 99cc1ad10469
$ docker push myregistry.com/maliciousimage
The push refers to repository [myregistry.com/maliciousimage]
e9f56d359a24: Pushed
…
latest: digest: sha256:8360f9950adc9... size: 735
$ docker tag myregistry.com/maliciousimage myimage:latest

And right after the scan triggers the pull in Step 4, you push the retagged malicious image:

$ docker push myimage:latest
e9f56d359a24: : Mounted from maliciousimage
…
latest: digest: sha256:8360f9950adc9... size: 735

That will reuse all of the layers from the previously pushed maliciousimage, so they won’t be uploaded again.

The attack could even be automated by modifying a repository client, like Containerd, to perform all of the steps required to push an image, as described in the Docker registry API, but hold the final manifest upload. This final step, that causes the tag mutation, would be triggered by a webhook from the image scanner, an event from Kubernetes audit log, etc.

Mitigating tag mutability issues

Problems related to tag mutation are very common and easy to reproduce, either due to bad luck when you hit an unexpected image update, or provoked intentionally.

Aside from generic security practices, such as using digitally signed images, avoiding public or untrusted repositories and implementing strict control of where your images are being pulled from, let’s talk about some good practices that will save you some trouble.

Avoid usage of highly unstable/mutable tags (latest, …)

Try to avoid tags like “latest” or similar ones for deployments in production, critical environments, or whatever isn’t a quick test. Stick to more stable tags, like specific version tags, although there is no guarantee that these cannot mutate either.

Furthermore, “latest” is just a convention. It’s not guaranteed to exist at all or point to the latest existing version. It could point to a version three years old!

Avoid creating mutant tags

If you have control over the image tagging process, avoid pushing tags that already exist to keep from creating a mutant tag. Some registries have options to create immutable tags, failing if you try to push an already existing tag.

If there is no option for tag immutability, you can add additional measures in your development process or CI/CD pipeline, like using the Docker API to retrieve a manifest by tag and check its existence, aborting the push.

Digest instead of tag

Use the image digest like @sha256:8360f9950adc9... instead of named tags. It guarantees that the image you use will always be the same.

You can use docker images --digests to display the image digest, or inspect the pod once deployed with kubectl inspect pod.

Our mutating admission controller

Sysdig Admission Controller uses Sysdig Secure as an image scanner to validate the pod. This ensures that the scheduler pulls exactly the same image that was previously scanned. Don’t worry, the original image and tag is kept in an annotation for reference, so it’s not lost.

Using other non-mutating admission controllers or other approaches, like scanning after building or when an image is pushed to the registry, can expose your cluster to the TOCTOU issue.

Already using Sysdig?
We actively support the Sysdig Admission Controller extension.
Find the most up to date details in the project page →

Inmutable tags

There are other problems with mutant tags, like garbage collection not working as expected. Or the fact that the Registry API deletes images by digest but images listed by tag can cause some confusion and bugs.

Some registries, like Harbor since 1.10, ECR since July 2019, and others, offer configuration settings to make tags immutable. In Harbor, you can define rules per repository, or by using wildcard expressions. Enabling immutability, the registry won’t allow pushing a tag that already exists.

Conclusion

Mutable or immutable tags?

Mutability is useful and convenient, but it can also be dangerous if you aren’t aware and prepared to manage it.

For more information, you can check these slides from my presentation at Virtual Rejekts 2020 and our blog post on Sysdig Admission Controller.

So, “with great power comes great responsibility”, and “knowledge is power.” Take these two principles, unleash the power of mutant tags and apply your knowledge, good practices and tools, like Sysdig Secure image scanning and Sysdig Admission Controller, to keep them under control and prevent unexpected issues.