Mitigating CVE-2021-20291: DoS affecting CRI-O and Podman
The CVE-2021-20291 medium-level vulnerability has been found in containers/storage Go library, leading to Denial of Service (DoS) when vulnerable container engines pull an injected image from a registry.
The container engines affected are:
Any containerized infrastructure that relies on these vulnerable container engines are affected as well, including Kubernetes and OpenShift.
Exploiting the vulnerabilities, adversaries could stop the engine execution after causing a Denial of Service against any containerized infrastructure which uses the reported vulnerable container engine.
In this article, you’ll understand how CVE-2021-20291 works, what parts of Kubernetes are affected, and how to mitigate it.
Before we start talking about CVE-2021-20291 itself, we need to briefly understand how containers engines pull images.
A Docker image is an immutable (unchangeable) file that contains the source code, libraries, dependencies, tools, and other files needed for an application to run. Due to their read-only quality, these images are sometimes referred to as snapshots
Each of the files that make up a Docker image is known as a layer. These layers form a series of intermediate images, built one on top of the other in stages, where each layer is dependent on the layer immediately below it
One of the actions the engine does during the pull image action from a registry is to read the layers list and start downloading and upack all the listed components.
The CVE-2021-20291 issue
The CVE-2021-20291 affects the containers/storage Go library till v1.28.0. The container engines affected are:
To check the current version you have installed in your system you can do:
Podman --version # or crio --version
To exploit CVE-2021-20291, the attacker has to add an extra malicious layer to exploit the vulnerability in an image and upload it into the registry. Once the victim pulls the image, the injected layer is downloaded and unpacked by the engine, triggering the vulnerability.
The engine execution flow involves three routines in the following steps:
1. The main routine downloads the layers of the injected image, including the malicious one.
2. A second routine is launched to decompress the layers downloaded, writing the output to stdout. A third routine waits till all the layers are ended to close the chdone channel.
3. The output of step 2 is used to untar the data. At this point, if the decompressed data isn’t a tar archive the action will end with an error which blocks the reading of the rest of the data left.
4. The third routine waiting will never end due to the error occurred, leaving it in a deadlock. Same for the main routine, waiting for the third routine to end.
The execution ends in a deadlock where the lock is acquired and never gets released. This eventually causes a DoS since the other threads and processes stop their execution and wait forever.
The impact of CVE-2021-20291
According to the CVSS system, CVE-2021-20291 scores 6.5 as medium severity.
The vulnerability has a low attack complexity and a very high impact on availability since it stops the functionalities of the container engines. On the other hand, it has no impact in terms of Confidentiality and integrity.
The container engines affected are CRI-O and Podman which are both supported by Kubernetes. This means that Kubernetes clusters using the mentioned engines are affected as well by this vulnerability.
Deploying the malicious image in different nodes, it costs the break of all of them, requiring to restart the nodes.
If you cannot patch your systems immediately, detecting exploitation attempts of this vulnerability is critical to preventing or stopping an attack. Even though you might have already upgraded your system and containers affected by the vulnerability, it is still necessary to detect any exploitation attempts and post-breach activities in your environment
Following docker images best practices will help you mitigate this kind of vulnerability. In particular, using security techniques like private registries and signing that ensure the deployed images are trusted and security signed to prevent any tampering attempts.
As previously said, to exploit the vulnerability the victim has to pull the malicious image into the cluster from a registry. In this case, using private registry helps to mitigate the vulnerability.
The most well-known container registry is DockerHub, which is the standard registry for Docker and Kubernetes. The problem with public registries is that you don’t have full control over their actions and anyone can upload a custom image. The customisation might include some hidden code used by an adversary to get access to your environment or exploit well known vulnerabilities.
Private registries provide multiple different storage and authentication options and can be customized to your individual requirements.
In addition from a security prospective private registry helps for:
- Control where your images are stored.
- Control how you can access them.
- More privacy for proprietary and private image
Most image scanners offer tools to ensure the deployed images come from your own private registry.
Combining the image scanner with an admission controller, you can block from deployment any image that doesn’t pass the scan. Applied to this case, stopping any images that came from public registries, and allowing those from private repositories.
Container Image Signing
In addition to using a private registry, to avoid deploying malicious images in your environment, container image signing comes to your help. With image signing and verification process is possible to add private cryptographic signatures to an image, adding a digital signature that helps to ensure proof-of-origin, authenticity and provenance for images.
Container image signing permits:
- Validating where a container image came from,
- Checking that the image has not been tampered
- Setting policies to determine which validated images can be pulled to a host.
On the verification side, if pulling an image without a signature or incorrect signature, the host won’t be able to pull it from the registry.
With an admission controller mechanism you can ensure that just signed and trusted images are deployed in your environment, preventing any intentional malicious images from getting deployed.
We’ve seen how CVE-2021-20291 can be exploited with a crafted image layer to cause a DoS in your infrastructure.
Luckily, upgrading to the latest versions, and following container images best practices, will help you avoid this and similar attacks.