Exploiting and detecting CVE-2021-25735: Kubernetes validating admission webhook bypass

By Stefano Chierici - APRIL 28, 2021
Topics: Kubernetes & Container Security, Open Source, Threat Research

SHARE:

Facebook logo LinkedIn logo X (formerly Twitter) logo

content:

Table of contents Preliminary The issue Exploiting CVE-2021-25735 The impact Mitigating CVE-2021-25735 Conclusion

The CVE-2021-25735 medium-level vulnerability has been found in Kubernetes kube-apiserver that could bypass a Validating Admission Webhook and allow unauthorised node updates.

The kube-apiserver affected are:

  • kube-apiserver v1.20.0 to v1.20.5
  • kube-apiserver v1.19.0 to v1.19.9
  • kube-apiserver <= v1.18.17

You are only affected by this vulnerability if both of the following conditions are valid:

  1. You are using Validating Admission Webhook for nodes
  2. You are using Validating Admission Webhook with old values

By exploiting the vulnerability, adversaries could bypass the Validating Admission Webhook checks and allow update actions on Kubernetes nodes.

In this article, you’ll understand how CVE-2021-25735 works, what parts of Kubernetes are affected, and how to exploit and mitigate it with Falco.

Preliminary

Before we start exploring the CVE-2021-25735 itself, let’s take a quick look at what admission controllers are.

Admission controllers are special controllers that can intercept Kubernetes API requests, and modify or reject them based on custom logic.

You can learn more in our “Kubernetes admission controllers in 5 minutes” article.

Admission Webhooks are HTTP callbacks that Admission Controllers call when there is an API request. There are two types of Admission Webhooks in Kubernetes:

  • Mutating Admission Webhooks
  • Validating Admission Webhooks.

For this specific CVE-2021-25735, we’ll focus our attention on the Validating Admission controller.

The Validation Admission Controller receives resource requests after passing authentication and authorization, but before admitting those resources into the cluster. Then, the admission controller validates the request, returning a fail or successful response to the API Requests. Only after receiving a successful response, the resource will be admitted to the cluster.

The CVE-2021-25735 issue

The CVE-2021-25735 kube-apiserver vulnerability could allow node updates bypassing a Validating Admission Webhook. The kube-apiserver affected are:

  • kube-apiserver v1.20.0 to v1.20.5
  • kube-apiserver v1.19.0 to v1.19.9
  • kube-apiserver <= v1.18.17

The vulnerability can be exploited when an update action on node resources is performed and an admission controller is in place and configured to validate the action. The information that is provided to the admission controller might not be correct, thus, leading the admission controller to accept requests that should be blocked.

In this case, kube-apiserver passes request.object and request.oldObject to Validating Admission Webhook. The request.object contains new node configs and request.oldObject contains old configs.

As you can see from the code changes, the overwrite of a part of oldNode.spec with node.spec causes this issue and it has been removed in the fixed version. In particular, Node.ObjectMeta overwrites the old values which may be used for admission decisions. Since the overwriting happens before the admission evaluation, editing the parameters used for the decision may cause unauthorised change in the cluster nodes.

Exploiting CVE-2021-25735

Let’s go now through how to exploit the CVE-2021-25735. To replicate the issue you need to have one of the vulnerable kube-apiserver versions deployed in your environment and a validating admission controller working with nodes update actions into your cluster.

In the repo you can find an example of a nodejs server admission controller webhook on nodes changes.

In this scenario the custom label changeAllowed is set in advance to enable or disable the opportunity to apply changes to specific cluster nodes. The admission controller uses this custom label value to make the admission decision on the submitted change. If the label is set to false, the admission controller refuses the change otherwise it returns a success validation.

When the node label changeAllowed is set to false we get the following error from the admission controller, since we are not allowed to perform any changes on the node:

error: nodes "ip-172-20-46-130.ec2.internal" could not be patched: admission webhook "validationwebhook.validationwebhook.svc" denied the request: Validation failed
You can run `kubectl replace -f /tmp/kubectl-edit-irc64.yaml` to try this update again.

Let’s try now to change the changeAllowed: "true" label, and add a new one in the node. As we can see from the screen below, this time the update action has been accepted by the Admission controller and the new label is added without any blocks.

By performing the combined action to change the changeAllowed label to true and adding a new one, the vulnerability has been triggered and the new value has been overwritten by the admission controller.

In this way the admission controller validation has been completely bypassed, permitting any kind of changes on the cluster nodes without any validation.

The impact of CVE-2021-25735

According to the CVSS system, CVE-2021-25735 scores 6.5 as medium severity.


To learn more about how a vulnerability score is calculated, Are Vulnerability Scores Tricking You? Understanding the severity of CVSS and using them effectively

As we have seen, the vulnerability has a low attack complexity and a high impact in terms of integrity and confidentiality.

In the circumstances described before, the vulnerability permits to completely bypass the controls in place in the validating Admission Controller over nodes updates.

Adversaries might be able to perform unwanted and unauthorised changes on nodes settings without getting stopped by the admission controller.

Mitigating CVE-2021-25735 with Falco

If you’re impacted by this CVE, the issue is fixed in the following versions:

If you cannot patch your systems immediately, detecting exploitation attempts is important to preventing or stopping an attack. Even though you might have already upgraded your system and containers affected by the vulnerability, it is still necessary to detect any exploitation attempts in your environment.

Using Falco open source tool it’s possible to detect when the vulnerability is triggered for this specific scenario. To detect this kind of event you need to enable the Kubernetes Audit Logging in the cluster.

In a Kubernetes cluster, kube-apiserver is in charge to perform the auditing. When an event is processed by kube-apiserver, it compares the list of rules in the audit policy in order. The first matching rules also dictate the audit level of the event. The audit policy to enable the audit events on nodes is the following:

Once the Kubernetes Auditing is enabled, you can use and customize the following rule based on your environment, based on which old values of nodes are used for the admission decision.

In this specific case with the following rule we check if someone is editing and setting the custom node label ChangeAllowed to true to bypass the admission controller.

- rule: "Bypass Validating Webhook Node label - Validation Webhook Bypass CVE-2021-25735"
  desc: "Bypass Validating Webhook label - The node label changeAllowed has been set to true, enabling changes on the nodes. Possible Validation Webhook Bypass CVE-2021-25735"
  condition: "ka.verb in (create,update,patch) and ka.target.resource=nodes and jevt.value[/requestObject/metadata/labels/changeAllowed] = true"
  exceptions:
  output: "The changeAllowed label in the k8s node %ka.target.name has been modified. Possible Validation Webhook Bypass CVE-2021-25735 (user=%ka.user.name node=%ka.target.name label value=%jevt.value[/requestObject/metadata/labels])"
  priority: "WARNING"
  tags:
  - "k8s"
  source: "k8s_audit"
  append: false

You can use and modify the rule reported here to check the changes in your environment on the parameters involved in your admission decision.

With this rule enabled in Falco, we are able to detect if someone is trying to edit the node and use the vulnerability to bypass the admission controller.

Conclusion

We’ve seen how CVE-2021-25735 can be exploited to bypass a Validating Admission Webhook and how to detect the exploitation using Falco.

Luckily, upgrading to the latest versions, and following container images best practices, will help you avoid this and similar attacks.

Sysdig Secure can help you benchmark your Kubernetes cluster and check your security posture with security standards like PCI, NIST, CIS, and more. Try it today!

If you would like to find out more about Falco:

Subscribe and get the latest updates