How to detect Kubernetes vulnerability CVE-2019-11246 using Falco

A recent CNCF-sponsored Kubernetes security audit uncovered CVE-2019-11246, a high-severity vulnerability affecting the command-line kubectl tool. If exploited, it could lead to a directory traversal, allowing a malicious container to replace or create files on a user’s workstation. This vulnerability stemmed from an incomplete fix of a previously disclosed vulnerability (CVE-2019-1002101).

CVE Overview

The CVE consists of two phases:

1. Crafting a malicious tar binary seeded in a container.
2. The execution of vulnerable kubectl cp command

Are you vulnerable?

Run kubectl version --client and if it does not say client version 1.12.9, 1.13.6, or 1.14.2 or newer, you are running a vulnerable version. How do you upgrade? Follow installation instructions here https://kubernetes.io/docs/tasks/tools/install-kubectl/ Not every installation guide will provide up-to-date kubectl versions at the time of this announcement. So, always confirm with kubectl version.

CVE Exploit Detection

In this blog post, we’re going to show you how to use Falco, a CNCF open source project, to detect both phases of the exploit described in this CVE. Falco is an open source container security monitor designed to detect anomalous activity in containers and hosts. Falco, originally a Sysdig project, taps into system calls to generate an event stream of all system activity. Falco’s rules engine then allows you to create rules based on this event stream, allowing you to alert on system call events that seem abnormal. Falco’s rich language allows you to write rules at the host level and identify suspicious activity.

1. Use Falco to detect the replacement of /bin/tar with a malicious binary

- rule: Write below monitored dir
  desc: an attempt to write to any file below a set of binary directories
  condition: >
    evt.dir = < and open_write and monitored_dir
    and not package_mgmt_procs
    and not coreos_write_ssh_dir
    and not exe_running_docker_save
    and not python_running_get_pip
    and not python_running_ms_oms
    and not google_accounts_daemon_writing_ssh
    and not cloud_init_writing_ssh
    and not user_known_write_monitored_dir_conditions
  output: >
    File below a monitored directory opened for writing (user=%user.name
    command=%proc.cmdline file=%fd.name parent=%proc.pname pcmdline=%proc.pcmdline gparent=%proc.aname[2] container_id=%container.id image=%container.image.repository)
  priority: ERROR
  tags: [filesystem, mitre_persistence]

Output:

10:16:16.067583414: Error File below a known binary directory opened for writing (user=root command=cp /tmp/tar.hacked /bin/tar file=/bin/tar parent=bash pcmdline=bash gparent=containerd-shim container_id=404b298fb6ee image=nginx) k8s.ns=default k8s.pod=nginx container=404b298fb6ee k8s.ns=default k8s.pod=nginx container=404b298fb6ee

From the falco rule alert output, there is an attempt to overwrite the /bin/tar file with a file named tar.hacked from /tmp folder. This seems really suspicious. The output also gives rich detail about the container, image, pod, namespace information etc.

2. Use Falco to detect vulnerable kubectl cp:

- macro: safe_kubectl_version
  condition: (jevt.value[/userAgent] startswith "kubectl/v1.15" or
              jevt.value[/userAgent] startswith "kubectl/v1.14.3" or
              jevt.value[/userAgent] startswith "kubectl/v1.14.2" or
              jevt.value[/userAgent] startswith "kubectl/v1.13.7" or
              jevt.value[/userAgent] startswith "kubectl/v1.13.6" or
              jevt.value[/userAgent] startswith "kubectl/v1.12.9")

# CVE-2019-1002101
# Run kubectl version --client and if it does not say client version 1.12.9,
1.13.6, or 1.14.2 or newer,  you are running a vulnerable version.
- rule: K8s Vulnerable Kubectl Copy
  desc: Detect any attempt vulnerable kubectl copy in pod
  condition: kevt_started and pod_subresource and kcreate and
             ka.target.subresource = "exec" and ka.uri.param[command] = "tar" and
             not safe_kubectl_version
  output: Vulnerable kubectl copy detected (user=%ka.user.name pod=%ka.target.name ns=%ka.target.namespace action=%ka.target.subresource command=%ka.uri.param[command] userAgent=%jevt.value[/userAgent])
  priority: WARNING
  source: k8s_audit
  tags: [k8s]

Output:

09:51:26.488577024: Warning Vulnerable kubectl copy detected (user=kubernetes-admin pod=nginx ns=default action=exec command=tar userAgent=kubectl/v1.13.4 (linux/amd64) kubernetes/c27b913)

As the CVE fix describe, kubectl version 1.13.4 suffers the vulnerability which should catch DevOps’ attention so that they can setup the patch plan accordingly. This highlights the potential of the Falco engine to accurately identify malicious activity occurring in containers and Kubernetes environments.

Things to consider

As some attacks come in multiple phases e.g. bitcoin mining, it will be very important to correlate malicious activities to provide a better insight into the attack. In the kubectl cp vulnerability exploit example above, both /bin/tar file replacement and kubectl cp happened inside the same pod. Adding the two events up based on pod name makes perfect sense to reveal a true kubectl cp vulnerability exploit.

Conclusion

Kubernetes is evolving fast and code issues sometimes evolve into security vulnerabilities. From a security standpoint, it is always important to keep your application/infrastructure patched in time. More importantly, it is crucial to have a robust and comprehensive security solution for your cloud native environment in production.

Reference links:

https://www.cvedetails.com/cve/CVE-2019-1002101/ https://github.com/kubernetes/kubernetes/pull/75037 https://cve.mitre.org/cgi-bin/cvename.cgi?name=2019-11246 https://www.openwall.com/lists/oss-security/2019/06/21/1