A recent CNCF-sponsored Kubernetes security audit uncovered CVE-2019-11246, a high-severity vulnerability affecting the command-line kubectl tool. If exploited, it could lead to a directory traversal, allowing a malicious container to replace or create files on a user’s workstation. This vulnerability stemmed from an incomplete fix of a previously disclosed vulnerability (CVE-2019-1002101).
The CVE consists of two phases:
- Crafting a malicious tar binary seeded in a container.
- The execution of vulnerable
Are you vulnerable?
kubectl version --client and if it does not say client version 1.12.9, 1.13.6, or 1.14.2 or newer, you are running a vulnerable version.
How do you upgrade?
Follow installation instructions here https://kubernetes.io/docs/tasks/tools/install-kubectl/
Not every installation guide will provide up-to-date kubectl versions at the time of this announcement. So, always confirm with
CVE Exploit Detection
In this blog post, we’re going to show you how to use Falco, a CNCF open source project, to detect both phases of the exploit described in this CVE.
Falco is an open source container security monitor designed to detect anomalous activity in containers and hosts. Falco, originally a Sysdig project, taps into system calls to generate an event stream of all system activity. Falco’s rules engine then allows you to create rules based on this event stream, allowing you to alert on system call events that seem abnormal. Falco’s rich language allows you to write rules at the host level and identify suspicious activity.
1. Use Falco to detect the replacement of /bin/tar with a malicious binary
- rule: Write below monitored dir desc: an attempt to write to any file below a set of binary directories condition: > evt.dir = < and open_write and monitored_dir and not package_mgmt_procs and not coreos_write_ssh_dir and not exe_running_docker_save and not python_running_get_pip and not python_running_ms_oms and not google_accounts_daemon_writing_ssh and not cloud_init_writing_ssh and not user_known_write_monitored_dir_conditions output: > File below a monitored directory opened for writing (user=%user.name command=%proc.cmdline file=%fd.name parent=%proc.pname pcmdline=%proc.pcmdline gparent=%proc.aname container_id=%container.id image=%container.image.repository) priority: ERROR tags: [filesystem, mitre_persistence]
10:16:16.067583414: Error File below a known binary directory opened for writing (user=root command=cp /tmp/tar.hacked /bin/tar file=/bin/tar parent=bash pcmdline=bash gparent=containerd-shim container_id=404b298fb6ee image=nginx) k8s.ns=default k8s.pod=nginx container=404b298fb6ee k8s.ns=default k8s.pod=nginx container=404b298fb6ee
From the falco rule alert output, there is an attempt to overwrite the
/bin/tar file with a file named
/tmp folder. This seems really suspicious. The output also gives rich detail about the container, image, pod, namespace information etc.
2. Use Falco to detect vulnerable kubectl cp:
- macro: safe_kubectl_version condition: (jevt.value[/userAgent] startswith "kubectl/v1.15" or jevt.value[/userAgent] startswith "kubectl/v1.14.3" or jevt.value[/userAgent] startswith "kubectl/v1.14.2" or jevt.value[/userAgent] startswith "kubectl/v1.13.7" or jevt.value[/userAgent] startswith "kubectl/v1.13.6" or jevt.value[/userAgent] startswith "kubectl/v1.12.9") # CVE-2019-1002101 # Run kubectl version --client and if it does not say client version 1.12.9, 1.13.6, or 1.14.2 or newer, you are running a vulnerable version. - rule: K8s Vulnerable Kubectl Copy desc: Detect any attempt vulnerable kubectl copy in pod condition: kevt_started and pod_subresource and kcreate and ka.target.subresource = "exec" and ka.uri.param[command] = "tar" and not safe_kubectl_version output: Vulnerable kubectl copy detected (user=%ka.user.name pod=%ka.target.name ns=%ka.target.namespace action=%ka.target.subresource command=%ka.uri.param[command] userAgent=%jevt.value[/userAgent]) priority: WARN source: k8s_audit tags: [k8s]
09:51:26.488577024: Warning Vulnerable kubectl copy detected (user=kubernetes-admin pod=nginx ns=default action=exec command=tar userAgent=kubectl/v1.13.4 (linux/amd64) kubernetes/c27b913)
As the CVE fix describe, kubectl version 1.13.4 suffers the vulnerability which should catch DevOps’ attention so that they can setup the patch plan accordingly. This highlights the potential of the Falco engine to accurately identify malicious activity occurring in containers and Kubernetes environments.
Things to consider
As some attacks come in multiple phases e.g. bitcoin mining, it will be very important to correlate malicious activities to provide a better insight into the attack. In the
kubectl cp vulnerability exploit example above, both
/bin/tar file replacement and
kubectl cp happened inside the same pod. Adding the two events up based on pod name makes perfect sense to reveal a true
kubectl cp vulnerability exploit.
Kubernetes is evolving fast and code issues sometimes evolve into security vulnerabilities. From a security standpoint, it is always important to keep your application/infrastructure patched in time. More importantly, it is crucial to have a robust and comprehensive security solution for your cloud native environment in production.