Recently, a new Kubernetes related vulnerability was announced that affected the kube-apiserver. This was a denial of service vulnerability where authorized users with write permissions could overload the API server as it is handling requests. The issue is categorized as a medium severity (CVSS score of 6.5) and can be resolved by upgrading the kube-apiserver to v1.11.8, v1.12.6, or v1.13.4.
What is Kubernetes API?
In Kubernetes, the control plane on the master node consists of the API Server, the Controller Manager and Scheduler(s). The API Server is the central management entity that directly communicates with etcd and serves the Kubernetes API used both for internal cluster communication and external communication via kubectl or other clients.
Affected kube-apiserver versions:
The following versions of kube-apiserver are vulnerable:
- v1.0.0-1.10.x
- v1.11.0-1.11.7
- v1.12.0-1.12.5
- v1.13.0-1.13.3
You can also mitigate the vulnerability prior to upgrade by removing ‘patch’ permissions from untrusted users.
How Sysdig can Mitigate CVE-2019-1002100 Exploitation
Sysdig has built the only cloud-native intelligence platform that is designed to secure, monitor and troubleshoot your next-generation environment. Users can create alerts, monitor and detect abnormal behavior, and correlate user activity to understand the entire security incident.
1. Set up an alert
Here we can see that the current traffic conditions are normal, but if something spikes into the red zone (abnormally high inbound traffic that hits the API server), an alert will be triggered and the appropriate teams will be notified.
2. Detect abnormal inbound and outbound traffic
When would a spike happen? A DoS attack on the API Server would lead to the server not being able to handle additional requests. This type of attack occurs at Layer 7 and it is hard to detect based on HTTP request signatures. Signature based detection works well for detecting abnormal string patterns with limited size. The DoS attack on Kubernetes is conducted by a malicious user whom’s be authenticated and authorized, the patch request(URI, Verb) looks normal but with abnormal payload size.
One typical DoS attack come by generating huge amount of request to consume all the process capacity of the server (Syn flooding etc.). From the metric above, the connection numbers seem stable, but it is misleading. This shows that the increase in traffic is not due to a connection spike, but rather a burst in traffic that flags this as a potential DoS attempt.
Using Sysdig Monitor, users can understand you k8s API server inbound traffic size posture and get details such as container name, specific threshold crossed, etc easily.
As a next step in your research, a user can assess outbound traffic out of that compromised pod to understand the DoS attack better. Given the k8s API server is internal facing, a DoS attack towards k8s API server should be launched from a compromised pod with high probability.
Here we can see outbound traffic leaving pod in the example above and understand which specific service is generating abnormal traffic.
Disclaimer: The k8s API server inbound traffic size is dependent on your cluster size and the workload. You should see a inbound traffic hike when a mass deployment happens. However, it is necessary to monitor your k8s API server traffic status.
By this point, a user has been alerted on a abnormal traffic spike, proceeded to see both inbound/outbound traffic of a particular service as well other container/pod details. The last step would be to audit and understand the commands ran during this specific time frame.
3.Inspect user commands post incident
Now, in Sysdig Secure, you can see exactly what commands were run during this attack on the k8 api server. In this attack, a malicious actor ran a wget to download some content from a web server, as well as bash, ls and then a denial of service (DoS) attack to overload the server.
Conclusion
To wrap up, the alerting, event monitoring and dashboard features in Sysdig Monitor can be leveraged to easily detect anomalies such as a DoS attack. Even though this CVE was characterized as medium, your API server can be seriously exploited by malicious users that compromised a container in which a service account with patch privilege granted.
To learn more about Sysdig platform visit the page or sign up for a free trial.