In the following blog post we are going to get a closer look at Falco CVE-2019-8339 vulnerability. Recently, a member of the Falco community privately disclosed a capacity related vulnerability which, under circumstances where a malicious actor has already gained access to your system, could allow the actor to further bypass Falco’s detection of abnormal activity. The final details are still being worked out, but we believe CVE-2019-8339 will be classified as Medium severity according to the CVSS methodology. We have addressed the vulnerability in version 0.15.0, and this blog post explains the vulnerability and ways to mitigate it in more detail.
Users may want to upgrade to 0.15.0, which provides new ways to detect that Falco’s handling of system call events is being interrupted as a fix for CVE-2019-8339. More importantly, we encourage all users to read our updated documentation to best optimize your deployment. We also want to thank the community for their diligent work in finding and resolving the issue.
Understanding CVE-2019-8339.
Within the kernel module or eBPF program, we use a buffer that is shared between kernel and user space to store events for processing. This buffer is used to pass information about each system call, including:
- the system call id
- any arguments to the system call
- other useful context including thread/user information
- the result (success/failure)
- any resources created by the system call (new fds, processes, etc).
When a new system call event is processed, the kernel module/eBPF program checks that there is enough space in the buffer for the event. If there isn’t, the event is discarded, a counter that tracks the number of dropped events is incremented, and the kernel module/eBPF program moves on to the next event.
If an attacker already has access to the system where Falco is running, they can run a program that executes a flood of system calls. Along with that flood, the attacker can mix in malicious syscalls which might be dropped by the kernel module/eBPF program and be undetected. This can be executed on both containers and hosts.
The community member also provided a useful proof-of-concept program to demonstrate the vulnerability. The program forked 256 subprocesses per hardware processor. All subprocesses then simultaneously opened 5 files back-to-back in an unrolled loop, where each opened file had a 64k filename. On a 16 core processor, this would be approximately 20k system calls in ~500ms. The number of system calls itself is actually fairly reasonable. Even on lightly loaded machines you can easily expect to see tens of thousands of system calls per second. The real cost is in the 64k filename. A typical open system call event takes ~56 bytes, excluding the filename. Assuming a ~10 character filename, These opens are ~1000x larger than a typical open system call. Converting this to typically-sized system calls, the proof-of-concept program executes ~20 million system calls, all in 500ms!
One way to reduce the risk of dropped system calls is by increasing the size of the shared buffer between user/kernel space. For example, increasing the size of the default kernel buffer from 8mb to 128mb (per cpu) resulted in no dropped system calls, even under the extreme workload used by the proof-of-concept program. However, a determined attacker can always spawn enough processes and threads and use large enough system call arguments to cause system call drops.
The concept of resource starvation and/or memory sizing is not unique to Falco. Here are a few other examples of the same kind of issue:
- Snort Capacity Planning And Buffer Sizing: http://mikelococo.com/2011/08/snort-capacity-planning/
- CentOS 6 Backlog limit exceeded error and freeze: https://www.hungred.com/how-to/backlog-limit-exceeded-error-freeze-centos-6/
- Improving the Performance of Passive Network Monitoring Applications with Memory Locality Enhancements: http://www.syssec-project.eu/m/page-media/3/pcap-comcom.pdf
- Audisp is flooding /var/log/messages (RedHat): https://access.redhat.com/discussions/2106451
- Audit buffering and rate limiting: http://blog.siphos.be/2015/05/audit-buffering-and-rate-limiting/
- Wazhu Anti-flooding mechanism: https://documentation.wazuh.com/3.0/user-manual/capabilities/antiflooding.html
We used these resources to help shape the features we added to Falco to mitigate the CVE-2019-8339 vulnerability. Specifically, we focused on improvements that help identify when Falco is missing system call events and as a result might be under attack by a flood of system calls.
Detecting dropped system call events (CVE-2019-8339) in Falco.
Even prior to Falco 0.15.0, there are existing mechanisms to detect dropped system call events:
When run with -v (verbose output), at exit Falco will print a summary of the number of system call events handled and processed. Here’s an example:
Driver Events:59613
Driver Drops:0
Elapsed time: 0.029, Captured Events: 2417, 83534.94 eps
The key metric is the “Driver Drops” count, which should ideally be 0 or very low compared to the number of Driver Events.
When run with -s (stats file), Falco will periodically print statistics on the number of events processed and dropped. Here’s an example:
{"sample": 1, "cur": {"events": 12458, "drops": 0, "preemptions": 0}, "delta": {"events": 12458, "drops": 0, "preemptions": 0}, "drop_pct": 0},
{"sample": 2, "cur": {"events": 26145, "drops": 0, "preemptions": 0}, "delta": {"events": 13687, "drops": 0, "preemptions": 0}, "drop_pct": 0},
The key metrics are the drops count as well as the drop_pct value.
Changes in Falco 0.15.0 to handle dropped system calls to address CVE-2019-8339.
In Falco 0.15.0, we made several changes to how Falco handles dropped system call events to address CVE-2019-8339. The changes were a mix of new features and performance improvements:
- Falco can now take configurable actions such as sending alerts, logging messages, or even exiting falco entirely when it notes that there were dropped system call events. This allows you to more easily detect when there are dropped events and would definitely alert you to an attack of the form described in CVE-2019-8339. There’s a new page on the falco.org site devoted to this feature and its configuration options.
- We made a change to how we read events from the system call buffer to use an adaptive algorithm that reads more frequently when the reader finds a non-empty buffer. This decreases the likelihood of dropped events.
- We started fetching container meta-data asynchronously. This, too, reduces the likelihood of dropped events in busy environments with containers being created constantly.
The first change is the most important one, as it provides an explicit signal that denotes dropped system calls without having to monitor the drop rate/stats file separate from Falco. The other changes help reduce the chances of false positives, where dropped events are mistaken as an indicator of an attack.
Other upcoming changes.
One item on the Falco roadmap is to provide a Prometheus endpoint that exposes metrics on Falco’s operation, including metrics on the number of system calls processed and dropped. This should make it easier to dynamically monitor the behavior of Falco and look for dramatic changes in the number of system calls, which would be a very strong indicator that an attack like this is underway.
We are also going through a security audit for both Falco and sysdig open source. This audit will help us harden the Falco engine, the shared libraries from sysdig, and the underlying kernel interface Falco uses. We’d like to thank the CNCF for contributing to the financial support required for the Falco security audit.
Conclusion.
We hope this blog post provided useful details on the CVE-2019-8339 vulnerability, the real-world risks, and how to mitigate it in Falco. If you’d like more information, you can always reach us at Falco community Slack channel or Github.
Acknowledgements.
We’d like to thank Aviv Sasson for reporting CVE-2019-8339. We’d also like to thank the members of the Falco community for testing out preview releases of Falco that had these fixes and providing valuable feedback.