The history of successful open source tooling built on common Linux libraries,
by Loris Degioanni
As Falco grows in popularity, many new users get exposed to it on a daily basis. As should be expected, most of these users are not aware of what the architecture underneath Falco is. What components play a role in powering it? How do these components relate to each other? I thought it would be fun to write a blog post that answers these questions. And I thought it would be fun to write it with an historical perspective. Falco has many pieces of software and is inspired by work that has been done way before it was created. As a consequence, understanding the history that brought it to its creation can be as interesting as understanding how it works.
Naming clarification: Sysdig Inc. vs. sysdig
Currently, the word sysdig is used to indicate two different things:
- An Open Source, Apache 2 licensed tool for Linux system call capture and analysis, hosted at https://github.com/draios/sysdig/
- A company selling security and visibility products, with the website https://sysdig.com/
The word sysdig in this document strictly refers to the former, and what is discussed in this document is strictly related to the free and open source (FOSS) sysdig libraries and CLI tools. (By the way: yes, the community is realizing that this can lead to confusion and is working on solving it).
The origins: libpcap, tcpdump and snort
The ideas behind sysdig and Falco originate with packet capture, a field in which many of the original team developers have spent many years. Among other things, we’ve been for example inspired by technologies like BPF, libpcap, tcpdump and snort.
The tcpdump stack, as described by McCanne and Jacobson in [1], looks like this:
We can observe three fundamental components:
- A packet capture component (BPF) sitting in the kernel of the operating system.
- A user level capture library (libpcap) controlling the capture component. This library implements most of the capture logic: live capture control, capture file management, packet filtering.
- A program (tcpdump) linking the packet capture library and offering a user interface around it.
Historically, these 3 components have been part of the same Open Source organization. Nowadays, libpcap and tcpdump still are, while the packet capture component is sometimes in the same repository [2] and sometimes, like in Linux, part of the operating system.
Based on the usefulness of libpcap as a generic packet capture and filtering interface, many subsequent tools have been created on top of it. For example, Snort, a network intrusion detection tool, adds a rule engine on top of libpcap to detect anomalies in network communications. Snort uses libcap as an external dependency.
Ideally, since libpcap is used by many projects, it should be separated from tcpdump, creating an independent organization for it. However, as of today, libpcap still sits together with tcpdump at tcpdump.org. This is because tcpdump is still considered the reference implementation for libpcap. In addition to that, tcpdump is small enough to not justify moving it to a separate place, plus its developers have historically been the most active maintainers of libpcap.
sysdig and Falco
Let’s now take a look at the sysdig stack:
We can see four main components:
- A capture component (sysdig-probe) sitting in the kernel of the operating system. The component this time captures system calls instead of packets, but its architecture is heavily inspired by the one of the original BPF.
- A user level capture library (libscap) controlling the capture component. This library implements most of the capture logic: live capture control, capture file management, and so on.
- A further user level library (libsinsp) offer a higher level of abstraction when dealing with the system state. It also implements filtering.
- A program (sysdig) linking the capture libraries and offering a user interface around them.
The similarities are pretty obvious, and not random. We tried to get what we believed worked well in the packet capture workflows and we transposed it to system calls. To the point that the sysdig stack saves system calls in files that have the same format of packet captures, called pcap-ng.
In a way that mirrors libpcap and tcpdump, this stack was originally developed for just one tool, sysdig. But with time, additional tools now leverage it, for example [4], [5] and [6].
Falco, of course, is one of them.
Does Falco depend on sysdig?
Falco, in reality, does not depend on sysdig, but on sysdig-probe, libscap and libsinsp, with a clean interface – and in a way that is not at all different from what [4] [5] and [6] do.
It should also be noted that, other than some historic baggage related to release tagging (and some unfortunate naming choices), there is nothing in sysdig-probe, libscap and libsinsp that is falco-specific.
Libscap and libsinsp were very much modeled around libpcap, with the same goal of being the base for multiple tools. To the point that, recently, we changed their license to Apache V2 to accommodate a more liberal use of them.
sysdig was modeled around tcpdump and, mostly out of pattern matching to our previous experiences, we included it in the same repo as the libraries, exactly like tcpdump.
Falco was instead modeled around snort. It implements a rule engine loosely inspired by the snort one, but on top of system calls rather than packets. To do that, it uses libraries that are part of the sysdig project in a way that is very similar to what snort does with the library in the tcpdump project.
How we can improve independence
Our goal for the future is separating the kernel probe, libscap and libsinsp from any independent vendor or organization.
We can have them live in a different repository. We can manage them in a way that guarantees stability and support for all of the clients. We can also make sure that Falco is fully disentangled from this repository, in particular with regards to release name tagging.
The Falco and sysdig teams are committed to these goals and are willing to put the proper resources in place to make this happen.
References
[1] The BSD Packet Filter: A New Architecture for User-level Packet Capture
[2] An architecture for high performance network analysis
[3] Snort, lightweight intrusion detection for networks
[4] sysflow
[5] Combining System and Network Visibility using nProbe and Sysdig
[6] Display Filter Reference: Sysdig System Call
A personal note
If you actually clicked on the references above, you have probably noticed that some of this stuff is VERY old, like over 25 years! While writing about them a couple of thoughts came to my mind.
The first one is that, yes, I’m getting old.
The second one is that, as a developer, it is rather satisfying to keep working on the evolution of what started many years ago as my graduation thesis. In this world of frameworks that last less than the time you need to deploy them, being able to focus on a multi-decade lasting passion is something that I consider a real privilege.