Falco vs. AuditD from the HIDS perspective

In this blog, we will compare and contrast Falco vs. AuditD from a Host Intrusion Detection (HIDS) perspective.

AuditD is a native feature to the Linux kernel that collects certain types of system activity to facilitate incident investigation.

Falco is the CNCF open-source project for runtime threat detection for containers and Kubernetes.

We will dig deeper into the technical details and cover the installation, detection, resource consumption, and integration between both products.

What is the Linux Auditing System (aka AuditD)?

AuditD is the userspace component to the Linux Auditing System, and it’s responsible for writing audit records to the disk.

Red Hat concurs as they note, “due to its close integration with the kernel, Auditd is capable of monitoring system calls, file access and modifications, as well as a list of preconfigured auditable events that are maintained by RHEL.” (Redhat, n.d.)

Also, note that AuditD lacks capability that enriches events with container runtime information. That’s why we limit the comparison scope as HIDS.

Comparing Falco vs. AuditD

Falco, a CNCF incubating project, can help detect any anomalous activities in cloud native environments with rich, out-of-the-box default rules.

Besides cloud native environments, Falco can also be used as a HIDS tool to detect any anomalous behavior in Linux.

When we started thinking about comparing Falco with other open source HIDS tools, AuditD’s treatment in the SANS’s report caught our attention. After reviewing the report, we decided to choose AuditD as the comparison target based on two reasons:

It natively integrates with Linux.
It also deeply relies on syscalls for detection.

After digging deeper into Falco and AuditD, we found that both are powerful HIDS tools. Although both rely on the syscalls to detect intrusion, their approaches on creating the rules and outputting the events data have huge differences.

	Falco	AuditD
Installation	Easy Uses ebpf or a kernel module	Easy
Capacity Management	Drops over a flood of syscall events	Drops over a flood of audit events
Granularity	More than 150 filters	About 40 filters
Context	Host/OS, Containers and Kubernetes Context for rules and outputs	Host/OS only
Event Verbosity	Good No decoding is required Supports customization The actual file is not included if a symlink file is executed A rule triggers one single event	Good May require decoding Does not support customization Includes both symlink file and the actual file A rule triggers multiple events
Resource Consumption	Decent	Decent
Event Forwarding	Most popular event hubs with Sidekick	Rsyslog server, Elasticsearch

Installing Falco vs. AuditD

Both Falco and AuditD are easy to install, and are available in most package managers. For debian systems, you can use apt-get:

# Trust the falcosecurity GPG key, configure the apt repository, and update the package lis
curl -s https://falco.org/repo/falcosecurity-3672BA8F.asc | sudo apt-key add -
echo "deb https://dl.bintray.com/falcosecurity/deb stable main" | sudo tee -a /etc/apt/sources.list.d/falcosecurity.list
sudo apt-get update -y
# Install kernel headers
sudo apt-get -y install linux-headers-$(uname -r)
# Install falco
sudo apt-get install -y falco

And to install AuditD:

sudo apt-get install auditd audispd-plugins

Both of them will be run as a system service, managed by systemd.

Also, note that auditd, the kernel code that hooks into syscalls and monitors them, has been part of Linux kernel since the days of 2.6. As opposed to Falco, no kernel module installation is required for AuditD.

How detection rules are in Falco and AuditD?

As we are focusing on behavior detection capabilities, we’ll start comparing how Falco and AuditD define such behaviors.

Both Falco and AuditD have the concept of detection rule, a quite common term in security, and they use their own format. Here is how they compare.

Falco rules

Falco rules are written in YAML format, and they support macro and list objects to reuse code:

Rule: Conditions under which an alert should be generated. A rule is accompanied by a descriptive output string that is sent with the alert.
Macro: Rule condition snippets that can be reused inside rules and even other macros. Macros provide a way to name common patterns and factor out redundancies in rules.
List: Collections of items that can be included in rules, macros, or other lists. For example, processes’ name, files’ names, or ports. Unlike rules and macros, lists cannot be parsed as filtering expressions.

A simple Falco rule example is like following:

- rule: Nmap Launched
  desc: Detect Nmap is launched
  condition: spawned_process and proc.name = nmap and container.id = host
  output: Nmap launched (user=%user.name parent=%proc.pname cmdline=%proc.cmdline)
  priority: WARNING

The rule above is triggered when the reconnaissance tool Nmap is launched.

If such an alert would be triggered in a production system, it would be a good indicator that an attacker has compromised the machine and started reconnaissance in your internal network.

This rule condition consists of three parts:

spawed_process: This predefined macro triggers when a system call execve is called (e.g., a process launched).
proc.name = nmap: The process name (full path excluded) equals to nmap.
container.id = host: The execve event happens inside the host, as opposed to a container.

You can get further details on how Falco rules are created in our blog, and you can check all the filters available to construct Falco rule conditions in the Falco user guide.

AuditD rules

AuditD rules can be file centric or syscall centric, and are created and managed with the auditctl client tool.

An example of a file centric rule would be:

auditctl -w /etc/ -p aw -k write_below_etc

This rule audits any events that write to files below /etc folder. Each flag sets a different property of the rule:

-w Inserts a path to watch, /etc/ in our case.
-p Sets permission filter on watch, append, and execute (aw) in the example.
-k Sets the filter key a.k.a rule name, like write_below_etc

The file centric rules are good for detecting any file tampering, reading sensitive data, anomalous process execution, etc.

An example of syscall centric rule would be:

auditctl -a always,exit -F arch=b64 -S socket -F success=1 -F a0=2 -k socket_activity

The rule above audits all of the successful socket syscall events over IPv4, and is defined with the following flags:

-a append the rule to the end. In the example, it will always audit socket syscall on its exit.
-S The syscall name or number that will trigger the audit, like socket.
-F Filters, or conditions, the event must pass for the audit record to start. They are based on event field values. For example, arch=b64 means 64 bit machine, success=1 means socket exits with success, and a0=2 means IPv4 protocol.

Why use syscalls?

You may have realized that both Falco and AuditD rely heavily on syscalls to detect intrusion on Linux machines. But why?

Syscalls are a programmatic way for applications to interact with the operating system kernel in order to access resources such as files, devices, the network, and so on.

It is the ultimate uniform way and source of truth to understand applications’ behavior. It doesn’t matter if the application is written in C/C++, Java, Python, Golang, and so on. From the syscall events, you still know what file has been created, read or write, or which IP address the application established a TCP connection with.

So, if both Falco and AuditD rely on syscall events … What’s the major difference between them?

Let’s answer this question in the next section.

Detection capabilities in Falco and in AuditD

We just stated that syscalls are the ultimate source of truth of application behavior, and that makes them the perfect tool to detect intrusions on Linux machines. With that in mind, one would think that, given enough computation resources, nothing is hidden from Falco or AuditD. This would mean unlimited detection capability.

But, is that the case?

In reality, security products should only use very limited resources to complete tasks. This limits detection capability is constrained to resource limits.

There is also a limitation of granularity of detection: what exactly can you detect?

Last but not least, there might be product design or implementation limitations to the verbosity of an event, preventing you from getting all the information you need to investigate an incident.

Capacity Management

In order to work with those limited computational resources, both Falco and AuditD are implemented to ensure the quality of service.

Without proper capacity management, the product would either crash when it can’t occupy enough resources, or it would occupy too many resources from the Linux machine under detection. Either way, it’s bad.

To avoid those cases, Falco uses a shared buffer between the kernel and userspace to pass system call information. The shared buffer size is 8MB. When the buffer becomes full, system calls will be dropped and Falco will throw an alert to warn the users.

The buffer size in AuditD can be configured via auditctl. The default number of buffers is 64 (and it is recommended to set it to 8192). When the audit events number exceeds the backlog limit, you should see an error message:

audit: backlog limit exceeded

Both Falco and AuditD are production ready. Falco has been adopted by companies Gitlab, Shopify, Sumo Logic, and more.

Rule Granularity

One important aspect to keep in mind is how fine grained our rules are. Will I be able to filter out only the events I’m interested in?

This is a key difference between Falco and AuditD.

Filtering sensitive file access events

Let’s think about a simple rule, like detecting processes reading the /etc/shadow file, which contains all of the system encrypted passwords.

In Falco’s out-of-the-box-rules, we have:

- rule: Read sensitive file untrusted
  desc: >
    an attempt to read any sensitive file (e.g. files containing user/password/authentication
    information). Exceptions are made for known trusted programs.
  condition: >
    sensitive_files and open_read
    and proc_name_exists
    ...
    and not user_known_read_sensitive_files_activities
  output: >
    Sensitive file opened for reading by non-trusted program (user=%user.name user_loginuid=%user.loginuid program=%proc.name
    command=%proc.cmdline file=%fd.name parent=%proc.pname gparent=%proc.aname[2] ggparent=%proc.aname[3] gggparent=%proc.aname[4] container_id=%container.id image=%container.image.repository)
  priority: WARNING
  tags: [filesystem, mitre_credential_access, mitre_discovery]

Some key parts in the condition are:

sensitive_files: Is a macro that defines sensitive files and directories.
open_read: Is a macro that defines read related syscall events (e.g., evt.type=openat).
We include exceptions like user_known_read_sensitive_files_activities.

In AuditD, we could create a new rule, like:

auditctl -a always,exit -F arch=b64 -S openat -F path=/etc/shadow -F perm=r -F key=read_etc_shadow

Both Falco and AuditD have the right filters to detect processes reading the sensitive /etc/shadow file.

Filtering network connections to malicious IPs

However, that’s not the case for other scenarios. Let’s cover one example where Falco rules are more fine grained than AuditD.

In our next scenario, we’ll write a rule to detect any network traffic to known C2 (Command and Control) servers.

In Falco’s out-of-the-box rules, we have:

# the list is subject to be updated
- list: c2_server_ip_list
  items: [...]
- rule: Outbound Connection to C2 Servers
  desc: Detect outbound connection to command & control servers
  condition: outbound and fd.sip in (c2_server_ip_list)
  output: Outbound connection to C2 server (command=%proc.cmdline connection=%fd.name user=%user.name user_loginuid=%user.loginuid container_id=%container.id image=%container.image.repository)
  priority: WARNING
  tags: [network]

In the rule condition above:

The outbound macro is used to detect that there is an outgoing network connection syscall event, using filters like evt.type (e.g,. connect, sendto), fd.typechar (e.g., IPv4), and so on. You can check the macro definition for more details.
The fd.sip filter retrieves the server IP address from the syscall event like connect and sendto.
Then, the Falco rule compares the IP address against the ones defined in c2_server_ip_list.
If there is a match, Falco will send an alert.

This is how filters work to create a fine grained detection rule. Not only for all system activities, like file read/write, or all of the network connection activities. Filters in Falco are letting you detect a specific IPv4 connection to a C2 server.

Check the full list of filters that can be used to create Falco rules, which include filters that understand the cloud native context, like container image repositories, or Kubernetes namespaces.

In AuditD, it’s unfortunately impossible to create a similar rule. The best we can do is create a rule to detect all kinds of network related syscall events (e.g., socket, connect):

auditctl -a always,exit -F arch=b64 -S socket -F success=1 -F a0=2 -k socket_activity

With this rule, we are filtering out all successful IPv4 connections (via filters a0=2 and success=1). All network traffic will be recorded, including the legit connections like DNS. In order to filter out/detect the exact event (to a particular IP address), you will also need to use tools like ausearch.

Cloud native context

More and more applications run as containers and are deployed in Kubernetes clusters. It is important for a HIDS tool to understand the cloud native context, and allow you to build detection rules based on it.

This is where AuditD falls short compared to Falco.

For example, in graboid, a crypto-worm attack, one of the significant IOCs is running the docker client inside a container, which is rare in production systems. However, running the docker client at the host level may be a legitimate operation.

Here is a Falco rule written specifically to detect such an IOC:

- rule: The docker client is executed in a container
  desc: Detect a k8s client tool executed inside a container
  condition: spawned_process and container and not user_known_k8s_client_container_parens and proc.name in (k8s_client_binaries)
  output: "Docker or kubernetes client executed in container (user=%user.name user_loginuid=%user.loginuid %container.info parent=%proc.pname cmdline=%proc.cmdline image=%container.image.repository:%container.image.tag)"
  priority: WARNING
  tags: [container, mitre_execution]

In this rule’s condition:

The container filter determines that the rule will only be triggered when an event happens inside a container.
We then compare the proc.name against the k8s_client_binaries list, which includes docker.

With filters like container that understand the cloud native context, you can build the fine grained rules you need to secure your cloud native environment.

Overall, Falco supports more than 150 filters (more related to cloud native environments like Kubernetes and containers), while AuditD supports about 40. Still, it is worth mentioning that AuditD’s filters like effective UID/GID, filesystem UID/GID, and SELinux User/Role/Type are not yet available in Falco.

Event verbosity in Falco vs. AuditD

The more verbose a security event is, the faster your security team can respond to it. Some relevant information may be the filter that you used to create detection rules, and auxiliary attributes like host name, login user, and so on. That’s why verbosity is an important factor of any HIDS products’ detection capability.

In Falco, the output of an event is also defined by the user, who can leverage most of the filters supported in Falco. Most of the rules’ output include filters like command line, parent process name, file name, username, and login username. Take a look to the output field in this out-of-the-box rule:

- rule: Write below etc
  desc: an attempt to write to any file below /etc
  condition: write_etc_common
  output: "File below /etc opened for writing (user=%user.name user_loginuid=%user.loginuid command=%proc.cmdline parent=%proc.pname pcmdline=%proc.pcmdline file=%fd.name program=%proc.name gparent=%proc.aname[2] ggparent=%proc.aname[3] gggparent=%proc.aname[4] container_id=%container.id image=%container.image.repository)"
  priority: ERROR
  tags: [filesystem, mitre_persistence]

If we were to trigger this rule by, let’s say, executing touch /etc/hello_falco, you would get all of this information:

23:36:25.600378940: Error File below /etc opened for writing (user=root command=touch_link /etc/hello_falco parent=bash pcmdline=bash file=/etc/hello_falco program=touch gparent=sudo ggparent=bash gggparent=sshd container_id=host image=<NA>)23:36:25.600378940: Error File below /etc opened for writing (user=root command=touch /etc/hello_falco parent=bash pcmdline=bash file=/etc/hello_falco program=touch gparent=sudo ggparent=bash gggparent=sshd container_id=host image=<NA>)

In comparison, the same AuditD rule can be written as following:

auditctl -w /etc/ -p aw -k write_below_etc

Unlike Falco, there is no way for users to define the output fields. And, if we were to trigger this rule by executing touch_link /etc/hello, the audit event would look like the following:

node=172.31.91.221 type=SYSCALL msg=audit(1607709712.085:1465620): arch=c000003e syscall=257 success=yes exit=3 a0=ffffff9c a1=7ffc3a4647cc a2=941 a3=1b6 items=2 ppid=21183 pid=21409 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts2 ses=24151 comm="touch_link" exe="/bin/touch" key="write_below_etc"^]ARCH=x86_64 SYSCALL=openat AUID="ubuntu" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root"
node=172.31.91.221 type=CWD msg=audit(1607709712.085:1465620): cwd="/home/ubuntu/auditd"
node=172.31.91.221 type=PATH msg=audit(1607709712.085:1465620): item=0 name="/etc/" inode=206 dev=103:01 mode=040755 ouid=0 ogid=0 rdev=00:00 nametype=PARENT cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0^]OUID="root" OGID="root"
node=172.31.91.221 type=PATH msg=audit(1607709712.085:1465620): item=1 name="/etc/hello" inode=90 dev=103:01 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0^]OUID="root" OGID="root"
node=172.31.91.221 type=PROCTITLE msg=audit(1607709712.085:1465620): proctitle=2E2F746F7563685F6C696E6B002F6574632F68656C6C6F

This output contains three types of messages (type=):

SYSCALL: Event triggered to record a system call record.
CWD: Event triggered to record the current working directory.
PATH: Event triggered to record the path information.
PROCTITLE: Gives the full command line that triggered this Audit event, and is triggered by a system call to the kernel.

Some highlights on what information is included, and how it is distributed:

The SYSCALL type event in AuditD doesn’t include the filename that was created but the PATH type event does.
The PATH type event also includes some useful information like object GID (OGID) and capabilities that Falco doesn’t.
Similarly, the SYSCALL type event doesn’t record the entire command line but the PROCTITLE type event does.
One advantage of AuditD is that it provides both the actual file and the symlink file when a symlink is executed. In the output above, the comm field contains the symlink name touch_link while the exe field contains the actual binary file name /bin/touch.

Note that the information in the PROCTITLE type event is encoded and it has to be decoded via ausearch, as following:

node=172.31.91.221 type=PROCTITLE msg=audit(12/11/20 17:42:56.734:1465459) : proctitle=./touch_link /etc/hello
...
node=172.31.91.221 type=SYSCALL msg=audit(12/11/20 17:42:56.734:1465459) : arch=x86_64 syscall=openat success=yes exit=3 a0=0xffffff9c a1=0x7fff9a49c7cc a2=O_WRONLY|O_CREAT|O_NOCTTY|O_NONBLOCK a3=0x1b6 items=2 ppid=18991 pid=20989 auid=ubuntu uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=pts0 ses=24148 comm=touch_link exe=/bin/touch key=write_below_etc

The proctitle value 2E2F746F7563685F6C696E6B002F6574632F68656C6C6F has been decoded as ./touch_link /etc/hello.

The full list of event fields (as output) can be found in the RHEL documentation.

Overall, both Falco and AuditD have verbose output:

Falco allows users to define what to output when a rule gets triggered, and each rule will generate only one single event.
AuditD doesn’t support customized output, and generates multiple records (of different types) for a single syscall event. AuditD output contains rich information about the object (e.g., inode, GID). However, some critical information like full command line (and IP address) need to be decoded via ausearch.

Resource consumption in Falco vs. AuditD

As we teased in this article’s introduction, the resource consumption of a detection tool can affect the performance of the workloads running on the detected host.

Keep in mind that to have a proper idea of the effect in your systems, you should evaluate resource consumption for each concrete use case. Depending on how detection rules are defined, they may generate too many false positive events, and the computational resource consumption can hike as a side effect. Carefully writing the detection rules will trigger few false positive events, and the resource consumption should be stable and low.

Though Falco provides a decent amount of detection rules out-of-the-box, it’s hard to find the AuditD rules that have the equal detection capability (there are quite a handful Falco detection rules for containers). So in our comparison test, we decided to go a bit extreme: Stress the system with a single operation that triggers both the Falco rule and the AuditD rule (this rarely happens in production environments).

Below is the test script that we used to trigger both AuditD and Falco rules:

#!/bin/bash
# infinite loop
while [ 0 -eq 0 ]
do
	touch /etc/hello
done

Given a pressure like this, both AuditD and Falco perform decently well.

AuditD occupies around 15% of an 8-core node CPU:

And the CPU usage for Falco is around 9%:

Overall, both Falco and AuditD performed OK within an extreme stress test, and no syscall events were dropped. While Falco consumed less CPU than AuditD in this particular test, note that this is not a comprehensive benchmark.

Event Forwarding in Falco vs. AuditD

As important as generating security events, it is to take action over them, whether it be automatically killing a process or notifying the appropriate team.

In this section, we’ll cover how the triggered events can be consumed or exported. This can affect what tools your security team can use to analyze security events. As they may have different preferences over those tools, we consider this a usability factor to evaluate in a detection product.

There are multiple ways to forward Falco events remotely:

Program output: Launches a third-party program to send Falco events (e.g., using curl to send Falco events to a Slack channel).
HTTP output: Sends Falco events to an HTTP endpoint.
Grpc server: Sets up a Grpc server that could listen on either unix socket or network so that Falco events can be consumed programmatically.
Falco Sidekick: A facilitating tool to forward Falco events to the most popular events hub or message bus, like ElasticSearch, Kafa, GCP PubSub, Influxdb, NATS, and more.

Similarly, AuditD supports several ways to forward audit events:

Remote syslog: audispd’s plugin, audisp-remote can be configured to forward audit events to a remote syslog server.
Elasticsearch: Filebeat provides an AuditD module to forward audit events to Elasticsearch.

Note that only the raw audit events will be forwarded. Encoded fields will still need to be decoded to become meaningful before consumed by events hub.

Overall, both Falco and AuditD support events forwarding. With the help of Sidekick, a tool created by Thomas Labarussias, consuming Falco events is considerably easier.

Conclusion

Falco and AuditD are powerful HIDS tools. They both rely on the syscalls to detect intrusion, but their approaches on creating the rules and outputting the events data have huge differences.

Besides the raw capabilities, also remember to consider other factors when choosing a HIDS tool, like the skill set in the team or the long-term security roadmap.

We hope this HIDS analysis framework helps you choose the right HIDS tool.

To learn more about Falco, please check out the Falco repo and Falco website. Or join our CNCF Falco slack channel. Also, if you would like to contribute to the Falco project, take a look to the Contributor page with tips on how to get started.