Getting started with runtime security and Falco

Discover how to get started with Falco to overcome the challenges of implementing runtime security for cloud-native workloads.

If you are adopting containers and cloud, you are probably enjoying benefits like automated deployments and easier scalability. However, you may also find that when it comes to security, this is a whole new world with new rules, and traditional security tools struggle to keep up. As a new paradigm, cloud-native environments need new cloud-native tools.

Let’s focus on runtime security.

Containers are lightweight, and it’s common to run dozens of them on the same host machine. Containers are also black boxes, often invisible to traditional host-based tools.

The first struggle translates to: “How do I know what is happening during runtime inside these containers?” After all, how can you secure what you don’t see?

And if you detect suspicious activity, you may wonder: “How do I get the proper context to respond?” Like, in which container did the activity take place? What workload is it associated with? And who is the right team to respond to that event?

Cloud-native tools like Falco can see everything each individual container is doing and easily flag suspicious behavior, like “Hey, this redis container shouldn’t be opening connections outside the network” or “Why did a file change on this Apache server?”

Let’s briefly introduce you to runtime security, uncovering how Falco works its magic and how easy it is to get started with Falco.

What is runtime security?

You may find blocking vulnerabilities from reaching production quite reassuring. This kind of preventive discovery of vulnerabilities is usually achieved with image scanning tools.

If your scans look pretty clean, you might think you are safe. After all, “how are you going to be attacked if you didn’t deploy any vulnerable applications?”

Well, not all vulnerabilities are known. If a vulnerability is disclosed after your application is running in production, you may be at risk and not know it yet. Additionally, it’s not always possible to correct all vulnerabilities immediately, and you may need to closely monitor your known-vulnerable systems in the meantime. There’s a lot that can happen during runtime.

Focusing on container runtime security, several security threats, by their very nature, only manifest during runtime. This includes:

• Zero-day vulnerabilities – issues previously unknown to a software creator.
• Privilege escalation attempts.
• Bugs that cause erratic behavior or resource leaking.

Runtime security is one of the most complex aspects of container security because it involves multiple moving pieces, which can vary depending on the type of container application stack you use. In most cases, however, runtime security is based on securing the following:

• The container runtime: This is the process on the server that actually executes containers. You should ensure that your runtime software is up-to-date and patched against known security vulnerabilities.
  • Famous examples of container runtime are containerd, CRI-O, or Docker.
• The orchestrator: The container orchestrator deploys and manages containers. Most orchestrators offer a variety of tools to help restrict containers’ privileges and minimize security risks, but you should also use third-party monitoring and analysis tools to help detect security issues at the orchestrator level.
  • In terms of container orchestrator, Kubernetes is the most representative.
• Nodes: Nodes are the servers that host containers. You need to secure the host operating system, user accounts, networking configurations, and other resources in order to ensure that a breach at the node level doesn’t allow attackers to impact your entire environment.
• Cloud environments: You could deploy your container in your cloud platform. For example, in AWS Fargate. These platforms are starting to offer mechanisms like ptrace that allow you to export system calls, and use tools to monitor what happens inside your containers to avoid security incidents.

To secure all this stuff, the most popular OSS cloud and container security tool is Falco, the cloud-native runtime security project is the de facto Kubernetes threat detection engine. It is the first runtime security project to join the CNCF as an incubation-level project. Sysdig created Falco in 2016. It was designed to detect unexpected application behavior and alert on threats at runtime.

How does Falco work?

Falco works by analyzing kernel system calls to provide deep visibility into container, host, and cluster activity. It also taps into other data sources, such as Kubernetes API audit events. Equally important, Falco adds Kubernetes application context to its findings to help DevOps, security, and cloud teams understand exactly who did what and where.

To understand how Falco works, we should first take a look at its three main components:

• The userspace program handles signals, parses information from a Falco driver, and sends alerts. You can use the CLI tool to interact with Falco.
• The configuration defines how Falco is run, what rules to assert, and when to trigger alerts.
• A driver must be installed to run Falco. This is software that adheres to the Falco driver specification and sends a stream of system call information. Currently, Falco supports the following drivers:
  • (Default) Kernel module built on libscap and libsinsp C++ libraries
  • eBPF probe built from the same modules
  • Userspace instrumentation

One of the main things about Falco is that it is a monitoring-only tool. It can send alerts to one or more channels, but it cannot take any action by itself. Alert channel examples include:

• Standard Output
• A file
• Syslog
• A spawned program
• A HTTP[s] end point
• A client via the gRPC API

The list of possible Falco outputs is extended by another open source tool, Falcosidekick. If you would like to know more about this, take a look at Extend Falco outputs with Falcosidekick.

Falco uses rules to detect suspicious behavior under a given condition. For example, a bash in our container may be unusual, and you only need to focus on the condition that triggers the alert, in this case the proc.name is bash :

- rule: Detect bash in a container
  desc: You shouldn't have a shell run in a container
  condition: container.id != host and proc.name = bash
  output: Bash ran inside a container (user=%user.name command=%proc.cmdline %container.info)
  priority: INFO
Code language: Perl (perl)

We can see how easy it is to detect this kind of behavior, using simple language in the rule. If you want to know more about Falco rules, take a look at the documentation rules.

When this rule detects an event, if we have configured an output option, we will receive a message like this:

20:07:06.837415779: Notice A shell was spawned in a container with an attached terminal (user=root container=1dab04047700 shell=bash parent=runc cmdline=bash terminal=34816 container_id=1dab04047700 image=docker.io/falcosecurity/falco) container=1dab04047700
Code language: Perl (perl)

And that’s it! We are able to see runtime container activity and detect a potential security event.

How solid is the Falco project?

“Ok, ok, Falco is great” – I hear you say – “But how solid is this project?”

And this is a reasonable question. After all, no one wants to commit to a solution only to migrate to something else a short time later. So here are some interesting facts to put things in perspective.

Falco is an open source project under the CNCF umbrella, which is part of the Linux Foundation.

Falco is broadly adopted, and the project’s popularity has grown 4x since joining the CNCF. Falco images have been downloaded over 30 million times.

Additionally, the number of users following Falco’s github repo has doubled since incubation.

Other projects within Falco’s ecosystem are also thriving, like Falcosidekick, which was started by the community and just hit 3 million downloads..

Adoption is so ubiquitous that Falco is now part of the curriculum for the official CKS exam.

Falco’s technical architecture is carefully designed to meet the needs of modern environments. Due to the underlying eBPF technology, Falco barely has a performance impact, with almost half the CPU usage of its alternatives under extreme stress tests.

Falco is being deployed in high performance environments. One user, skyscanner, reported that “deploying Falco as a Daemonset in a K8s cluster hasn’t shown any negative effects in the services performance (believe me, we really tried to break it).”

It just takes a quick look at the Falco website to find big names on the list of end-users:

And for us, the icing on the cake is being recognized by Gartner. Among the reasons for using Falco, Gartner analysts note that “Falco enables detection of anomalous behavior in application deployments with Kubernetes context awareness.” [1]

Falco is a strong project with a thriving community, and you should consider it a serious contender for your runtime container security needs.

First steps with Falco

Installing Falco on your Linux machine can be as easy as installing the kernel modules, and then running:

apt-get install -y falco
Code language: Perl (perl)

That’s all.

However, in a cloud-native world, chances are that you either don’t have direct access to the host or that you want to deploy Falco the way you deploy the rest of your applications.

Let’s cover how to install Falco in a Kubernetes cluster using Helm, the package manager for Kubernetes, and how to customize the available rules.

Wait, I don’t have a Kubernetes cluster to test this on!

Worry not! We have you covered.

Follow the instructions to install Minikube, and you’ll get a single node Kubernetes cluster in a matter of minutes.

Remember to start Minikube with:

$ minikube start
Code language: Perl (perl)

And perform a quick test to confirm that it’s up and running:

$ minikube kubectl -- get nodes
NAME       STATUS   ROLES                  AGE     VERSION
minikube   Ready    control-plane,master   6m18s   v1.22.2
Code language: Perl (perl)

From now on, we will assume the kubectl client is installed locally, so our command would just become kubectl get nodes. Check the minikube kubectl documentation for instructions on how to create an alias to simplify these commands, or go here to install kubectl on your machine.

Let’s deploy Falco on Kubernetes with Helm

For that, make sure you have Helm installed.

We will use the community supported chart to install Falco.

First, add the falcosecurity charts repository and update:

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update
Code language: Perl (perl)

Now, it’s time to install Falco using the helm install command:

helm install falco falcosecurity/falco
Code language: Perl (perl)

Wait a few seconds for the containers to start, identify the Falco Pod, and have a look at the container logs:

$ kubectl get pods
`NAME READY STATUS RESTARTS AGE
`falco-467d7 1/1 Running 0 4m13s
$ kubectl logs falco-467d7
…
Wed Nov 3 15:05:17 2021: Falco initialized with configuration file /etc/falco/falco.yaml
Wed Nov 3 15:05:17 2021: Loading rules from file /etc/falco/falco_rules.yaml:
Wed Nov 3 15:05:17 2021: Loading rules from file /etc/falco/falco_rules.local.yaml:
Wed Nov 3 15:05:17 2021: Starting internal webserver, listening on port 8765
…
Code language: Perl (perl)

Looks like Falco was deployed successfully!

Putting Falco to the test.

Now that Falco is deployed and running, let’s see if it can detect some suspicious activity.

As containers are not supposed to be interactively accessed, one of the biggest red flags of a compromised container is when a terminal shell spawns. There are legit reasons for this to happen, but they are usually limited to troubleshooting tasks.

Falco ships with rules enabled by default to detect such scenarios. In this case, the relevant rule is “Terminal shell in container.”

Let’s take the same falco Pod we deployed earlier, and let’s run some commands inside it:

$ kubectl exec -it falco-4gsbr -- /bin/bash
root@falco-4gsbr:/#
Code language: Perl (perl)

Note that this will work with any other Pod you have deployed, we are using the falco one, as it is the one we know for sure you have in your cluster 😉.

If we now check the falco logs, we’ll see a warning:

$ kubectl logs falco-4gsbr
…
16:37:58.199209616: Notice A shell was spawned in a container with an attached terminal (user=root user_loginuid=-1 k8s.ns=default k8s.pod=falco-4gsbr container=b1cfe2fbfcdd shell=bash parent=runc cmdline=bash terminal=34816 container_id=b1cfe2fbfcdd image=falcosecurity/falco) k8s.ns=default k8s.pod=falco-4gsbr container=b1cfe2fbfcdd
…
Code language: Perl (perl)

Notice how Falco provides contextual information about the pod and namespace where the event was triggered (k8s.ns=default k8s.pod=falco-4gsbr container=b1cfe2fbfcdd).

What’s next?

Now that Falco is working, there’s a lot of fun you can have.

For example, you can send alerts through a different channel than logs. For this effect, you may want to check out the falcosidekick project, which allows you to forward Falco events to many common services.

Continue your journey with:

• This Getting started with container runtime security using Falco webinar.
• The Falco community mantained hands-on labs.
• The Runtime Security and Falco 101 course by Sysdig.
• This Falco workshop, and these too.

And, if you want to get involved with the Falco project:

• Get started at Falco.org.
• Check out the Falco project on GitHub.
• Get involved with the Falco community.
• Meet the maintainers on the Falco Slack.
• Follow @falco_org on Twitter.

What about commercial products based on Falco?

Open source solutions require DIY effort by your team. You have full control of the tooling, but you are also responsible for setting up, configuring, and maintaining all of the components.

Teams that require quicker onboarding, lower management overhead, greater scalability, customer support, and other premium features may prefer a commercial product.

Commercial products can add value if they address multiple security use cases at once, instead of focusing only on runtime security.

For example, a commercial product can provide a single pane of glass where you can check for cloud misconfigurations, runtime security events, and image scanning results. Having all of this information in the same place will save you from jumping between different dashboards and will enable you to be more efficient in incident response and forensics.

Commercial tools aim not only to make it easier to secure your infrastructure, but also to help you keep track of your cloud security posture over time.

Many security products are built to help you meet compliance requirements right out of the box. By tapping into all the aspects of your applications lifecycle, commercial tools are able to tell you where you are falling out of compliance, and present you with remediation steps.

Read how Sysdig Secure extends Falco security by providing a comprehensive security workflow across the application lifecycle.

Conclusion

Falco is a solid open source solution to implement runtime security in containers and Kubernetes.

Falco’s cloud-native design enables it to provide the context you need to quickly and effectively resolve the security events in your modern application environments.

If you would like to find out more about Falco:

• Get started at Falco.org.
• Check out the Falco project on GitHub.
• Get involved with the Falco community.
• Meet the maintainers on the Falco Slack.
• Follow @falco_org on Twitter.

---

---

[1] Gartner, “Open-Source Options for Threat Detection and Incident Response”, Anna Belak and Eric Ahlm, 1 December 2020.