“Chain”ging the Game - how runtime makes your supply chain even more secure

There is a lot of information out there (and growing) on software supply chain security. This info covers the basics around source and build, but does it cover all of your full software supply chain lifecycle? Is your build env at runtime protected? Is your application post deploy protected at runtime?

This article will not only discuss what these concepts are, but provide additional discussions around the following:

What are Software Supply Chain attacks, SBOM, and the executive order?
Falco and Sysdig Secure capabilities that will have the most impact on securing your software supply chain.
Tips to think about implementing protection in your software supply chain so that you and your teams are better equipped to deal with them.

Read on brave reader…

What is software supply chain attack

A software supply chain attack occurs when a cyber threat actor infiltrates a software vendor’s network and employs malicious code to compromise the software before the vendor sends it to their customers. The affected software then compromises the customer’s data or system.

What is software supply chain security?

Supply chain security activities aim to enhance the security of the supply chain.

What is the executive order?

The Executive Order (EO) charges multiple agencies – including NIST– with enhancing cybersecurity through a variety of initiatives related to the security and integrity of the software supply chain.

Section 4 of the EO directs NIST to solicit input from the private sector, academia, government agencies, and others, and to identify existing – or develop new – standards, tools, best practices, and other guidelines to enhance software supply chain security.

Those guidelines are to include:

Criteria to evaluate software security.
Criteria to evaluate the security practices of the developers and suppliers themselves.
Innovative tools or methods to demonstrate conformance with secure practices.

What is an SBOM?

A software bill of materials (SBOM) is a list of components in a piece of software.

Software vendors often create products by assembling open source and commercial software components. The SBOM describes the components in a product. It is analogous to a list of ingredients on food packaging. Where you might consult a label to avoid foods that may cause an allergic reaction, SBOMs can help companies avoid consumption of software that could harm their organization.

The tl/dr:

SBOMs are important for security.
But you can only rely on an SBOM if you can trust the process that generated it.
This means you need a secure runtime for your build environment. We discuss this in detail below.

Falco, the cloud-native runtime security project, is the de facto Kubernetes threat detection engine. Falco was created by Sysdig in 2016 and is the first runtime security project to join CNCF as an incubation-level project. Falco detects unexpected application behavior and alerts on threats at runtime. Let’s discuss how Falco is able to do this.

Falco uses system calls to secure and monitor a system, by:

Parsing the Linux system calls from the kernel at runtime.
Asserting the stream against a powerful rules engine.
Alerting when a rule is violated.

Falco provides over 120 out-of-the-box rules in order to protect you and your software supply chain at runtime. The practical examples are numerous, however, specific to Supply Chain, there are rules that address issues we’ve recently seen with attacks.

Attack vector: Control plane/builder component runtime security

We have seen attackers successfully infiltrate for months via a lowest common denominator, a build machine or build pod. We described an attack as one that involves an attacker dropping a malicious set of code into the build and/or container while it’s running, connecting to an external network, and sending key company assets literally unchecked…. (SCARY!) These machines or processes historically have not been appropriately locked down nor had any degree of audit enabled to better understand what anomalous behavior is occurring.

This Outbound SSH Connection rule is one of the OOTB (out-of-the-box) rules that will detect any outbound connection to help identify the scenario attack vector we mention above.

- rule: Outbound SSH Connection
desc: Detect Outbound SSH Connection
condition: >
((evt.type = connect and evt.dir=<) or
(evt.type in (sendto,sendmsg))) and ssh_port
output: >
Outbound SSH connection (user=%user.name server_ip=%fd.sip server_port=%fd.sport client_ip=%fd.cip)
priority: WARNING
tags: [network]

Below is an example of file integrity mon itoring, if an attacker drops a malicious payload to a sensitive directory or that process does something anomalous on the builder machine and/or the pod. Falco detects this and can notify the team instantly through many methods (Falco Sidekick has over 40 outputs to choose from).

These rules, in particular, could address specific file access and integrity that could plague a build machine or build pod/container

Detect New File: Detects when a new file is created.
Detect New Directory: Detects when a new directory is created.
Detect File Permission or Ownership Change: Detects file permissions or ownership change.
Detect Directory Change: Detects directory changes, including mkdir, rmdir, mvdir, mv.
Kernel Module Modification: Detects kernel modules changes via modprobe or insmod.
Node Created in Filesystem: Detects when a node is created via mknod.
Listen on a New Port: Detects when a new port is listening.

These rules were community created and added to the Security Hub.

Sysdig Secure

The key to securing your supply chain is to block containers from the build process, should they not comply – or there are specific things in the image itself that aren’t compliant – with your internal policies.

This can be done via image scanning integrations with your CI/CD process or build software (e.g., Jenkins, Bamboo, Tekton).

Sysdig Secure embeds security and compliance into the build, run, and respond stages of the container and Kubernetes lifecycle.

A core component of this for runtime protection is Falco!

Here is an example, via the Sysdig Secure UI, where a “write to below root” is detected. Note that it includes the contextual detail of where it occurred from the cluster name, all the way down to the pod it happened on… to the process… to the syscall.

The Needle in the proverbial haystack found!

Sysdig Secure also provides a capability called runtime profiling which will look at network, process, file system, and syscalls in a container at the speed of the syscall itself!

Remember when we discussed attackers? In this case, if Sysdig runtime profiling sees a deviation, you can create an actionable response and have a full forensic path to all commands run to better understand what the attacker did.

Overall Tips to up your Software Supply “Chain Game”

To summarize, here are some tips you may use as a checklist:

Create a culture of security for not only security professionals, but also developers and operators. Reward the team for being security focused and diligent in patching dependencies.
Sign your software via cosign (sigstore).
Implement hardware level security (tpm and beyond!).
Ensure your systems are patched and control plan components are up to date (CIS benchmarking capabilities).
Ensure pipeline security.
Implement runtime security (Falco/ Sysdig Secure) for your control plane/build servers, as well as for your running applications.

If you want to learn more, you may be interested in these links:

And related to Falco and Sysdig:

Conclusion

The threat of software supply chain attacks is real. Without solutions in the OSS/Commercial world, you run the risk of being another compromised organization and adding to US$5.2 trillion (per Accenture) lost in cybercrime due to misconfiguration.

Think in broad strokes, implement organization-wide culture and tools that encompasses pipeline security, but also underlying builder and control plane runtime protection.

Special thanks to Dan Lorenc and Stefano Chierici for providing feedback on this article.

Get involved!

If you would like to find out more about Falco:

Get started at Falco.org.
Check out the Falco project in GitHub.
Become involved with the Falco community.
Meet the maintainers on the Falco Slack.
Follow @falco_org on Twitter.

At Sysdig Secure, we extend Falco with out-of-the-box rules along with other open source projects, making them even easier to work with and manage Kubernetes security. Register for our Free 30-day trial and see for yourself!

“Chain”ging the Game – how runtime makes your supply chain even more secure