Runtime security for Azure Kubernetes Service (AKS) environments requires putting controls in place to detect unexpected and malicious behavior across your applications, infrastructure, and cloud environment.
Runtime threats include things like:
- Exploits of unpatched and new vulnerabilities
- Insecure configurations
- Leaked or weak credentials
- Unauthorized activity
Even if you’re taking advantage of tools like container image vulnerability scanning, Kubernetes pod security policies, and Kubernetes network policies with AKS, not every risk will be addressed. You still need mechanisms to confirm that the security barriers you’ve put in place are effective. Furthermore, having a last line of defense will help you to identify zero-day vulnerabilities and unexpected activity.
Why runtime security with AKS?
“Container security” is often thought of as being synonymous with image scanning and vulnerability management. The thinking is, if you’ve scanned your images for vulnerabilities, misconfigurations, and compliance violations early in the CI/CD pipeline, you will successfully prevent threats in your container and Kubernetes environment. Although image scanning is an important first step, additional security measures are needed. Runtime security, zero-trust networking, audit, forensics, and incident response should also be components of your secure DevOps workflow.
Several security threats, by their very nature, only manifest during runtime. This includes:
- Zero-day vulnerabilities – issues previously unknown to a software creator
- Privilege escalation attempts
- Bugs that cause erratic behavior or resource leaking
In this post, we’ll look at how you can implement runtime security detection using an open-source based approach with the Falco project. We’ll also discuss how Sysdig Secure builds on Falco to detect and alert on runtime threats at scale in AKS environments.
Falco: Open-source Kubernetes runtime detection
Falco, the cloud-native runtime security project, is the de facto Kubernetes threat detection engine. It is the first runtime security project to join the CNCF as an incubation-level project. Sysdig created Falco in 2016. It was designed to detect unexpected application behavior and alerts on threats at runtime.
Falco works by analyzing kernel system calls to provide deep visibility into container and environment activity. It also taps into other data sources such as Kubernetes API audit events. Equally important, Falco adds Kubernetes application context to its findings to help DevOps, security, and cloud teams understand exactly who did what — and where.
Why Falco for runtime security in Azure Kubernetes Service?
A weakness of commonly used vulnerability and malware detection approaches is the reliance on prior knowledge of possible threats. Signature-based approaches, for example, must list each possible exploit, vulnerability, or attack in some way (e.g., malware signatures). This means the detection tool is engaged in a never-ending game of catch up with a constant stream of new threats.
Behavioral approaches, in contrast, look at what is happening on a system. In other words, they will detect the things that a user or attacker does once they have access to a system.
Falco detects unexpected or malicious application and environment behavior using security rules. Rules are available from the community and you can also write your own detection rules using Falco’s flexible rules engine. You can then plug Falco into your security detection and response workflows to identify and alert on runtime threats.
A Falco rules file is a YAML file containing three types of elements:
| Element | Description |
| Rules | Conditions under which an alert should be generated. A rule is accompanied by a descriptive output string that is sent with the alert. |
| Macros | Rule condition snippets that can be reused inside rules and even other macros. |
| Lists | Collections of items that can be included in rules, macros, or other lists. Unlike rules and macros, lists cannot be parsed as filtering expressions. |
Falco for threat detection in AKS
Runtime threat detection built on open source Falco helps you identify and block suspicious activity and anomalies in your container environment. Here are a few examples:
Terminal shell in a container
Falco detects command-line Interface execution (terminal shell) in a running container in violation of a configured policy. This could indicate an attacker attempting to manipulate the system, download malware, or initiate other malicious activity. Below is an example of a Falco rule that detects this threat to help you better meet compliance, auditing, and intrusion detection requirements.
- rule: Terminal shell in container
desc: >
A shell was used as the entrypoint/exec point into a container with an
attached terminal.
condition: >
spawned_process and container
and shell_procs and proc.tty != 0
and container_entrypoint
output: >
A shell was spawned in a container with an attached terminal
(user=%user.name %container.info shell=%proc.name parent=%proc.pname
cmdline=%proc.cmdline terminal=%proc.tty container_id=%container.id
image=%container.image.repository)
priority: NOTICE
tags: [container, shell, mitre_execution]
Attempt to make an unusual outbound network connection
If a standard system binary like ls or ps makes an outbound TCP connection – something is wrong. A likely explanation is that the host has been rootkit’ed. Ideally you’d want to detect a rootkit installation when it happens (possibly using the kinds of rules described above). However, it remains important to defend in depth and detect behaviors that can happen after an attack is underway.
In this case, here is a Falco rule that helps capture suspicious connections in AKS environments:
- rule: Unexpected outbound connection destination
desc: >
Detect any outbound connection to a destination outside of an allowed set
of ips, networks, or domain names
condition: >
consider_all_outbound_conns and outbound and not
((fd.sip in (allowed_outbound_destination_ipaddrs)) or
(fd.snet in (allowed_outbound_destination_networks)) or
(fd.sip.name in (allowed_outbound_destination_domains)))
output: >
Disallowed outbound connection destination
(command=%proc.cmdline connection=%fd.name user=%user.name
container_id=%container.id image=%container.image.repository)
priority: NOTICE
tags: [network]
MITRE ATT&CK framework detections
Falco detects system events that seem abnormal based on the adversary tactics and techniques as defined by the MITRE ATT&CK framework. From this information, activities deemed to be a threat or anomalous can be remediated by isolating the involved pods and containers. The below example is for privilege escalation detection in AKS:
- rule: Launch Privileged Container
desc: >
Detect the initial process started in a privileged container. Exceptions are
made for known trusted images.
condition: >
container_started and container
and container.privileged=true
and not falco_privileged_containers
and not user_privileged_containers
output: >
Privileged container started
(user=%user.name command=%proc.cmdline %container.info
image=%container.image.repository:%container.image.tag)
priority: INFO
tags: [container, cis, mitre_privilege_escalation, mitre_lateral_movement]
How Sysdig Secure extends runtime security in AKS
Sysdig Secure is built on the open-source Falco detection engine. It extends what Falco has to offer to provide comprehensive security across the container and Kubernetes lifecycle. In addition, Sysdig Secure features an easy-to-navigate user interface to simplify operating at cloud-scale.
Runtime security policies for your container, Kubernetes, and cloud environments are available out-of-the box. These policies are built on Falco rules and can be simply toggled on or off within the Sysdig Secure user interface:
Each policy can be tweaked to apply to a specific part of your deployments, to take specific actions, and to customize where to send notifications.
In the interface, you can view the Falco syntax to understand the make-up of the Falco rule on which the policy is built. Plus, Sysdig Secure makes it easy to add additional detections into an existing policy by selecting from the included rules library.
Monitoring and responding to security events on AKS
Using Sysdig Secure, DevOps and security operations teams can view a summary of triggered events based on activity across your environment. (These events can also be forwarded to your favorite SIEM). You can drill into policy violations to get additional context and access detailed records to aid with forensics and incident response.
For example, an attempt to read sensitive files (e.g. files containing user, password, or authentication information) will trigger an event alert. You’ll then be able to identify where the issue is happening, see what was accessed, understand the impacted containers or services, and be able to further investigate the issue with full context.
In addition to what we’ve highlighted above, to help you further manage and automate security for your AKS environment, Sysdig Secure provides:
- Remediation with automated response actions to kill, stop or pause containers
- Automated image profiling, a flexible policy editor, and centralized management
- Image scanning for CI/CD pipelines and registries to spot and block vulnerabilities in development and production
- Detailed system call captures that support forensics, incident response, and audit
- Tooling and visualizations that simplify network policy creation with Kubernetes network security policies
- Checks and policies mapped to compliance standards to simplify compliance validation
You can check out more about this solution on the Sysdig Secure product page.
Conclusion
Runtime security for containers, Kubernetes, and cloud services provides an important protection for modern application environments built on Microsoft Azure. Open source Falco gives users the ability to filter and detect malicious activity with community sourced and custom rules. Sysdig Secure takes this functionality a step further, simplifying runtime security detection and response at scale on Azure.
To learn more about security for containers on Azure, grab a copy of our Azure security guide.
If you would like to find out more about Falco:
- Get started in Falco.org
- Check out the Falco project in GitHub
- Get involved in the Falco community
- Meet the maintainers on the Falco Slack
- Follow @falco_org on Twitter