Case Studies

Blog Post

Implementing Docker/Kubernetes runtime security.

Sysdig Falco performs live monitoring of the behavior of your Docker containers and Kubernetes pods at runtime. Using Falco you can create a Docker runtime security policy to detect attacks and anomalous activity on production environments, in real-time, so you can react to unknown and 0-day vulnerabilities, attacks caused by weak or leaked credentials or compliance breaches.

Sysdig Falco has its own security policy rules syntax, based on Sysdig filtering language which is very similar to tcpdump syntax, self-descriptive and fast to learn. In addition, it is really easy to install Falco in any container orchestration platform. If you are looking for a commercial product instead, check out Sysdig Secure, built on top of Sysdig Falco open-source technology.

Implementing runtime security policies requires to know exactly how your containers and applications behave. Since most of us build our Cloud Native applications using open-source components, we wanted to help you in this process providing base security profiles for a growing list of the most popular container images, through a GitHub repository: falco-extras, including:

  • Kubernetes 1.10 cluster components:
    • apiserver
    • controller-manager
    • kube-dns
    • kube-scheduler
    • dashboard
  • Google Kubernetes Engine components
  • Apache
  • Consul
  • ElasticSearch
  • etcd
  • Fluentd
  • HAproxy
  • MongoDB
  • Nginx
  • PHP-FPM
  • PostgreSQL
  • Redis
  • Traefik

Previously, we did cover how to setup a few runtime security policy examples with Sysdig Secure, now we will see how to do this with open-source.

What can you find in a Docker runtime security policy?

We already showed how to implement runtime security and harden/secure your Kubernetes kube-system namespace, now we will see how to expand this to the rest of your Dockerized applications.

For each of your container images, you can create a whitelisting policy for:

  • Which processes / commands are allowed to run inside the Docker container
  • Processes that can start an outbound connection or accept an inbound connection
    • Including allowed listening ports or remote ports
  • Files and directories that can be read
    • By which processes
  • Files and directories that can be written
    • By which processes
  • Even the set of Linux system calls that your container processes are expected to use!

Basically, we are creating a tightly constrained execution environment. Anything else, any unexpected behavior will raise an alarm.

For example, this is the default Docker runtime security policy for a Nginx container:

And of course, that’s on top of many other suspicious behaviors that Falco will detect by default:

  • A shell is run inside a container
  • A container is running in privileged mode or is mounting a sensitive path like /proc from the host
  • A server process spawns a child process of an unexpected type
  • Unexpected read of a sensitive file (like /etc/shadow)
  • A non-device file is written to /dev
  • A standard system binary (like ls) makes an outbound network connection
  • And many more policy examples

How to customize these rules to create your runtime security profiles and policies

Using these default Falco runtime security rulesets will hopefully save you a lot of time writing boilerplate. However, bear in mind that every version of a Docker container image, even for the same microservice, it’s probably going to have some differences: user-defined data directories, different binary paths, scripts that need to access some external port or device, etc.

Thus, you will need to adapt the templates to your specifics before actually using them.

How to apply runtime security policy to specific container images

In this generic rule we just used we targeted any container image which name contains “nginx”:

- macro: app_nginx
  condition: container and container.image contains "nginx"

But you can be much more specific, using the unique hash code of the image you are deploying:

- macro: app_nginx
  condition: container and container.image contains "73acd1f0cfadf6f56d30351ac633056a4fb50d455fd95b229f564ff0a7adecda"
Falco can also leverage metadata from Kubernetes, allowing you to create more accurate and context-specific rulesets that make use of Pod, Deployment name or Namespace name.

You have some example rules of Falco using Kubernetes metadata in Detecting Cryptojacking with Sysdig’s Falco:

- macro: node_app_frontend
  condition: k8s.ns.name = node-app and k8s.pod.label.role = frontend and k8s.pod.label.app = node-app

- rule: Detect crypto miners using the Stratum protocol
  desc: Miners typically specify the mining pool to connect to with a URI that begins with 'stratum+tcp'
  condition: node_app_frontend and spawned_process and container.id != host and proc.cmdline contains stratum+tcp
  output: Possible miner ran inside a container (command=%proc.cmdline %container.info)
  priority: WARNING

Detect not allowed or forbidden Docker container images

Additionally, you can create your own image whitelist, to make sure no unknown or not allowed container image is ever run on your hosts / cluster:

- list: container_image_whitelist
  items: ["sha256:8ac48589692a53a9b8c2d1ceaa6b402665aa7fe667ba51ccc03002300856d8c7", "sha256:f3fcd0775c4e15d4b73d2b62f60efb1872bf4d3328be52a0fc9a2b0951bbcc1e"]

- rule: Unknown container image running
  desc: There is a container running that doesn't match any of the whitelisted sha256 codes
  condition: container and not container.image in (container_image_whitelist)
  output: Unknown container (command=%proc.cmdline pid=%proc.pid file=%fd.name %container.info image=%container.image)
  priority: WARNING

Fine tune your runtime security: Sysdig + Sysdig Inspect (also forensics and post-mortem analysis!)

If you want a fine-grained inspection tool coupled together with a nice GUI, without leaving FOSS territory, you can always use sysdig together with Sysdig Inspect.

For example, if you want to record a live capture of every activity that mynginx _container is registering, just run:

# sysdig -pc -s 4096 container.name=mynginx -w nginxcap.scap

After a few seconds, you can directly open the capture file on your laptop using Sysdig Inspect to browse the container activity and figure out if there is any additional constraints you can add to your runtime security profile.

Docker Runtime Security Sysdig Inspect Dashboard

From here, I can easily browse the processes that the container was running:

Docker Runtime Security Sysdig Inspect Process

Or its network connections:

Docker Runtime Security Sysdig Inspect Network

And much more! File system activity, system calls… everything really! That’s why Sysdig Inspect is often used for troubleshooting (How to troubleshoot a Kubernetes CrashLoopBackOff) or forensics too.

Help us build the Docker and Kubernetes runtime security library!

We would love to hear about your experience implementing runtime security in Docker and Kubernetes. Are you using Sysdig Falco already? Or wondering how it compares with seccomp or SELinux/AppArmor? If you have created a set of profiles and are proud of them why not contributing back? Fork and send us Pull Requests to falco-extras!. Cannot use Falco because you are missing a specific feature or maybe you hit a bug? We also want to know! Find us on Sysdig Slack, #falco channel.




Get the Guide! Kubernetes Security

Best practices for securing Kubernetes in your organization

  • Understanding Kubernetes role-based access controls and transport layer security certificates
  • Implementing security at the pod level
  • Securing Kubernetes components
  • Hardening Kube-system components with security policies

Download your copy and learn best practices for Kubernetes Security

Get it at http://bit.ly/2xTn9vO

Share This

Stay up to date

Sign up to recieve our newest.

Related Posts

Getting Started Writing Falco Rules

Active Kubernetes security with Sysdig Falco, NATS, and Kubeless.

Kubernetes security guide.