Comprehensive best practices for container security

But aren’t containers meant to be safe and isolated? Well, kind of.

For example, an exploitable vulnerability inside a container, combined with exposed metadata and a wrong credentials configuration, can compromise your whole cloud infrastructure. As described in our Cloud lateral movement post, a hacker can use this chain of exploits and wrong configurations to run crypto mining applications in your cloud account.

Containers were designed as a distribution mechanism for self-contained applications, allowing them to execute processes in an isolated environment. For isolation purposes, they employ a lightweight mechanism using kernel namespaces, removing the requirement of several additional layers in VMs, like a full operating system, CPU and hardware virtualization, etc.

The lack of these additional abstraction layers, as well as tightly coupling with the kernel, operating system, and container runtime, make it easier to use exploits to jump from inside the container to the outside and vice versa.

Container security best practices don’t just include the delivered applications and the securing containers image itself, but also the full component stack used for building, distributing, and specially executing the container.

This includes:

The host or VM

• The container runtime
• Cluster technology
• Cloud provider configuration
• And more.

Container security can be applied at each of the different phases: development, distribution, execution, detection and response to threats.

Let’s dive into the interesting details, breaking down the general ideas into 17 concrete container security best practices that you can apply in your DevOps workflows.

A complex stack

Containers’ success is often fueled by two really useful features:

• They are a really convenient way to distribute and execute software, as a self-contained executable image which includes all libraries and dependencies, while being much lighter than classical VM images.
• They offer a good level of security and isolation by using kernel namespaces to execute processes in their own “jail”, including mounts, PID, network, IPC, etc., and also resource limiting CPU usage and memory via kernel cgroups. Memory protection, permission enforcement, etc. are still provided via the standard kernel security mechanisms.

The container security model might be enough in most cases, but for example, AWS adds additional security for their serverless solution. It does so by running containers inside Firecracker, a micro virtual machine that adds another level of virtualization to prevent cross-customer breaches.

Does this mean containers are not safe?

You can see it as a double-edged sword.

An application running inside a container is no different than an application running directly in a machine, sharing a file-system and processes with many other applications. In a sense, they are just applications that could contain exploitable vulnerabilities.

Running inside a container won’t prevent this, but will make it much harder to jump from the application exploit to the host system, or access data from other applications.

On the other hand, containers depend on another set of kernel features, a container runtime, and usually a cluster or orchestrator that might be exploited too.

So, we need to take the whole stack into account, and we can apply container security best practices at the different phases of the container lifecycle. Let’s reflect these two dimensions in the following diagram and focus on the practices that can be applied in each block:

	Prevent	Protect	Detect	Respond
Code	(1) Code Scanning(2) Dependencies			
CI/CD	(1) Code Scanning(2) Dependencies(3) Image Scanning			
Registry	(3) Image Scanning(4) Signing	(10) Verify signature(12) Vuln. Mgmt.		
Cloud	(5) Configuration(6) IaC Scanning	(9) Security Groups(9) Network rules	(13) Event and logs(14) Isolate, investigate, forensics 	(15) Fix configuration (16) Fix configuration
Host	(5) Configuration(7) Host Scanning	(5) Configuration(12) Vuln. Mgmt.(9) Harden security	(13) Events and logs(13) Syscall audit	(14) Isolate, investigate, forensics(15) Fix configuration (16) Patch vulnerabilities
Kernel	(7) Host Scanning	(12) Vuln. Mgmt.	(13) Syscall audit	(16) Patch vulnerabilities
Container Runtime	(8) Admission Controller(5) Configuration	(5) Configuration(9) Network policies(12) Vulnerabilities Management(10) Verify signature	(3) Image Scanning(13) Events and logs	(15) Fix configuration (16) Patch vulnerabilities
Live container		(11) User Privileges Network Storage(12) Vulnerabilities Management	(13) Logs(13) Syscall audit	(16) Update base image Update OS packages Update dependencies Fix application

There will be cases like the serverless compute engine ECS Fargate, Google Cloud Run, etc., where some of these pieces are out of our control, so we work on a shared responsibility model.

• The provider is responsible for keeping the base pieces working and secured
• And you can focus on the upper layers.

Prevention: 8 steps for shift-left container security

Before your application inside a container is executed, there are several places where you can start applying different techniques to prevent threats from happening.

The software supply chain has become a top entry point for cyber threats in recent years. Modern businesses routinely depend on third-party applications, dependencies, and packages, and there have been a number of major security incidents where threat actors exploited the supply chain.

Prevention and applying container security as early as possible is key and will save you a lot of trouble, time, and money with minimal effort if you apply some good practices during the development and distribution of the container images.

1. Integrate Code Scanning at the CI/CD Process

Security scanning is the process of analyzing your software, configuration or infrastructure, and detecting potential issues or known vulnerabilities. Scanning can be done at different stages:

• Code
• Dependencies
• Infrastructure as code
• Container Images
• Hosts
• Cloud configuration
• … and more

Let’s focus on the first stage: code. Before you ship the application or even build your application, you can scan your code to detect bugs or potentially exploitable code (a new vulnerability).

For application code, you can use a tool like Snyk Code or Checkmarx One for SAST (static application security testing) to scan for vulnerabilities and other issues. There are also open source SAST tools like sonarqube, which provide vulnerability scanners for different languages, gosec for analyzing go code and detecting issues based on rules, linters, etc.

You can run them at the developer machine, but integrating code scanning tools at the CI/CD process can make sure that a minimum level of code quality is assured. For example, you can block pull requests by default if some checks are failing.

A Github Action running gosec:

name: "Security Scan"
on:
  push:
jobs:
  tests:
    runs-on: ubuntu-latest
    env:
      GO111MODULE: on
    steps:
      - name: Checkout Source
        uses: actions/checkout@v2
      - name: Run Gosec Security Scanner
        uses: securego/gosec@master
        with:
          args: ./...
Code language: Perl (perl)

2. Reduce external vulnerabilities via dependency scanning

Only very minimal and toy applications are free of third-party libraries or frameworks. But reusing code from external dependencies means you will be including bugs and vulnerabilities from these dependencies as part of your application. Dependency scanning should be included as a best practice in any application build process.

Tools like Snyk that offer SCA (software composition analysis) capabilities can be used to identify third-party dependencies. Package management tools, like npm, maven, go, etc., can match vulnerability databases with your application dependencies and provide useful warning.

For example, enabling the dependency-check plugin in Maven requires just adding a plugin to the pom.xml:

<project>
    ...
    <build>
        ...
        <plugins>
            ...
            <plugin>
              <groupId>org.owasp</groupId>
              <artifactId>dependency-check-maven</artifactId>
              <version>6.2.2</version>
              <executions>
                  <execution>
                      <goals>
                          <goal>check</goal>
                      </goals>
                  </execution>
              </executions>
            </plugin>
            ...
        </plugins>
        ...
    </build>
    ...
</project>
Code language: Perl (perl)

And every time maven is executed, it will generate a vulnerability report:

Avoid introducing vulnerabilities through dependencies by updating them to newer versions with fixes.

Note that although it is possible to also scan dependencies later, once the application is built, dependency scanning will be less accurate as some metadata information is not available, and it might be impossible for statically linked applications like Go or Rust.

3. Use image scanning to analyze container images

Once your application is built and packaged, it is common to copy it inside a container with a minimal set of libraries, dependent frameworks (like Python, Node, etc.), and configuration files. You can read our Top 20 Dockerfile best practices to learn about the best practices focused in securing containers building and runtime.

Use an image scanner to analyze your container images. The image scanning tool will discover vulnerabilities in the operating system packages (rpm, dpkg, apk, etc.) provided by the container image base distribution. It will also reveal vulnerabilities in package dependencies for Java, Node, Python, and others, even if you didn’t apply dependency scanning in the previous stages.

Image Scanning is easy to automate and enforce. It can be included as part of your CI/CD pipelines, triggered when new images are pushed to a registry, or verified in a cluster admission controller to make sure that non-compliant images are now allowed to run. Another option is using Sysdig to scan images as soon as they start running in the hosts where they are running.

An example is Github Action integration with the Sysdig Secure Inline Scan Action:

name: "Security Scan"
on:
  push:
jobs:
  build-and-scan:
    runs-on: ubuntu-latest
    steps:
    - name: Build the Docker image
      run: docker build . --file Dockerfile --tag my-image:latest
    - name: Scan image
      id: scan
      uses: sysdiglabs/scan-action@v3
      with:
        image-tag: my-image:latest
        sysdig-secure-token: ${{ secrets.SYSDIG_SECURE_TOKEN }}
        input-type: docker-daemon
        run-as-user: root
Code language: Perl (perl)

The previous example builds a Docker image and then scans it locally, from the Docker daemon.

Scan results are provided directly as part of the action output, and pull-request can be blocked from merging depending on the check status:

4. Enforce image content trust

Container image integrity can be enforced by adding digital signatures via Notary or similar, which then can be verified in the Admission Controller or the container runtime.

Let’s see a quick example:

$ docker trust key generate example1
Generating key for example1...
Enter passphrase for new example1 key with ID 7d7b320:
Repeat passphrase for new example1 key with ID 7d7b320:
Successfully generated and loaded private key. Corresponding public key available: /Users/airadier/example1.pub
Code language: Perl (perl)

Now, we have a signing key called “example1”. The public part is located in:

$HOME/example1.pub
Code language: Perl (perl)

and the private counterpart will be located in:

$HOME/.docker/trust/private/<key ID>.key
Code language: Perl (perl)

Other developers can also generate their keys and share the public part.

Now, we enable a signed repository by adding the keys of the allowed signers to the repository (airadier/alpine in the example):

$ docker trust signer add --key example1.pub example1 airadier/alpine
Adding signer "example1" to airadier/alpine...
Initializing signed repository for airadier/alpine...
...
Enter passphrase for new repository key with ID 16db658:
Repeat passphrase for new repository key with ID 16db658:
Successfully initialized "airadier/alpine"
Successfully added signer: example1 to airadier/alpine
Code language: Perl (perl)

And we can sign an image in the repository with:

$ docker trust sign airadier/alpine:latest
Signing and pushing trust data for local image airadier/alpine:latest, may overwrite remote trust data
The push refers to repository [docker.io/airadier/alpine]
bc276c40b172: Layer already exists
latest: digest: sha256:be9bdc0ef8e96dbc428dc189b31e2e3b05523d96d12ed627c37aa2936653258c size: 528
Signing and pushing trust metadata
Enter passphrase for example1 key with ID 7d7b320:
Successfully signed docker.io/airadier/alpine:latest
Code language: Perl (perl)

If the DOCKER_CONTENT_TRUST environment variable is set to 1, then pushed images will be automatically signed:

$ export DOCKER_CONTENT_TRUST=1
$ docker push airadier/alpine:3.11
The push refers to repository [docker.io/airadier/alpine]
3e207b409db3: Layer already exists
3.11: digest: sha256:39eda93d15866957feaee28f8fc5adb545276a64147445c64992ef69804dbf01 size: 528
Signing and pushing trust metadata
Enter passphrase for example1 key with ID 7d7b320:
Successfully signed docker.io/airadier/alpine:3.11
Code language: Perl (perl)

We can check the signers of an image with:

$ docker trust inspect --pretty airadier/alpine:latest
Signatures for airadier/alpine:latest
SIGNED TAG   DIGEST                                                             SIGNERS
latest       be9bdc0ef8e96dbc428dc189b31e2e3b05523d9...   example1
List of signers and their keys for airadier/alpine:latest
SIGNER     KEYS
example1   7d7b320791b7
Administrative keys for airadier/alpine:latest
  Repository Key:       16db658159255bf0196...
  Root Key:             2308d2a487a1f2d499f184ba...
Code language: Perl (perl)

When the environment variable DOCKER_CONTENT_TRUST is set to 1, the Docker CLI will refuse to pull images without trust information:

$ export DOCKER_CONTENT_TRUST=1
$ docker pull airadier/alpine-ro:latest
Error: remote trust data does not exist for docker.io/airadier/alpine-ro: notary.docker.io does not have trust data for docker.io/airadier/alpine-ro
Code language: Perl (perl)

You can enforce content trust in a Kubernetes cluster by using an Admission controller.

5. Common container security misconfigurations and remediations

Wrongly configured hosts, container runtimes, clusters, or cloud resources can leave a door open to an attack, or create an easy way to escalate privileges and perform lateral movement.

Benchmarks, best practices, and hardening guides provide you with information about how to spot those misconfigurations, why they are a problem, and how to remediate them. Among different sources of information, the Center for Internet Security (CIS) is paramount. It’s a non-profit organization that publishes free benchmarks for many different environments, where any person and company can contribute with their knowledge. It has become a de facto standard for security benchmarking.

The best way to make sure you can check this kind of setting for container security is to automate it as much as possible. Several tools exist for this, mainly based on static configuration analysis, allowing you to check configuration parameters at different levels and provide guidance in fixing them. Sysdig Secure includes a Compliance feature which can help you schedule, execute, and analyze all of your infrastructure (Linux hosts, Docker, Kubernetes, EKS, GKE, AKS, MKE, OpenShift clusters, etc.) based on CIS Benchmarks, as well as compliance standards, like PCI DSS, SOC 2, NIST 800-53, NIST 800-190, HIPAA, ISO 27001, GDPR and others, all in a single centralized dashboard.

Other tools you can use are linux-bench, docker-bench, kube-bench, kube-hunter, kube-striker, Cloud Custodian, OVAL, and OS Query.

Example host benchmark control

A physical machine where you just installed Linux, a virtual machine provisioned on a cloud provider, or on-prem may contain several insecure out-of-the-box configurations that you are not aware of. If you plan to use it for a prolonged period, with a production workload or exposure to the internet, you have to take special care of them. This is also true for Kubernetes or OpenShift nodes. After all, they are virtual machines; don’t assume that if you are using a cluster provisioned by your cloud provider that they come perfectly secured.

CIS has a benchmark for Distribution Independant Linux, and one specifically for Debian, CentOs, Red Hat, and many other distributions.

Examples of misconfigurations you can detect:

Example linux distribution

The following figure is provided by CIS Benchmark for Distribution Independent Linux, the configuration is to ensure rsh server is not enabled.

Example securing containers runtime benchmark control

If you install a container runtime like Docker by yourself in a server you own, it’s essential you use a benchmark to make sure any default insecure configuration is remediated. The next figure shows the configuration to ensure that authorization for Docker client commands is enabled.

Example orchestrator benchmark control

Kubernetes, by default, leaves many authentication mechanisms to be managed by third-party integrations. A benchmark will ensure all possible insecurities are dealt with. The image below shows us the configuration to ensure that the –anonymous-auth argument is set to false.

Example cloud benchmark control

Benchmarks on cloud provider accounts, also called Cloud Security Posture Management (CSPM), are essential, as they will check the security on every single asset on the account. All settings that could lead to an attack, resources that should be private but are made public (e.g., S3 buckets), or storage that lacks encryption are defined in this kind of benchmark. This is a benchmark that is essential to automate, as the assets in the cloud account change all the time, and you have to constantly watch that everything is as secure as possible. The following image is an example of configuration check that ensures credentials unused for 90 days or greater are disabled.

6. Incorporate IaC scanning and policy as code

Cloud resource management is a complex task, and tools like Terraform or CloudFormation can help leverage this burden. Infrastructure is declared as code – aka “Infrastructure as Code” – stored and versioned in a repository, and automation takes care of applying the changes in the definition to keep the existing infrastructure up to date with the declaration.

If you are using infrastructure as code, incorporate tools with IaC scanning capabilities to validate the configuration of your infrastructure before it is created or updated. Similar to other linting tools, apply IaC scanning tools locally and in your pipeline, and consider blocking changes that introduce security issues.

Policy as code is a related approach in which the policies governing container security, infrastructure, and more are defined and managed through code. This approach can be leveraged to automate compliance and governance across IaC and Kubernetes environments to ensure scalability and consistency across an organization. 

Sysdig includes IaC security capabilities, scanning for misconfigurations across IaC tools including Terraform, Helm, or YAML files and mapping them back to the source, as well as leveraging OPA, the OSS standard for policy management, to enforce consistent policies across multiple IaC sources and Kubernetes clusters.

7. Secure your host with host scanning

Securing your host is just as important as securing containers. The host where the containers are running is usually composed of an operating system with a Linux kernel, a set of libraries, a container runtime, and other common services and helpers running in the background. Any of these components can be vulnerable or misconfigured, and could be used as the entry point to access the running containers or cause a denial of service attack.

For example, issues in the container runtime itself can cause an impact in your running containers, like this DoS attack that prevents creating new containers in a host.

We already talked about hardening host configuration in the “Unsafe configuration” section. But how do we detect vulnerable components? A host scanning tool can detect known vulnerabilities in the kernel, standard libraries like glibc, services, and even the container runtime living in the host (quite similar to what image scanning does for a container image).

Sysdig will transparently scan your hosts and report found vulnerabilities. The following figure shows how easy it is to detect risks at a glance on the dashboard.

Use this information to update the operating system, kernel, packages, etc. Get rid of the most critical and exploitable vulnerabilities, or at least be aware of them, and apply other protection mechanisms like firewalls, restricting user access to the host, stopping unused services, etc.

8. Prevent unsafe containers from running

As a last line of defense, Kubernetes Admission Controllers can block unsafe containers from running in the cluster. While the image scanning best practices described earlier can mitigate this issue, not everything you deploy will go through your CI/CD pipeline or known registries. There are also third-party images and manual deployments. For example, you may skip protocol to perform a manual deploy in a rush, or an attacker with access to your cluster could deploy images skipping your image scanner.

With an admission controller, you can define policies to accept or block containers based on the pod specification (e.g., enforce annotations, detect privileged pods, or using host paths) and the status of the cluster (e.g. require all ingress hosts to be unique within the cluster).

Protection – Running your containers safely

Adhering to build time and configuration container security best practices right before runtime still won’t make your container 100 percent safe. New container vulnerabilities are discovered daily, so your actual container, quite safe today, can become a potential victim of new disclosed exploits tomorrow.

In this section, we will introduce container security best practices for including container vulnerability management and protection measures in your workload.

Protect your resources

Your containers and host might contain vulnerabilities, and new ones are discovered continually. However, the danger is not in the host or container vulnerability itself, but rather the attack vector and exploitability.

For example, you can protect from a network exploitable vulnerability by impeding connections to the running container or the vulnerable service. And if the attack vector requires local access to the host (being logged in the host), you can restrict the access to that host.

So, limit the number of users that have access to your hosts, cloud accounts, and resources, and block unnecessary network traffic by using different mechanisms:

• VPCs, Security groups, network rules, firewall rules, etc. in cloud providers to restrict communication between VMs, VPCs, and the Internet.
• Firewalls at hosts levels to expose only the minimal set of required services.
• Kubernetes Network Policies for clusters, and additional tools, like Service Mesh or API Gateways, can add an additional security layer for filtering network requests.

Verify image signatures

As described in “Image content trust”, image signatures are a protection mechanism to guarantee that the image has not been tampered. Verifying image signatures can also prevent some attacks with tag mutability, assuring that the tag corresponds to a specific digest that has been signed by the publisher. The figure below shows an example of this attack.

Restrict container privileges at runtime

The scope or “blast radius” of an exploited vulnerability inside a container largely depends on the privileges of the container, and the level of isolation from the host and other resources. Runtime configuration can mitigate the impact of existing and future vulnerabilities in the following ways:

• Effective user: Don’t run the container as root. Even better, use randomized UIDs (like Openshift) that don’t map to real users in the host, or use the user namespace feature in Docker and in Kubernetes when ready (not available at time of publish).
• Restrict container privileges: Docker and Kubernetes offer ways to drop capabilities and don’t allow privileged containers. Seccomp and AppArmor can add more restrictions to the range of actions a container can perform.
• Add resource limits: Avoid container consuming all the memory or CPUs and starve other applications.
• Be careful with shared storage or volumes: Specifically, things like hostPath, and sharing the filesystem from the host.
• Other options like hostNetwork, hostPID or hostIPC: Kubernetes will make the container share a namespace with the host, reducing isolation.
• Define Pod Security Standards (PSSs) and Security Context Constraints (SCCs for Openshift): Set guardrails in your cluster and prevent misconfigured containers. PSSs and SCCs are Admission Controllers that will reject pods in case their security context does not comply with the defined policies.

Manage container vulnerabilities wisely

Manage and assess your vulnerabilities wisely. Not all vulnerabilities have fixes available, or may now be able to be applied easily.

However, not all of them might be easily exploitable, or they may require local or even physical access to the hosts to be exploited.

You need to have a good strategy, including:

• Prioritize what needs to be fixed: You should focus on host and container vulnerabilities that are in use at runtime. Addressing vulnerabilities that are not in your production environment wastes precious time that could be spent on those that actually put your organization at risk, and vulnerabilities without an active exploit are less risky than those that are being exploited by malicious actors. Only 1% of critical or high severity vulnerabilities have an available fix, have an active exploit, and are in use in production. Prioritizing these riskiest vulnerabilities will save you significant time while addressing risk that matters.
• Plan applying fixes as countermeasures to protect your containers and hosts: Create and track tickets, making vulnerability management part of your standard development workflows.
• Create exceptions for vulnerabilities when you conclude that you are not impacted: This will reduce the noise. Consider snoozing instead of permanently adding an exception, so you can reevaluate later.

Your strategy should translate in policies that a container vulnerability scanner can use to trigger alerts for detected vulnerabilities according to some criteria, and to apply prevention and protection at different levels, like:

• Ticketing: Notify developers of detected vulnerabilities so they can apply the fixes.
• Image registry: Prevent vulnerable images from being pulled at all.
• Host / kernel / container: Block running containers, add additional protection measures or respond by killing, quarantine or shut down hosts or containers for critical issues.

It is also important to perform continuous vulnerability scanning and reevaluation to make sure that you get alerts when new vulnerabilities that apply to running containers are discovered. Sysdig Secure can help here, as it will reevaluate your scanning policies every time the vulnerability feeds are updated.

Detection – Alerts for abnormal behavior

So far, we put the focus on prevention and protection, getting your containers up and running in the best possible shape, and anticipating potential and known attacks. Applying prevention techniques when building, distributing, and running your container with the correct privileges and protections, as well as ensuring the underlying stack, will limit the range of action that an attacker can take. But that doesn’t mean you can just forget about running containers and trust the applied security measures. Once the security measures are running, they can be attacked. We need to detect abnormal and unexpected behavior in our applications in order to take corrective action and prevent security incidents from happening again.

Many different attack vectors exist. For example MITRE ATT&CK provides an extensive list of tactics and techniques “based on real world observations”, which can be used both to apply prevention measures and to analyze the activity to detect abnormal behaviors, which can mean an attack or intrusion is being performed. The MITRE ATT&CK Matrix for Containers covers techniques specifically targeted against container technologies.

Set up real-time event and log auditing

Threats to container security can be detected by auditing different sources of logs and events, and analyzing abnormal activity. Sources of events include:

• Host and Kubernetes logs
• Cloud logs (CloudTrail in AWS, Activity Audit in GCP, etc.)
• System calls in containers

Falco is capable of monitoring the executed system calls and generating alerts for suspicious activity. It includes a community-contributed library of rules, and you can create your own by using a simple syntax. Kubernetes audit log is also supported.

You can see nice examples of Falco in action in our Detecting MITRE ATT&CK articles:

• Privilege escalation with Falco
• Defense evasion techniques with Falco.

Sysdig Secure extends the capabilities of Falco and can also ingest events from different cloud providers.

As an example, the following rule would trigger an alert whenever a new ECS Task is executed in the account:

rule: ECS Task Run or Started
condition: aws.eventSource="ecs.amazonaws.com" and (aws.eventName="RunTask" or aws.eventName="StartTask") and not aws.errorCode exists
output: A new task has been started in ECS (requesting user=%aws.user, requesting IP=%aws.sourceIP, AWS region=%aws.region, cluster=%jevt.value[/requestParameters/cluster], task definition=%aws.ecs.taskDefinition)
source: aws_cloudtrail
description: Detect a new task is started in ECS.
Code language: Perl (perl)

Sysdig also includes an ever growing set of rules tagged with the corresponding compliance standards and controls, and provides a centralized dashboard for exploring all security events in your infrastructure:

Incident response and forensics

Once you detect a security incident is happening in your system, take action to stop the threat and limit any additional harm. Instead of just killing the container or shutting down a host, consider isolating it, pausing it, or taking a snapshot. A good forensics analysis will provide many clues and reveal what, when, and how it happened. It is critical to identify:

• If the security incident was a real attack or just a component malfunction.
• What exactly happened, where did it occur, and are any other potentially impacted components?
• How can you prevent the security incident from happening again?

Isolate and investigate

When a security incident is detected, you should quickly stop it first to limit any further damage.

• Stop and snapshot: When possible, isolate the host or container. Container runtimes’ offer was to “pause” the container (i.e., “docker pause” command) or take a snapshot and then stop it. For hosts, you might take a snapshot at the filesystem level, then shut it down. For EC2 or VM instances, you can also take a snapshot of the instance. Then, proceed to isolation. You can copy the snapshot to a safe sandbox environment, without networking, and resume the host or container.
• Explore and forensics: Once isolated, you can ideally explore the live container or host, and investigate running processes. If the host or container is not alive, then you can just focus on the snapshot of the filesystem. Explore the logs and modified files. There are tools like, Sysdig Secure captures, that greatly enhance forensics capabilities by recording all the system calls around an event and allowing you to explore even after the container or host is dead.
• Kill the compromised container and/or host as a last resource: Simply destroying the suspicious activity will prevent any additional harm in the short term. But missing details about what happened will make it impossible to prevent it from happening again, and you can end up in a never ending whack-a-mole situation, repeatedly waiting for the next attack to happen just to kill it again.

Fix misconfigurations

Investigation should reveal what made the attack possible. Once you have discovered the attack source, take security measures to prevent it from happening again. The cause of a host, container, or application being compromised can be a bad configuration, like excessive permissions, exposed ports or services, or an exploited vulnerability.

In the case of the former, fix the misconfigurations to keep it from happening again. In the latter case, it might be possible to prevent a vulnerability from being exploited (or at least limit its scope) by making changes in configurations, like firewalls, using a more restrictive user, and protecting files or directories with additional permissions or ACLs, etc.

If the issue applies to other assets in your environment, apply the fix in all of them. It’s especially important to do so in those that might be exposed, like applications that are reachable from the internet if the exploit can be executed over a remote network connection.

Patch vulnerabilities

When possible, fix the vulnerability itself:

• For operating system packages (dpkg, rpm, etc.): First check if the distribution vendor is offering an updated version of the package containing a fix. Just update the package or the container base image.
• Older distribution versions: The vendor will stop providing updated versions and security fixes. Keep your hosts and images using supported versions before it is too late.
• For language packages, like NodeJS, Go, Java, etc.: Check for updated versions of the dependencies. Search for minor updates or patch versions that simply fix security issues if you can’t spend additional time planning and testing for breaking changes that can happen on bigger version updates. But plan ahead: old versions won’t be maintained forever.
• In case the distribution does not offer a patched version or there is no fix for unmaintained packages: It is still possible that a fix exists and can be manually applied or backported. This will require some additional work but it can be necessary for packages that are critical for your system and when there is no official fixed version yet. Check the vulnerability links in databases like NVD, vendor feeds and sources, public information in bug reports, etc. If a fix is available, you should be able to locate it.

If there is no fix available that you can apply on the impacted package, it might still be possible to prevent exploiting the vulnerability with configuration or protection measures (e.g., firewalls, isolation, etc.). Also, it might be complex and require a deep knowledge of the vulnerability, but you can add additional checks in your own code. For example, a vulnerability caused by an overflow in a JSON processing library that is used by a web API server could be prevented by adding some checks at the HTTP request level, blocking requests that contain strings that could potentially lead to the overflow.

Close the loop

Unfortunately, host and container security is not a one way trip where you just apply a set of security containers good practices once and can forget forever. Software and infrastructure are evolving everyday, and so complexity increases and new errors are introduced. This leads to vulnerabilities and configuration issues. New attacks and exploits are discovered continuously.

Start by including prevention and security best practices. Then, apply protection measures to your resources, mostly hosts and workloads, but also cloud services. Continue monitoring and detecting anomalous behavior to take action, respond, investigate and report the discovered incidents. Forensics evidence will close the loop: fix discovered vulnerabilities and improve protection to start over again, rebuilding your images, updating packages, reconfiguring your resources, and create incident reports to the future security incidents.

In the middle, you need to assess risk and manage vulnerabilities. The number of inputs to manage in a complex and big environment can be overwhelming, so classify and prioritize to focus on the highest risks first.

Conclusion

We’ve reviewed how container security best practices can be easily applied to your DevOps workflows. In particular, remember to:

• Shift left security, the first step is prevention.
• Protect all your assets.
• Know everything that happens in your organization, monitoring and detecting issues as fast as possible.
• Plan for incident response, because attacks are inevitable.

Remember that container security best practices don’t just include the delivered applications and container images themselves. You also need to include the full component stack used for building, distributing, and specifically executing the container.

70 percent of containers live for five minutes or less, which makes investigating anomalous behavior and breaches extremely challenging.

One of the key points of cloud-native security is addressing container security risks as soon as possible. Doing it later in the development life cycle slows down the pace of cloud adoption, while raising security and compliance risks.