GitOps Security with k8s-security-configwatch

the k8s-security-configwatch GitHub Action, an open source tool from Sysdig, secures your GitOps workloads by detecting changes on your Kubernetes security configuration. Imagine this scenario: The Secure DevOps team of the “Kubernetes Swag” store is going crazy investigating a security alarm; their Kubernetes containers keep being compromised. An hour after the crisis started, a team member looks at the configuration of their pods and yells, “I’ve found it!”. The network configuration is messed up, causing the containers to be open to the public, and attackers are exploiting this security breach. The team’s intern turns red. He had applied some configuration changes a few hours ago, and he’s most likely responsible for the configuration error. How was this possible? Wasn’t someone watching the changes in the cluster configuration? GitOps aims to prevent scenarios like the one above, placing configuration in a git repository and enforcing the review of any changes via pull requests (PR). K8s-security-configwatch is a GitHub action that will help review the security implications of configuration changes, building security into GitOps. Securing GitOps is easier with k8s-security-configwatch, an open source tool from Sysdig. Learn how to detect changes on your Kubernetes security configuration with this GitHub Action. Click to tweet

Introduction to Gitops

GitOps is a way to do Kubernetes cluster management and application delivery following the infrastructure as code principle. It uses Git as a single source of truth, storing the infrastructure and application’s declarations. Configuration or code changes in the git repository can be easily applied to the Kubernetes cluster, and changes done directly in the cluster can also be easily corrected. With Git at the center of your delivery pipelines, any changes leave an audit trail. What is most important, however, is that review processes can be put in place to prevent bad configurations from reaching production. Developers can request those changes via pull requests (PR). Then, several teammates will review them, requesting changes and explanations when needed. Only after they are approved will the changes be deployed to the cluster. However, this system is not perfect and has several potential issues:

1. What happens if the right reviewer is not assigned to a pull request?
2. Reviewers can still miss some configuration changes, leading to privilege escalation.

This is where k8s-security-configwatch, a new open source tool from Sysdig, shows its value. K8s-security-configwatch is a Github Action that reviews the changes in your Kubernetes configuration files, highlighting those that can affect the security of the cluster. Reviewers can then take special care in those areas and ensure that the right eyes are validating every pull request.

Introduction to Github Actions

GitHub Actions help you automate your software development workflows right from your GitHub repositories. You can write individual tasks, called actions and combine them to create complex workflows. An example workflow can be a CI/CD pipeline, where a commit in a repository triggers a build of the project, the execution of tests, a scan for security threats and even a deployment into production. In the case of k8s-security-configwatch, it could be part of the following workflow. A pull request triggers the workflow, k8s-security-configwatch analyzes the code and if the code changes in the PR affect security, a security engineer is automatically added as a reviewer.

What changes does k8s-security-configwatch track?

K8s-security-configwatch detects changes done on Kubernetes SecurityContext objects, PodSecurityContext objects and the host namespaces settings. This covers the majority of security attributes related to Kubernetes workloads. The full list of attributes that K8s-security-configwatch tracks is:

• privileged
• HostPID
• HostIPC
• HostNetwork
• capabilities
• ReadOnlyRootFileSystem
• RunAsUser (root/non-root)
• RunAsGroup (root/non-root)
• volume types

For each of those attributes, K8s-security-configwatch gives the following feedback:

• no change: If the attribute remains the same.
• escalated: If the attribute relaxed its security. This could open a security gap.
• reduced: If the attribute became more restricted. This could cause applications to fail.

Let’s see an example of those attributes in a configuration file. Below is a SecurityContext in a pod:

apiVersion: v1
kind: Pod
metadata:
  name: packetbeat-pod
  labels:
    app: packetbeat
spec:
  containers:
  - name: packetbeat
    image: elastic/packetbeat:7.6.0
    securityContext:
      privileged: false
      capabilities:
        add:
        - NET_ADMIN
  hostNetwork: true

Here, packetbeat-pod can access the host network namespace. The packetbeat container inside the pod also has the extra NET_ADMIN capability added. All of those attributes should trigger alarms in most pods, but they are fine in this case as packetbeat needs them to perform network monitoring tasks.

Integrating K8s-security-configwatch in a Github Actions workflow

K8s-security-configwatch is a GitHub action that you can find in the Github Marketplace. The following example steps could be added to a Github actions workflow triggered by a pull request to integrate k8s-security-configwatch. Check out our image scanning with Github Actions article if you need some tips on creating a Github Actions workflow.

# checkout master branch
- uses: actions/checkout@v2
    with:
      ref: master
      path: master
# checkout PR branch
- uses: actions/checkout@v2
    with:
      ref: ${{ github.event.pull_request.head.sha }}
      path: candidate
# pass the yamls directory to k8s-privilege-check git action
- name: Kubernetes Security configwatch
  uses: sysdiglabs/[email protected]
  with:
    sourceDir: '/master/yamls'
    targetDir: '/candidate/yamls'
# evaluate escalation report
- name: Post Privilege Check
  run: |
    echo ${{ toJSON(steps.k8s_privilege_check.outputs.escalation_report) }}
    # slack
    # or other git action like adding another reviewer
    # or send configwatch result to Sysdig Monitor

Let’s check the steps one by one. The first step is downloading the source code from the reference branch, the master branch in our case. The source code will be downloaded into a folder called master.

# checkout master branch
- uses: actions/checkout@v2
    with:
      ref: master
      path: master

The next step is downloading the source code from the branch containing the changes. The source code will be downloaded into a folder called candidate.

# checkout PR branch
- uses: actions/checkout@v2
    with:
      ref: ${{ github.event.pull_request.head.sha }}
      path: candidate

Now, we can use k8s-security-configwatch to compare the code in those two folders:

# pass the yamls directory to k8s-privilege-check git action
- name: Kubernetes Security configwatch
  uses: sysdiglabs/[email protected]
  with:
    sourceDir: '/master/yamls'
    targetDir: '/candidate/yamls'

Under the hood, k8s-security-configwatch will use the Sysdig open source tool kube-psp-advisor to perform the comparison. Our article on enabling Kubernetes pod security with kube-psp-advisor will introduce you to this tool and explain how it helps generating adaptive pod security policies for Kubernetes clusters. The output from k8s-security-configwatch will be a configwatch report under the escalation_report attribute. What to do next is up to you.

# evaluate escalation report
- name: Post Privilege Check
  run: |
    echo ${{ toJSON(steps.k8s_privilege_check.outputs.escalation_report) }}

You could parse the report and decide to take action if one security attribute has been updated. For example, sending a message to a Slack channel or automatically adding a security engineer as reviewer for the pull request. You could also raise this issue as an event in Sysdig Monitor, where you can also track Kubernetes cluster metrics, health status, events and more in a centralized place. Here is how the resulting report would look in Sysdig Monitor:

Conclusion

Configuration changes need to be audited the same way your application code is. GitOps, following the infrastructure as code principle, helps you build the tools to achieve that goal. Github action k8s-security-configwatch detects Kubernetes security changes to help secure your GitOps workloads. The Kubernetes Policy Advisor in Sysdig Secure automates the generation of pod security policies and validates them pre-deployment so they don’t compromise applications when applied. Wait no longer and request a demo now!