the
k8s-security-configwatch GitHub Action, an open source tool from Sysdig, secures your GitOps workloads by detecting changes on your
Kubernetes security configuration.
Imagine this scenario: The Secure DevOps team of the “Kubernetes Swag” store is going crazy investigating a security alarm; their Kubernetes containers keep being compromised. An hour after the crisis started, a team member looks at the configuration of their pods and yells, “I’ve found it!”. The network configuration is messed up, causing the containers to be open to the public, and attackers are exploiting this security breach. The team’s intern turns red. He had applied some configuration changes a few hours ago, and he’s most likely responsible for the configuration error.
How was this possible? Wasn’t someone watching the changes in the cluster configuration?

GitOps
aims to prevent scenarios like the one above, placing configuration in a git repository and enforcing the review of any changes via pull requests (PR).
K8s-security-configwatch is a GitHub action that will help review the security implications of configuration changes, building security into GitOps.
Introduction to Gitops
GitOps is a way to do Kubernetes cluster management and application delivery following the
infrastructure as code principle. It uses Git as a single source of truth, storing the infrastructure and application’s declarations. Configuration or code changes in the git repository can be easily applied to the Kubernetes cluster, and changes done directly in the cluster can also be easily corrected.
With Git at the center of your delivery pipelines, any changes leave an audit trail. What is most important, however, is that review processes can be put in place to prevent bad configurations from reaching production.
Developers can request those changes via pull requests (PR). Then, several teammates will review them, requesting changes and explanations when needed. Only after they are approved will the changes be deployed to the cluster.

However, this system is not perfect and has several potential issues:
- What happens if the right reviewer is not assigned to a pull request?
- Reviewers can still miss some configuration changes, leading to privilege escalation.
This is where
k8s-security-configwatch, a new open source tool from Sysdig, shows its value. K8s-security-configwatch is a Github Action that reviews the changes in your Kubernetes configuration files, highlighting those that can affect the security of the cluster. Reviewers can then take special care in those areas and ensure that the right eyes are validating every pull request.
Introduction to Github Actions
GitHub Actions
help you automate your software development workflows right from your GitHub repositories. You can write individual tasks, called actions and combine them to create complex workflows.
An example workflow can be a CI/CD pipeline, where a commit in a repository triggers a build of the project, the execution of tests, a
scan for security threats and even a deployment into production.

In the case of k8s-security-configwatch, it could be part of the following workflow. A pull request triggers the workflow, k8s-security-configwatch analyzes the code and if the code changes in the PR affect security, a security engineer is automatically added as a reviewer.
What changes does k8s-security-configwatch track?
K8s-security-configwatch detects changes done on Kubernetes SecurityContext objects, PodSecurityContext objects and the
host namespaces settings. This covers the majority of security attributes related to Kubernetes workloads. The full list of attributes that K8s-security-configwatch tracks is:
- privileged
- HostPID
- HostIPC
- HostNetwork
- capabilities
- ReadOnlyRootFileSystem
- RunAsUser (root/non-root)
- RunAsGroup (root/non-root)
- volume types
For each of those attributes, K8s-security-configwatch gives the following feedback:
- no change: If the attribute remains the same.
- escalated: If the attribute relaxed its security. This could open a security gap.
- reduced: If the attribute became more restricted. This could cause applications to fail.
Let’s see an example of those attributes in a configuration file. Below is a SecurityContext in a pod:
apiVersion: v1
kind: Pod
metadata:
name: packetbeat-pod
labels:
app: packetbeat
spec:
containers:
- name: packetbeat
image: elastic/packetbeat:7.6.0
securityContext:
privileged: false
capabilities:
add:
- NET_ADMIN
hostNetwork: true
Here,
packetbeat-pod
can access the host network namespace. The
packetbeat
container inside the pod also has the extra NET_ADMIN capability added. All of those attributes should trigger alarms in most pods, but they are fine in this case as packetbeat needs them to perform network monitoring tasks.
Integrating K8s-security-configwatch in a Github Actions workflow
K8s-security-configwatch is a GitHub action that you can find in the
Github Marketplace. The following example steps could be added to a Github actions workflow triggered by a pull request to integrate k8s-security-configwatch. Check out our
image scanning with Github Actions article if you need some tips on creating a Github Actions workflow.
# checkout master branch
- uses: actions/checkout@v2
with:
ref: master
path: master
# checkout PR branch
- uses: actions/checkout@v2
with:
ref: ${{ github.event.pull_request.head.sha }}
path: candidate
# pass the yamls directory to k8s-privilege-check git action
- name: Kubernetes Security configwatch
uses: sysdiglabs/[email protected]
with:
sourceDir: '/master/yamls'
targetDir: '/candidate/yamls'
# evaluate escalation report
- name: Post Privilege Check
run: |
echo ${{ toJSON(steps.k8s_privilege_check.outputs.escalation_report) }}
# slack
# or other git action like adding another reviewer
# or send configwatch result to Sysdig Monitor
Let’s check the steps one by one.
The first step is downloading the source code from the reference branch, the master branch in our case. The source code will be downloaded into a folder called
master
.
# checkout master branch
- uses: actions/checkout@v2
with:
ref: master
path: master
The next step is downloading the source code from the branch containing the changes. The source code will be downloaded into a folder called
candidate
.
# checkout PR branch
- uses: actions/checkout@v2
with:
ref: ${{ github.event.pull_request.head.sha }}
path: candidate
Now, we can use k8s-security-configwatch to compare the code in those two folders:
# pass the yamls directory to k8s-privilege-check git action
- name: Kubernetes Security configwatch
uses: sysdiglabs/[email protected]
with:
sourceDir: '/master/yamls'
targetDir: '/candidate/yamls'
Under the hood, k8s-security-configwatch will use the Sysdig open source tool
kube-psp-advisor to perform the comparison. Our article on
enabling Kubernetes pod security with kube-psp-advisor will introduce you to this tool and explain how it helps generating adaptive pod security policies for Kubernetes clusters.
The output from k8s-security-configwatch will be a configwatch report under the
escalation_report
attribute. What to do next is up to you.
# evaluate escalation report
- name: Post Privilege Check
run: |
echo ${{ toJSON(steps.k8s_privilege_check.outputs.escalation_report) }}
You could parse the report and decide to take action if one security attribute has been updated. For example, sending a message to a Slack channel or automatically adding a security engineer as reviewer for the pull request.
You could also raise this issue as an event in
Sysdig Monitor, where you can also track Kubernetes cluster metrics, health status, events and more in a centralized place. Here is how the resulting report would look in Sysdig Monitor:
Conclusion
Configuration changes need to be audited the same way your application code is. GitOps, following the infrastructure as code principle, helps you build the tools to achieve that goal. Github action k8s-security-configwatch detects Kubernetes security changes to help secure your GitOps workloads.
The
Kubernetes Policy Advisor in Sysdig Secure automates the generation of pod security policies and validates them pre-deployment so they don’t compromise applications when applied. Wait no longer and
request a demo now!