the k8s-security-configwatch GitHub Action, an open source tool from Sysdig, secures your GitOps workloads by detecting changes on your Kubernetes security configuration.
Imagine this scenario: The Secure DevOps team of the “Kubernetes Swag” store is going crazy investigating a security alarm; their Kubernetes containers keep being compromised. An hour after the crisis started, a team member looks at the configuration of their pods and yells, “I’ve found it!”. The network configuration is messed up, causing the containers to be open to the public, and attackers are exploiting this security breach. The team’s intern turns red. He had applied some configuration changes a few hours ago, and he’s most likely responsible for the configuration error.
How was this possible? Wasn’t someone watching the changes in the cluster configuration?
GitOps aims to prevent scenarios like the one above, placing configuration in a git repository and enforcing the review of any changes via pull requests (PR). K8s-security-configwatch is a GitHub action that will help review the security implications of configuration changes, building security into GitOps.
Introduction to Gitops
GitOps is a way to do Kubernetes cluster management and application delivery following the infrastructure as code principle. It uses Git as a single source of truth, storing the infrastructure and application’s declarations. Configuration or code changes in the git repository can be easily applied to the Kubernetes cluster, and changes done directly in the cluster can also be easily corrected. With Git at the center of your delivery pipelines, any changes leave an audit trail. What is most important, however, is that review processes can be put in place to prevent bad configurations from reaching production. Developers can request those changes via pull requests (PR). Then, several teammates will review them, requesting changes and explanations when needed. Only after they are approved will the changes be deployed to the cluster.
However, this system is not perfect and has several potential issues:
- What happens if the right reviewer is not assigned to a pull request?
- Reviewers can still miss some configuration changes, leading to privilege escalation.
Introduction to Github Actions
GitHub Actions help you automate your software development workflows right from your GitHub repositories. You can write individual tasks, called actions and combine them to create complex workflows. An example workflow can be a CI/CD pipeline, where a commit in a repository triggers a build of the project, the execution of tests, a scan for security threats and even a deployment into production.
In the case of k8s-security-configwatch, it could be part of the following workflow. A pull request triggers the workflow, k8s-security-configwatch analyzes the code and if the code changes in the PR affect security, a security engineer is automatically added as a reviewer.
What changes does k8s-security-configwatch track?
K8s-security-configwatch detects changes done on Kubernetes SecurityContext objects, PodSecurityContext objects and the host namespaces settings. This covers the majority of security attributes related to Kubernetes workloads. The full list of attributes that K8s-security-configwatch tracks is:- privileged
- HostPID
- HostIPC
- HostNetwork
- capabilities
- ReadOnlyRootFileSystem
- RunAsUser (root/non-root)
- RunAsGroup (root/non-root)
- volume types
- no change: If the attribute remains the same.
- escalated: If the attribute relaxed its security. This could open a security gap.
- reduced: If the attribute became more restricted. This could cause applications to fail.
apiVersion: v1
kind: Pod
metadata:
name: packetbeat-pod
labels:
app: packetbeat
spec:
containers:
- name: packetbeat
image: elastic/packetbeat:7.6.0
securityContext:
privileged: false
capabilities:
add:
- NET_ADMIN
hostNetwork: true
Here, packetbeat-pod can access the host network namespace. The packetbeat container inside the pod also has the extra NET_ADMIN capability added. All of those attributes should trigger alarms in most pods, but they are fine in this case as packetbeat needs them to perform network monitoring tasks.
Integrating K8s-security-configwatch in a Github Actions workflow
K8s-security-configwatch is a GitHub action that you can find in the Github Marketplace. The following example steps could be added to a Github actions workflow triggered by a pull request to integrate k8s-security-configwatch. Check out our image scanning with Github Actions article if you need some tips on creating a Github Actions workflow.# checkout master branch
- uses: actions/checkout@v2
with:
ref: master
path: master
# checkout PR branch
- uses: actions/checkout@v2
with:
ref: ${{ github.event.pull_request.head.sha }}
path: candidate
# pass the yamls directory to k8s-privilege-check git action
- name: Kubernetes Security configwatch
uses: sysdiglabs/[email protected]
with:
sourceDir: '/master/yamls'
targetDir: '/candidate/yamls'
# evaluate escalation report
- name: Post Privilege Check
run: |
echo ${{ toJSON(steps.k8s_privilege_check.outputs.escalation_report) }}
# slack
# or other git action like adding another reviewer
# or send configwatch result to Sysdig Monitor
Let’s check the steps one by one.
The first step is downloading the source code from the reference branch, the master branch in our case. The source code will be downloaded into a folder called master.
# checkout master branch
- uses: actions/checkout@v2
with:
ref: master
path: master
The next step is downloading the source code from the branch containing the changes. The source code will be downloaded into a folder called candidate.
# checkout PR branch
- uses: actions/checkout@v2
with:
ref: ${{ github.event.pull_request.head.sha }}
path: candidate
Now, we can use k8s-security-configwatch to compare the code in those two folders:
# pass the yamls directory to k8s-privilege-check git action - name: Kubernetes Security configwatch uses: sysdiglabs/[email protected] with: sourceDir: '/master/yamls' targetDir: '/candidate/yamls'Under the hood, k8s-security-configwatch will use the Sysdig open source tool kube-psp-advisor to perform the comparison. Our article on enabling Kubernetes pod security with kube-psp-advisor will introduce you to this tool and explain how it helps generating adaptive pod security policies for Kubernetes clusters. The output from k8s-security-configwatch will be a configwatch report under the
escalation_report attribute. What to do next is up to you.
# evaluate escalation report
- name: Post Privilege Check
run: |
echo ${{ toJSON(steps.k8s_privilege_check.outputs.escalation_report) }}
You could parse the report and decide to take action if one security attribute has been updated. For example, sending a message to a Slack channel or automatically adding a security engineer as reviewer for the pull request.
You could also raise this issue as an event in Sysdig Monitor, where you can also track Kubernetes cluster metrics, health status, events and more in a centralized place. Here is how the resulting report would look in Sysdig Monitor:
