Kubernetes admission controllers are a powerful Kubernetes-native feature that helps you define and customize what is allowed to run on your cluster. An admission controller intercepts and processes requests to the Kubernetes API prior to persistence of the object, but after the request is authenticated and authorized.
In this post we will demonstrate how to integrate the Kubernetes admission controllers functionality with Sysdig Secure to enforce image scanning and your custom security policies on any image that gets deployed on your cluster.
By extending the API functionality you can hook any piece of code that can either modify the request (MutatingAdmissionWebhook ) or decide whether it should be allowed to run at all (ValidatingAdmissionWebhook). As we just mentioned, this is done before the requests is persisted in etcd, which basically means, before it is executed. These features make Kubernetes admissions controllers a perfect candidate to deploy preventive security controls on your cluster.
These controllers are compiled into the kube-apiserver binary, and may only be enabled or updated by the cluster administrator. Below is a diagram that shows how admission controllers participate in the Kubernetes resource creation workflow:
As you can see in the diagram above, the communication between the Kubernetes API server and your custom code implementation uses a rather standard webhook interface, easing the integration and interface with any third party code.
What do Kubernetes admission controllers do?
Containerized microservices are typically running inside pods which are created dynamically and run ephemerally. These features are revolutionary for the software lifecycle, but they make cloud native security a fast moving target. Whenever we create or auto-scale new deployments and pods, there are many questions that need to be answered by the cluster administrators:
- Is the pod requesting too many resources?
- Are the base images used to spawn the microservice pods secure?
- What is the priority of this deployment compared to the others?
- For example, if the cluster is out of resources and needs to evict pods
- Which privileges are currently granted to the service account linked to this pods/deployments?
- Do they adhere to the principle of least privilege?
Being able to answer these questions is critical to run reliable and secure services. Each organization has their own policies and default set of best practices. This has created the need for a “plug-in” policy mechanism able to extend the Kubernetes API functionality without adding complexity to its base code.
You are probably already running several Kubernetes admission controllers that come pre-configured out of the box with your standard deployment. This is the set of recommended Kubernetes admission controllers that you may find in any vanilla deployment:
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,Priority,ResourceQuota,PodSecurityPolicy
Note that some aspects of Kubernetes that you may consider built-in are actually enforced by these controllers, like pod priorities or limits on resource requests.
These admission controllers can fulfill different roles, from resource control to provisioning and security aspects. Below are a few examples of admission controllers supported in Kubernetes:
- PodSecurityPolicy: This admission controller acts on creation and modification of the pod and determines if it should be admitted based on the requested security context and the available Pod Security Policies.
- PersistentVolumeClaimResize: This admission controller implements additional validations for checking incoming PersistentVolumeClaim resize requests.
- ValidatingAdmissionWebhook: This admission controller calls any validating webhooks which match the request. Matching webhooks are called in parallel; if any of them rejects the request, the request fails. We can easily adapt these webhooks to implement our custom security policies.
Enough theory, let’s get down to work with a real Kubernetes security use case.
Use Case: Integrate Sysdig Secure image scanning as one of your Kubernetes admission controllers
Sysdig Secure provides an elegant way to integrate your Kubernetes cluster with Sysdig Secure Image Scanning and Kubernetes compliance. A high level workflow diagram is available below:
In this case, the webhook provider will be a pod living in your own Kubernetes cluster. This pod will be in charge of registering itself as a ValidatingAdmissionWebhook, parsing the API requests and forwarding the image definition to Sysdig Secure.
Sysdig Secure will then retrieve the image and verify that it is compliant with the configured security policies. This security validation decision will be propagated back to the API, which will reply to the original requester and only persist the object in the etcd database if the image passed the checks.
Only a few steps are needed to integrate Sysdig Secure Image Scanning within the workflow mentioned above.
Step 1: Create your image scanning policy using Sysdig Secure
Sysdig Secure leverages anchore an open source scanning engine to analyze the contents of images in your running environment, registry, or CI/CD pipelines. When you create a policy that is going to be triggered by the Kubernetes admission controller, an obvious first step is to check for vulnerabilities in the image, but there is so much more that you can enforce with the different gates that Sysdig Secure policies put at your disposal: from Dockerfile rules to software licensing, file, file permissions, allowed effective users, language-specific packages and metadata attributes, just to name a few. Here you have a more comprehensive description of the Sysdig Secure image scanning policy gates and triggers that you may want to configure.
Once you have defined a security gate, you will decide if that is a stop condition (in which case, the admission controller will reject deployment) or just a security warning that will be reflected in the image report.
For the example policy that we configured in the image above, we are checking that:
- Vulnerability feed data for this image is not older than a week
- The image has no vulnerabilities (considered high or critical)
- Port 22 is not exposed in the Dockerfile
- Effective user for this image is not root
Step 2: Configure the webhook and deploy the admission controllers
Deploying this integration in your Kubernetes cluster is really simple. First make sure that kubectlis installed and configured to point at the target cluster.
Then, checkout the project repository and cd into the new directory:
git clone git@github.com:sysdiglabs/image-scanning-admission-controller.git
Then, you will need to declare two environment variables:
export ANCHORE_CLI_URL="https://api.sysdigcloud.com/api/scanning/v1/anchore"
export ANCHORE_CLI_USER="xxxxxxx-xxxx-xxxxxx-xxxxxx-xxxxx"
For the endpoint URL and API access token, if you are using the SaaS version of Sysdig Secure, you can use the URL above, otherwise you will need to replace the host section of that address. You can find your API access token accessing the Settings menu on your Sysdig Secure profile.
To effectively deploy, you just need to execute make deploy:
$ make deploy
+ deploy
./scripts/deploy.sh
namespace/image-scan-k8s-webhook-system created
clusterrole.rbac.authorization.k8s.io/image-scan-k8s-webhook-manager-role created
clusterrolebinding.rbac.authorization.k8s.io/image-scan-k8s-webhook-manager-rolebinding created
secret/image-scan-k8s-webhook-webhook-server-secret created
secret/sysdig-secure-token created
service/image-scan-k8s-webhook-controller-manager-service created
statefulset.apps/image-scan-k8s-webhook-controller-manager created
+ sleep 3
+ kubectl get all -n image-scan-k8s-webhook-system
NAME READY STATUS RESTARTS AGE
pod/image-scan-k8s-webhook-controller-manager-0 0/1 ContainerCreating 0 3s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/image-scan-k8s-webhook-controller-manager-service ClusterIP 10.104.71.12 <none> 443/TCP 3s
NAME READY AGE
statefulset.apps/image-scan-k8s-webhook-controller-manager 0/1 3s
Step 3: Verify setup
The last step has created the image-scan-k8s-webhook-system. Let’s make sure the webhook provider is up and running:
$ kubectl get pods -n image-scan-k8s-webhook-system
NAME READY STATUS RESTARTS AGE
image-scan-k8s-webhook-controller-manager-0 1/1 Running 1 25m
You can also verify that the ValidatingWebhookConfigurationhas been registered in the Kubernetes API:
$ kubectl get ValidatingWebhookConfiguration
NAME CREATED AT
validating-webhook-configuration 2019-05-29T10:45:10Z
If you inspect further inspect this object, you will be able to see the webhook service configuration (among many other attributes):
$ kubectl describe ValidatingWebhookConfiguration
...
Service:
Name: webhook-server-service
Namespace: image-scan-k8s-webhook-system
Path: /validating-create-pods
...
Step 4: Test the Sysdig Secure integration
Now your admission controller is running and ready to inspect any new image you spawn on the cluster. You can try this integration with your own images, but to make testing easier we also provide two default deployments. One that passes the security filter and another that does not.
Just run:
$ make test
And a few seconds later:
$ kubectl get deploy
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
apache-struts2 1 0 0 0 5m
nginx 1 1 1 1 5m
Nginx deployment passed the policy evaluation check, so the pod is running as shown in the AVAILABLE state. Deployment apache-struts2 failed to run as it contains a high severity vulnerability cve-2017-5638. You can inspect the Kubernetes logs to get the request response:
$ kubectl get events
LAST SEEN TYPE REASON OBJECT MESSAGE
67s Warning FailedCreate replicaset/apache-struts2-8bb7bd7b Error creating: admission webhook "validating-create-pods.k8s.io" denied the request: Image failed policy check: kaizheh/apache-struts2-cve-2017-5638
Additionally, you will have the full scanning report for any image that you spawn while this webhook is configured in your cluster by just accessing the Sysdig Secure interface. If you open the interface and navigate to the Image scanning, Repositories tab you will find a search box. Look for the name and tag of the image you submitted to display the security report:
Note: there is a 30 seconds timeout imposed by the Kubernetes API for the webhook validation, if this is the first time you deploy a given image, the admission controller will deny the request if there is no response from the API during this time span. You can tune the default timeout behaviour configuring the environment variable REJECT_IF_TIMEOUT in the image-scanning-admission-controller.yaml file.
Conclusion
Both the Kubernetes admission controllers and the Sysdig API offer a simple and secure mechanism to integrate with any third parties that support webhook invocation. By deploying this integration on your cluster you can enforce the full extent of Sysdig Secure image scanning policies and security gates for any pod that is requested on your cluster, before it even gets scheduled. Mandatory security by default is one of the best types of security and it will greatly increase your peace of mind as a Kubernetes operator.
You can get the source code for the webhook provider here.