Blog Icon

Blog Post

Kubernetes admission controllers for secure deployments.

Kubernetes admission controllers are a powerful Kubernetes-native feature that helps you define and customize what is allowed to run on your cluster. An admission controller intercepts and processes requests to the Kubernetes API prior to persistence of the object, but after the request is authenticated and authorized.

In this post we will demonstrate how to integrate the Kubernetes admission controllers functionality with Sysdig Secure to enforce image scanning and your custom security policies on any image that gets deployed on your cluster.

By extending the API functionality you can hook any piece of code that can either modify the request (MutatingAdmissionWebhook ) or decide whether it should be allowed to run at all (ValidatingAdmissionWebhook). As we just mentioned, this is done before the requests is persisted in etcd, which basically means, before it is executed. These features make Kubernetes admissions controllers a perfect candidate to deploy preventive security controls on your cluster.

These controllers are compiled into the kube-apiserver binary, and may only be enabled or updated by the cluster administrator. Below is a diagram that shows how admission controllers participate in the Kubernetes resource creation workflow:

Admission controllers diagram

As you can see in the diagram above, the communication between the Kubernetes API server and your custom code implementation uses a rather standard webhook interface, easing the integration and interface with any third party code.

What do Kubernetes admission controllers do?

Containerized microservices are typically running inside pods which are created dynamically and run ephemerally. These features are revolutionary for the software lifecycle, but they make cloud native security a fast moving target. Whenever we create or auto-scale new deployments and pods, there are many questions that need to be answered by the cluster administrators:

  • Is the pod requesting too many resources?
  • Are the base images used to spawn the microservice pods secure?
  • What is the priority of this deployment compared to the others?
    • For example, if the cluster is out of resources and needs to evict pods
  • Which privileges are currently granted to the service account linked to this pods/deployments?
    • Do they adhere to the principle of least privilege?

Being able to answer these questions is critical to run reliable and secure services. Each organization has their own policies and default set of best practices. This has created the need for a “plug-in” policy mechanism able to extend the Kubernetes API functionality without adding complexity to its base code.

You are probably already running several Kubernetes admission controllers that come pre-configured out of the box with your standard deployment. This is the set of recommended Kubernetes admission controllers that you may find in any vanilla deployment:

--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,Priority,ResourceQuota,PodSecurityPolicy

Note that some aspects of Kubernetes that you may consider built-in are actually enforced by these controllers, like pod priorities or limits on resource requests.

These admission controllers can fulfill different roles, from resource control to provisioning and security aspects. Below are a few examples of admission controllers supported in Kubernetes:

  • PodSecurityPolicy: This admission controller acts on creation and modification of the pod and determines if it should be admitted based on the requested security context and the available Pod Security Policies.
  • PersistentVolumeClaimResize: This admission controller implements additional validations for checking incoming PersistentVolumeClaim resize requests.
  • ValidatingAdmissionWebhook: This admission controller calls any validating webhooks which match the request. Matching webhooks are called in parallel; if any of them rejects the request, the request fails. We can easily adapt these webhooks to implement our custom security policies.

Enough theory, let’s get down to work with a real Kubernetes security use case.

Use Case: Integrate Sysdig Secure image scanning as one of your Kubernetes admission controllers

Sysdig Secure provides an elegant way to integrate your Kubernetes cluster with Sysdig Secure Image Scanning and Kubernetes compliance. A high level workflow diagram is available below:

Admission controllers Sysdig Secure integration

In this case, the webhook provider will be a pod living in your own Kubernetes cluster. This pod will be in charge of registering itself as a ValidatingAdmissionWebhook, parsing the API requests and forwarding the image definition to Sysdig Secure.

Sysdig Secure will then retrieve the image and verify that it is compliant with the configured security policies. This security validation decision will be propagated back to the API, which will reply to the original requester and only persist the object in the etcd database if the image passed the checks.

Only a few steps are needed to integrate Sysdig Secure Image Scanning within the workflow mentioned above.

Step 1: Create your image scanning policy using Sysdig Secure

Sysdig Image Scanning Policy

Sysdig Secure leverages anchore an open source scanning engine to analyze the contents of images in your running environment, registry, or CI/CD pipelines. When you create a policy that is going to be triggered by the Kubernetes admission controller, an obvious first step is to check for vulnerabilities in the image, but there is so much more that you can enforce with the different gates that Sysdig Secure policies put at your disposal: from Dockerfile rules to software licensing, file, file permissions, allowed effective users, language-specific packages and metadata attributes, just to name a few. Here you have a more comprehensive description of the Sysdig Secure image scanning policy gates and triggers that you may want to configure.

Once you have defined a security gate, you will decide if that is a stop condition (in which case, the admission controller will reject deployment) or just a security warning that will be reflected in the image report.

For the example policy that we configured in the image above, we are checking that:

  • Vulnerability feed data for this image is not older than a week
  • The image has no vulnerabilities (considered high or critical)
  • Port 22 is not exposed in the Dockerfile
  • Effective user for this image is not root

Step 2: Configure the webhook and deploy the admission controllers

Deploying this integration in your Kubernetes cluster is really simple. First make sure that kubectlis installed and configured to point at the target cluster.

Then, checkout the project repository and cd into the new directory:

git clone git@github.com:sysdiglabs/image-scanning-admission-controller.git

Then, you will need to declare two environment variables:

export ANCHORE_CLI_URL="https://api.sysdigcloud.com/api/scanning/v1/anchore"
export ANCHORE_CLI_USER="xxxxxxx-xxxx-xxxxxx-xxxxxx-xxxxx"

For the endpoint URL and API access token, if you are using the SaaS version of Sysdig Secure, you can use the URL above, otherwise you will need to replace the host section of that address. You can find your API access token accessing the Settings menu on your Sysdig Secure profile.

To effectively deploy, you just need to execute make deploy:

$ make deploy
+ deploy
./scripts/deploy.sh
namespace/image-scan-k8s-webhook-system created
clusterrole.rbac.authorization.k8s.io/image-scan-k8s-webhook-manager-role created
clusterrolebinding.rbac.authorization.k8s.io/image-scan-k8s-webhook-manager-rolebinding created
secret/image-scan-k8s-webhook-webhook-server-secret created
secret/sysdig-secure-token created
service/image-scan-k8s-webhook-controller-manager-service created
statefulset.apps/image-scan-k8s-webhook-controller-manager created
+ sleep 3
+ kubectl get all -n image-scan-k8s-webhook-system
NAME                                              READY   STATUS              RESTARTS   AGE
pod/image-scan-k8s-webhook-controller-manager-0   0/1     ContainerCreating   0          3s

NAME                                                        TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/image-scan-k8s-webhook-controller-manager-service   ClusterIP   10.104.71.12   <none>        443/TCP   3s

NAME                                                         READY   AGE
statefulset.apps/image-scan-k8s-webhook-controller-manager   0/1     3s

Step 3: Verify setup

The last step has created the image-scan-k8s-webhook-system. Let’s make sure the webhook provider is up and running:

$ kubectl get pods -n image-scan-k8s-webhook-system
NAME                                          READY   STATUS    RESTARTS   AGE
image-scan-k8s-webhook-controller-manager-0   1/1     Running   1          25m

You can also verify that the ValidatingWebhookConfigurationhas been registered in the Kubernetes API:

$ kubectl get ValidatingWebhookConfiguration
NAME                               CREATED AT
validating-webhook-configuration   2019-05-29T10:45:10Z

If you inspect further inspect this object, you will be able to see the webhook service configuration (among many other attributes):

$ kubectl describe ValidatingWebhookConfiguration      
...
   Service:
      Name:        webhook-server-service
      Namespace:   image-scan-k8s-webhook-system
      Path:        /validating-create-pods
...

Step 4: Test the Sysdig Secure integration

Now your admission controller is running and ready to inspect any new image you spawn on the cluster. You can try this integration with your own images, but to make testing easier we also provide two default deployments. One that passes the security filter and another that does not.

Just run:

$ make test

And a few seconds later:

$ kubectl get deploy 
NAME              DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
apache-struts2    1         0         0            0           5m
nginx             1         1         1            1           5m

Nginx deployment passed the policy evaluation check, so the pod is running as shown in the AVAILABLE state. Deployment apache-struts2 failed to run as it contains a high severity vulnerability cve-2017-5638. You can inspect the Kubernetes logs to get the request response:

$ kubectl get events
LAST SEEN   TYPE      REASON                    OBJECT                                MESSAGE
67s         Warning   FailedCreate              replicaset/apache-struts2-8bb7bd7b   Error creating: admission webhook "validating-create-pods.k8s.io" denied the request: Image failed policy check: kaizheh/apache-struts2-cve-2017-5638

Additionally, you will have the full scanning report for any image that you spawn while this webhook is configured in your cluster by just accessing the Sysdig Secure interface. If you open the interface and navigate to the Image scanning, Repositories tab you will find a search box. Look for the name and tag of the image you submitted to display the security report:

Sysdig Secure Scan Report

Note: there is a 30 seconds timeout imposed by the Kubernetes API for the webhook validation, if this is the first time you deploy a given image, the admission controller will deny the request if there is no response from the API during this time span. You can tune the default timeout behaviour configuring the environment variable REJECT_IF_TIMEOUT in the image-scanning-admission-controller.yaml file.

Conclusion

Both the Kubernetes admission controllers and the Sysdig API offer a simple and secure mechanism to integrate with any third parties that support webhook invocation. By deploying this integration on your cluster you can enforce the full extent of Sysdig Secure image scanning policies and security gates for any pod that is requested on your cluster, before it even gets scheduled. Mandatory security by default is one of the best types of security and it will greatly increase your peace of mind as a Kubernetes operator.

You can get the source code for the webhook provider here.

Share This

Stay up to date

Sign up to receive our newest.

Related Posts

GKE security using Falco, Pub/Sub, and Cloud Functions.

MITRE ATT&CK framework for container runtime security with Falco.

Announcing the Sysdig Cloud-Native Visibility + Security Platform 2.0