Kubernetes Security Guide, Chapter 1: Kubernetes RBAC and TLS certificates
RBAC security context is a fundamental part of your Kubernetes security best practices, as well as rolling out TLS certificates / PKI authentication for the core Kubernetes API server. We will learn how to create Kubernetes users, pod service account, set their permissions using RBAC and rotate their TLS certificates and security tokens.
Kubernetes Role-Based Access Control (RBAC)
RBAC is essentially an authorization and access control specification where you define the actions (GET, UPDATE, DELETE, etc) that Kubernetes subjects (i.e. human users, software, kubelets) are allowed to perform over Kubernetes entities (i.e. pods, secrets, nodes).
#Kubernetes security guide, chapter 1: RBAC and TLS certificates step by step & practical examples.Click to tweet
RBAC uses the "rbac.authorization.k8s.io" API group to drive authorization decisions.
You need to understand its building blocks first:
Namespaces: Logical segmentation and isolation, or "virtual clusters". Correct use of Kubernetes namespaces is fundamental for security, as you can group together users, roles and resources according to business logic without granting global privileges for the cluster. Typically you use a namespace to group a project, an application, a team or a customer.
Subjects: The security "actors".
- Regular users: Humans or other authorized accesses from outside the cluster. Kubernetes delegates the user creation and management to the administrator. In practice, this means that you can "refer" to a user, as we will see on the RBAC examples below, but there is no User API object per se.
- ServiceAccounts: Used to assign permissions to software entities. Kubernetes creates its own default serviceAccounts, and you can create additional ones for your pods/deployments. Any pod run by Kubernetes gets its own privileges through its serviceAccount, applied to all processes run within the containers of that pod.
- Groups of users. Kubernetes user groups are not explicitly created, instead, the API can implicitly group users using a common property, like the prefix of a serviceAccount or the organization field of a user certificate. As with regular Linux permissions, you can assign RBAC privileges to entire groups.
Resources: The entities that will be accessed by the subjects.
- Resources can refer to a generic entity ("pod", "deployment", etc), subresources like the logs coming from a pod ("pod/log") or the particular resource name like an Ingress: "ingress-controller-istio", including custom resources your deployment defines.
- Resources can also refer to Pod Security Policies or PSP, we will talk about them later.
Role and ClusterRole: A set of permissions over a group of resources, think of it as a "security profile", a Role is always confined to a single namespace, a ClusterRole is cluster-scoped.
- Before designing your security policy, take into account that Kubernetes RBAC permissions are explicitly additive, there are no "deny" rules.
- Some resources only make sense at the cluster level (i.e. nodes), you need to create a ClusterRole to control access in this case.
- Roles define a list of actions that can be performed over the resources or verbs: GET, WATCH, LIST, CREATE, UPDATE, PATCH, DELETE.
RoleBindings and ClusterRoleBindings: Grants the permissions defined in a Role or ClusterRole to a subject or group of subjects. Again, RoleBindings are bounded to a certain namespace, ClusterRoleBindings are cluster-global.
Let's start looking at these Role and RoleBinding definitions:
kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: namespace: default name: pod-reader rules: - apiGroups: [""] # "" indicates the core API group resources: ["pods"] verbs: ["get", "watch", "list"] # This role binding allows "jane" to read pods in the "default" namespace. kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: read-pods namespace: default subjects: - kind: User name: jane apiGroup: rbac.authorization.k8s.io roleRef: kind: Role name: pod-reader apiGroup: rbac.authorization.k8s.io
We define a Role that grants the verbs ["get", "watch", list"] over any pod resource, only in the "default" namespace. Then, we create a RoleBinding that grants the permissions defined in "pod-reader" to the user "jane".
Kubernetes RBAC security configuration, authentication and authorization
RBAC configuration: API server flags
First and foremost, make sure that your cluster configuration supports RBAC. The location of the configuration file is your kube-apiserver manifest, and this depends on the deployment method but it's usually inside
/etc/kubernetes/manifests in either the master node(s) or the apiserver pod.
You need to look for this flag:
Node authorization is used by the kubelets, we will discuss kubelet permissions in the 'Securing Kubernetes components' chapter of this guide.
The API server has plenty of flag options, some of them that you should avoid from a Kubernetes security best practices point of view:
--insecure-port: Opens up access to unauthorized, unauthenticated requests, if this parameter is equal to 0, it means no insecure port.
--insecure-bind-address: Ideally, you should avoid insecure connections altogether, but in case you really need them, you can use this parameter to just bind to localhost. Make sure this parameter is not set, or at least not set to a network-reachable IP address.
--anonymous-auth: Enables anonymous requests to the secure port of the API server.
How to create Kubernetes users and serviceAccounts
The key here is to always apply the principle of least privilege:
- Grant the minimum required access privileges for the task that a user or pod need to carry out.
- Prefer Role and RoleBinding to their cluster counterparts, when possible. Is much easier to control security when is bounded to independent namespaces.
- Avoid the use of wildcards
["*"]when defining access to resources or verbs over these resources, be specific.
ServiceAccounts are used to provide an identity to the processes that run in your pods (similar concept to the sshd or www-data users in a Linux system). If you don't specify a serviceAccount, these pods will be assigned to the default service account of their namespace.
Using default serviceAccounts can be vague and prone to oversights, try to use service-specific service accounts instead. This way you can granularly control the API access that you grant to any software entity inside your cluster.
How to create a Kubernetes serviceAccount step by step
Imagine, for example, that your app needs to query the Kubernetes API to retrieve pod information and state changes because you want to notify and send some updates using webhooks.
You just need 'read-only' access and just want to monitor one specific namespace. Using a serviceAccount you can grant these specific privileges (and nothing else) to your software agent.
The default serviceAccount (the one you will get if you don't specify any) is not able to retrieve this information:
$ kubectl auth can-i list pods -n default --as=system:serviceaccount:default:default no
We have created an example deployment to showcase this Kubernetes security feature.
Take a look at the rbac/flask.yaml file:
apiVersion: v1 kind: Namespace metadata: name: flask --- apiVersion: v1 kind: ServiceAccount metadata: name: flask-backend namespace: flask --- kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: name: flask-backend-role namespace: flask rules: - apiGroups: [""] resources: ["pods"] verbs: ["get", "list", "watch"] --- kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: flask-backend-role-binding namespace: flask subjects: - kind: ServiceAccount name: flask-backend namespace: flask roleRef: kind: Role name: flask-backend-role apiGroup: rbac.authorization.k8s.io --- kind: Deployment apiVersion: extensions/v1beta1 metadata: name: flask namespace: flask spec: replicas: 2 template: metadata: labels: app: flask spec: serviceAccount: flask-backend containers: - image: mateobur/flask:latest name: flask ports: - containerPort: 5000
This will create a serviceAccount ("flask backend"), a Role that grants some permissions over the other pods in this "flask" namespace, a RoleBinding associating the serviceAccount and the Role, and finally a deployment of pods that will use the serviceAccount:
$ kubectl create -f flask.yaml
If you query the secrets for the flask namespace, you can verify that an API access token was automatically created for your serviceAccount:
$ kubectl get secrets -n flask NAME TYPE DATA AGE default-token-fjfgn kubernetes.io/service-account-token 3 5m flask-backend-token-68b6q kubernetes.io/service-account-token 3 5m
Then, you can check that the permissions are working as you expect with the
kubectl authcommand, that can query access for verbs, subjects and impersonate other accounts:
$ kubectl auth can-i list pods -n default --as=system:serviceaccount:flask:flask-backend no $ kubectl auth can-i list pods -n flask --as=system:serviceaccount:flask:flask-backend yes $ kubectl auth can-i create pods -n flask --as=system:serviceaccount:flask:flask-backend no
To summarize, you will need to configure a serviceAccount and its related RBAC permissions if your software needs to interact with the hosting Kubernetes cluster. Other examples might include the kubelet agents or a Kubernetes Horizontal Pod Autoscaler.
How to create a Kubernetes user step by step
As we mentioned before, Kubernetes users do not have an explicit API object that you can create, list or modify.
Users are bundled as a parameter of a configuration context that defines the cluster name, (default) namespace and username:
$ kubectl config get-contexts CURRENT NAME CLUSTER AUTHINFO NAMESPACE * kubernetes-admin@kubernetes kubernetes kubernetes-admin
If you take a look at the current context, you will note that the user has client-certificate-data and client-key-data attributes (omitted in the output by default for security reasons).
$ kubectl config view ... users: - name: kubernetes-admin user: client-certificate-data: REDACTED client-key-data: REDACTED
If you have access to the Kubernetes root certification authority, you can generate a new security context that declares a new Kubernetes user.
So, in order to create a new Kubernetes user, let's start creating a new private key:
$ openssl genrsa -out john.key 2048
Then, you need to create a certificate signing request containing the public key and other subject information:
$ openssl req -new -key john.key -out john.csr -subj "/CN=john/O=examplegroup"
Please note that Kubernetes will use the Organization (O=examplegroup) field to determine user group membership for RBAC.
Now, you have to sign this CSR using the root Kubernetes CA, found in
/etc/kubernetes/pki for this example, the file location in your deployment may vary:
# openssl x509 -req -in john.csr -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -out john.crt Signature ok subject=/CN=john/O=examplegroup Getting CA Private Key
You can inspect the new certificate:
# openssl x509 -in john.crt -text Certificate: Data: Version: 1 (0x0) Serial Number: 11309651818125161147 (0x9cf3f46850b372bb) Signature Algorithm: sha256WithRSAEncryption Issuer: CN=kubernetes Validity Not Before: Apr 2 20:20:54 2018 GMT Not After : May 2 20:20:54 2018 GMT Subject: CN=john, O=examplegroup Subject Public Key Info: Public Key Algorithm: rsaEncryption Public-Key: (2048 bit)
Let's repeat this process for a second user, so we can show how to assign RBAC permissions to a group:
$ openssl genrsa -out mary.key 2048 $ openssl req -new -key mary.key -out mary.csr -subj "/CN=mary/O=examplegroup" # openssl x509 -req -in mary.csr -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -out mary.crt
You can now register the new credentials and config context:
$ kubectl config set-credentials john --client-certificate=/home/newusers/john.crt --client-key=/home/newusers/john.key $ kubectl config set-context john@kubernetes --cluster=kubernetes --user=john Context "john@kubernetes" created. $ kubectl config get-contexts CURRENT NAME CLUSTER AUTHINFO NAMESPACE * kubernetes-admin@kubernetes kubernetes kubernetes-admin john@kubernetes kubernetes john
If you want this file to be portable between hosts you need to embed the certificates inline. You can do this automatically appending the
--embed-certs=true parameter to the
kubectl config set-credentials command.
Let's use this new user / context:
$ kubectl config use-context john@kubernetes $ kubectl get pods Error from server (Forbidden): pods is forbidden: User "john" cannot list pods in the namespace "default"
Ok, this is expected because we haven't assigned any RBAC permissions to our "john" user.
Let's go back to our root admin user and create a new clusterrolebinding:
$ kubectl config use-context kubernetes-admin@kubernetes $ kubectl create clusterrolebinding examplegroup-admin-binding --clusterrole=cluster-admin --group=examplegroup clusterrolebinding "examplegroup-admin-binding" created $ kubectl config use-context john@kubernetes $ kubectl get pods NAME READY STATUS RESTARTS AGE flask-cap 1/1 Running 0 1m
Please note that we have assigned these credentials to the group rather than the user, so the user 'mary' should have exactly the same access privileges.
Kubernetes TLS certificates rotation and expiration
Modern Kubernetes deployments and managed cloud Kubernetes providers will properly configure TLS so the communication between the API server and the kubelets, users and pods is already secured, that's why we are going to just focus on the maintenance and rotation aspects of these certificates.
Setting a certificate rotation policy from the start will protect you against the usual key mismanagement or leaking that is bound to happen over long periods of time. This is often overlooked and never-expiring tokens are shared between administrators for convenience reasons.
We are going to cover 3 scenarios:
- kubelet TLS certificate rotation & expiration
- serviceAccount token rotation
- Kubernetes user cert rotation & expiration
It is worth mentioning that the current TLS implementation in the Kubernetes API has no way to verify a certificate besides checking the origin. Neither CRL (Certificate Revocation List) nor OCSP (Online Certificate Status Protocol) are implemented. This means that a lost or exposed certificate will be able to authenticate to the API as long as it hasn't expired.
There are a few ways to mitigate the impact:
- issue (very) short lived certificates to keep the period of potential exposure small
- remove the permissions in RBAC. You cannot re-use the username until the certificate has expired
- recreate the certificate authority and issue new certificates to all active users
- consider OIDC (OpenID Connect) as a alternative authentication method
Kubernetes kubelet TLS certificate rotation
The kubelet is a critical component from the security point of view since it serves as the bridge between the node operating system and the cluster logic. We will discuss specific kubelet security best practices in the next chapters, now we will focus on authentication.
By default the kubelet executable will load its certificates from a regular directory that is passed as argument:
--cert-dir=/var/lib/kubelet/pki/ /var/lib/kubelet/pki# ls kubelet-client.crt kubelet-client.key kubelet.crt kubelet.key
You can regenerate the certs manually using the root CA of your cluster, however, starting from Kubernetes 1.8 there is an automated approach at your disposal.
You can instruct your kubelets to renew their certificates automatically as the expiration date approaches using the config flags:
By default, the kubelet certificates expire in one year, you can tune this parameter passing the flag
--experimental-cluster-signing-duration to the kube-controller-manager binary.
Kubernetes ServiceAccount token rotation
Every time you create a serviceAccount, a Kubernetes secret storing its auth token is automatically generated.
$ kubectl get serviceaccounts NAME SECRETS AGE default 1 26d falco-account 1 18d sysdig-account 1 12d $ kubectl get secrets NAME TYPE DATA AGE default-token-f2lmn kubernetes.io/service-account-token 3 26d falco-account-token-jvgtz kubernetes.io/service-account-token 3 18d sysdig-account-token-9sjgd kubernetes.io/service-account-token 3 12d
You can request new tokens from the API and replace the old ones:
$ kubectl delete secret falco-account-token-jvgtz $ cat > /tmp/rotate-token.yaml <<EOF apiVersion: v1 kind: Secret metadata: name: falco-account-token annotations: kubernetes.io/service-account.name: falco-account type: kubernetes.io/service-account-token EOF
If you describe the new secret, you will be able to see the new token string. Please note that existing pods using this serviceAccount will continue using the old (invalid) token, you may want to plan a rolling update over the affected pods to start using the new token.
Updating serviceAccount tokens is not as common as updating user certs, passwords, etc. There is no fully automated way of doing this other than using the Kubernetes API at the moment. Consider whether or not this security artifact rotation makes sense for your use cases.
Kubernetes user TLS certificate rotation
As we have seen in the 'How to create a Kubernetes user' example, you can assign a certificate to a user, but there is no User API object per se.
When you sign the user certificate using Kubernetes root CA, you can assign an expiration date using the
-days parameter to enforce routinary rotation:
openssl x509 -req -in john.csr -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -days 365 -out john.crt $ openssl x509 -in john.crt -text Certificate: Data: Version: 1 (0x0) Serial Number: 11309651818125161149 (0x9cf3f46850b372bd) Signature Algorithm: sha256WithRSAEncryption Issuer: CN=kubernetes Validity Not Before: Apr 3 10:43:25 2018 GMT Not After : Apr 3 10:43:25 2019 GMT
Then you can replace the old user certificate using the config set-credentials command:
$ kubectl config set-credentials john --client-certificate=/home/newusers/john.crt --client-key=/home/newusers/john.key
In this section we have learned how to enable and configure RBAC permissions in Kubernetes, use credentials for users and services and rotate TLS certificates and tokens, covering the authentication and authorization part of Kubernetes security. Next will continue with Kubernetes Security Context, Pod Security Policies and Kubernetes Network Policy.
Acknowledgements: Daniel Kerwin contributed to this article.
Eager to learn how different security tools compare? Check our webinar Comparing Docker Security tools
We ran a webinar discussing and comparing some of the most interesting container security tools, join this session to learn:
- How to secure Docker containers and what are the best practices
- Why you need to both static and dynamic (run-time) scanning or your images
- What other container security measures and policies we can implement
- Demo: how configure and implement the basics with the most important tools in this list and how they compare
Start Your Free Trial Today