Kubernetes RBAC and TLS certificates – Kubernetes Security Guide (Part 1)

By on April 4, 2018
Falco GCSCC Kubernetes

Kubernetes RBAC security context is a fundamental part of your Kubernetes security best practices, as well as rolling out TLS certificates / PKI authentication for connecting to the Kubernetes API server and between its components. We will learn how to create a user in Kubernetes, set Kubernetes permissions using RBAC and setup/rotate TLS certificates and security tokens.

The 2nd part of this Kubernetes Security guide will focus on Kubernetes Security Context, Kubernetes Security Policy and Kubernetes Network Policy while part 3 goes on Securing Kubernetes components: kubelet, etcd and Docker registry.

Kubernetes Role-Based Access Control (RBAC)

Kubernetes RBAC is essentially an authorization and access control specification where you define the actions (GET, UPDATE, DELETE, etc) that Kubernetes subjects (i.e. human users, software, kubelets) are allowed to perform over Kubernetes entities (i.e. pods, secrets, nodes).

RBAC uses the "rbac.authorization.k8s.io" API group to drive authorization decisions. Before getting started, it is important to understand the API group building blocks:

  • Namespaces: Logical segmentation and isolation, or "virtual clusters". Correct use of Kubernetes namespaces is fundamental for security, as you can group together users, roles and resources according to business logic without granting global privileges for the cluster. Typically you use a namespace to group a project, an application, a team or a customer.

  • Subjects: The security "actors".

    • Regular users: Humans or other authorized accesses from outside the cluster. Kubernetes delegates the user creation and management to the administrator. In practice, this means that you can "refer" to a user, as we will see on the Kubernetes RBAC examples below, but there is no User API object per se.
    • ServiceAccounts: Used to assign permissions to software entities. Kubernetes creates its own default serviceAccounts, and you can create additional ones for your pods/deployments. Any pod run by Kubernetes gets its own privileges through its serviceAccount, applied to all processes run within the containers of that pod.
    • Groups of users. Kubernetes user groups are not explicitly created, instead, the API can implicitly group users using a common property, like the prefix of a serviceAccount or the organization field of a user certificate. As with regular Linux permissions, you can assign RBAC privileges to entire groups.

  • Resources: The entities that will be accessed by the subjects.

    • Resources can refer to a generic entity ("pod", "deployment", etc), subresources such as the logs coming from a pod ("pod/log") or the particular resource name like an Ingress: "ingress-controller-istio", including custom resources your deployment defines.
    • Resources can also refer to Pod Security Policies or PSP.

  • Role and ClusterRole: A set of permissions over a group of resources, think of it as a "security profile", a Role is always confined to a single namespace, a ClusterRole is cluster-scoped.

    • Before designing your security policy, take into account that Kubernetes RBAC permissions are explicitly additive, there are no "deny" rules.
    • Some resources only make sense at the cluster level (i.e. nodes), you need to create a ClusterRole to control access in this case.
    • Roles define a list of actions that can be performed over the resources or verbs: GET, WATCH, LIST, CREATE, UPDATE, PATCH, DELETE.

  • RoleBindings and ClusterRoleBindings: Grants the permissions defined in a Role or ClusterRole to a subject or group of subjects. Again, RoleBindings are bounded to a certain namespace, ClusterRoleBindings are cluster-global.

Let's start looking at these Role and RoleBinding definitions:

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
  namespace: default
  name: pod-reader
- apiGroups: [""] # "" indicates the core API group
  resources: ["pods"]
  verbs: ["get", "watch", "list"]

# This role binding allows "jane" to read pods in the "default" namespace.
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
  name: read-pods
  namespace: default
- kind: User
  name: jane
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

We define a Role that grants the verbs ["get", "watch", list"] over any pod resource, only in the "default" namespace. Then, we create a RoleBinding that grants the permissions defined in "pod-reader" to the user "jane".

Kubernetes RBAC security configuration, authentication and authorization

RBAC configuration: API server flags

Start by making sure your cluster configuration supports RBAC. The location of the configuration file is your kube-apiserver manifest, and this depends on the deployment method but it's usually inside /etc/kubernetes/manifests in either the master node(s) or the apiserver pod.

Look for this flag: --authorization-mode=Node,RBAC

Node authorization is used by the kubelets, we will discuss kubelet permissions in the 'Securing Kubernetes components' chapter of this guide.

While the API server has plenty of flag options, some are best avoided when taking a best practices approach to security:

  • --insecure-port: Opens up access to unauthorized, unauthenticated requests, if this parameter is equal to 0, it means no insecure port.
  • --insecure-bind-address: Ideally, you should avoid insecure connections altogether, but in case you really need them, you can use this parameter to just bind to localhost. Make sure this parameter is not set, or at least not set to a network-reachable IP address.
  • --anonymous-auth: Enables anonymous requests to the secure port of the API server.

How to create Kubernetes users and serviceAccounts

When it comes to Kubernetes users and permissions, the best approach is one that applies the principle of least privilege, which promotes minimal user profile privileges based on users' job necessities.:

  • Grant the minimum required access privileges for the task that a user or pod need to carry out.
  • Prefer Role and RoleBinding to their cluster counterparts, when possible. Is much easier to control security when is bounded to independent namespaces.
  • Avoid the use of wildcards ["*"] when defining access to resources or verbs over these resources, be specific.

ServiceAccounts are used to provide an identity to the processes that run in your pods (similar concept to the sshd or www-data users in a Linux system). If you don't specify a serviceAccount, these pods will be assigned to the default service account of their namespace.

Using default serviceAccounts can be vague and prone to oversights, try to use service-specific service accounts instead. This way you can granularly control the API access that you grant to any software entity inside your cluster.

How to create a Kubernetes serviceAccount step by step

Imagine, for example, that your app needs to query the Kubernetes API to retrieve pod information and state changes because you want to notify and send some updates using webhooks.

You just need 'read-only' access to monitor one specific namespace. Using a serviceAccount you can grant these specific privileges (and nothing else) to your software agent.

The default serviceAccount (the one you will get if you don't specify any) is not able to retrieve this information:

$ kubectl auth can-i list pods -n default --as=system:serviceaccount:default:default

We have created an example deployment to showcase this Kubernetes security feature.

Take a look at the rbac/flask.yaml file:

apiVersion: v1
kind: Namespace
  name: flask
apiVersion: v1
kind: ServiceAccount
  name: flask-backend
  namespace: flask
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
  name: flask-backend-role
  namespace: flask
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
  name: flask-backend-role-binding
  namespace: flask
  - kind: ServiceAccount
    name: flask-backend
    namespace: flask
  kind: Role
  name: flask-backend-role
  apiGroup: rbac.authorization.k8s.io
kind: Deployment
apiVersion: extensions/v1beta1
  name: flask
  namespace: flask
  replicas: 2
        app: flask
      serviceAccount: flask-backend
      - image: mateobur/flask:latest
        name: flask
        - containerPort: 5000

This will create a serviceAccount ("flask backend"), a Role that grants some permissions over the other pods in this "flask" namespace, a RoleBinding associating the serviceAccount and the Role, and finally a deployment of pods that will use the serviceAccount:

$ kubectl create -f flask.yaml

If you query the secrets for the flask namespace, you can verify that an API access token was automatically created for your serviceAccount:

$ kubectl get secrets -n flask
NAME                        TYPE                                  DATA      AGE
default-token-fjfgn         kubernetes.io/service-account-token   3         5m
flask-backend-token-68b6q   kubernetes.io/service-account-token   3         5m

Then, you can check that the permissions are working as you expect with the kubectl authcommand, that can query access for verbs, subjects and impersonate other accounts:

$ kubectl auth can-i list pods -n default --as=system:serviceaccount:flask:flask-backend

$ kubectl auth can-i list pods -n flask --as=system:serviceaccount:flask:flask-backend

$ kubectl auth can-i create pods -n flask --as=system:serviceaccount:flask:flask-backend

To summarize, you will need to configure a serviceAccount and its related Kubernetes RBAC permissions if your software needs to interact with the hosting Kubernetes cluster. Other examples might include the kubelet agents or a Kubernetes Horizontal Pod Autoscaler.

How to create a Kubernetes user step by step

As we mentioned before, Kubernetes users do not have an explicit API object that you can create, list or modify.

Users are bundled as a parameter of a configuration context that defines the cluster name, (default) namespace and username:

$ kubectl config get-contexts
CURRENT   NAME                          CLUSTER      AUTHINFO           NAMESPACE
*         kubernetes-admin@kubernetes   kubernetes   kubernetes-admin

If you look at the current context, you will note that the user has client-certificate-data and client-key-data attributes (omitted in the output by default for security reasons).

$ kubectl config view
- name: kubernetes-admin
    client-certificate-data: REDACTED
    client-key-data: REDACTED

If you have access to the Kubernetes root certification authority, you can generate a new security context that declares a new Kubernetes user.

So, in order to create a new Kubernetes user, let's start creating a new private key:

$ openssl genrsa -out john.key 2048

Then, you need to create a certificate signing request containing the public key and other subject information:

$ openssl req -new -key john.key -out john.csr -subj "/CN=john/O=examplegroup"

Note that Kubernetes will use the Organization (O=examplegroup) field to determine user group membership for RBAC.

You will sign this CSR using the root Kubernetes CA, found in /etc/kubernetes/pki for this example, the file location in your deployment may vary:

# openssl x509 -req -in john.csr -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -out john.crt
Signature ok
Getting CA Private Key

You can inspect the new certificate:

# openssl x509 -in john.crt -text
        Version: 1 (0x0)
        Serial Number: 11309651818125161147 (0x9cf3f46850b372bb)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=kubernetes
            Not Before: Apr  2 20:20:54 2018 GMT
            Not After : May  2 20:20:54 2018 GMT
        Subject: CN=john, O=examplegroup
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)

Let's repeat this process for a second user, so we can show how to assign Kubernetes RBAC permissions to a group:

$ openssl genrsa -out mary.key 2048
$ openssl req -new -key mary.key -out mary.csr -subj "/CN=mary/O=examplegroup"
# openssl x509 -req -in mary.csr -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -out mary.crt

You can now register the new credentials and config context:

$ kubectl config set-credentials john --client-certificate=/home/newusers/john.crt --client-key=/home/newusers/john.key
$ kubectl config set-context john@kubernetes --cluster=kubernetes --user=john
Context "john@kubernetes" created.

$ kubectl config get-contexts
CURRENT   NAME                          CLUSTER      AUTHINFO           NAMESPACE
*         kubernetes-admin@kubernetes   kubernetes   kubernetes-admin
          john@kubernetes               kubernetes   john

If you want this file to be portable between hosts you need to embed the certificates inline. You can do this automatically appending the --embed-certs=true parameter to the kubectl config set-credentials command.

Let's use this new user / context:

$ kubectl config use-context john@kubernetes
$ kubectl get pods
Error from server (Forbidden): pods is forbidden: User "john" cannot list pods in the namespace "default"

Ok, this is expected because we haven't assigned any RBAC permissions to our "john" user.

Let's go back to our root admin user and create a new clusterrolebinding:

$ kubectl config use-context kubernetes-admin@kubernetes
$ kubectl create clusterrolebinding examplegroup-admin-binding --clusterrole=cluster-admin --group=examplegroup
clusterrolebinding "examplegroup-admin-binding" created

$ kubectl config use-context john@kubernetes
$ kubectl get pods
flask-cap   1/1       Running   0          1m

Please note that we have assigned these credentials to the group rather than the user, so the user 'mary' should have exactly the same access privileges.

Kubernetes TLS certificates rotation and expiration

Modern Kubernetes deployments and managed cloud Kubernetes providers will properly configure TLS so the communication between the API server and the kubelets, users and pods is already secured. Thus, we will focus only on the maintenance and rotation aspects of these certificates.

Setting a certificate rotation policy from the start will protect you against the usual key mismanagement or leaking that is bound to happen over long periods of time - an occurrence that is often overlooked.

Let’s explore three Kubernetes TLS certificate rotation and expiration scenarios:

  • kubelet TLS certificate rotation & expiration
  • serviceAccount token rotation
  • Kubernetes user cert rotation & expiration

Note that the current TLS implementation in the Kubernetes API has no way to verify a certificate besides checking the origin. Neither CRL (Certificate Revocation List) nor OCSP (Online Certificate Status Protocol) are implemented. This means that a lost or exposed certificate will be able to authenticate to the API as long as it hasn't expired.

There are a few ways to mitigate the impact:

  • issue (very) short lived certificates to keep the period of potential exposure small
  • remove the permissions in Kubernetes RBAC. You cannot re-use the username until the certificate has expired
  • recreate the certificate authority and issue new certificates to all active users
  • consider OIDC (OpenID Connect) as a alternative authentication method

Kubernetes kubelet TLS certificate rotation

The kubelet serves as the bridge between the node operating system and the cluster logic and thus is a critical security component.

By default the kubelet executable will load its certificates from a regular directory that is passed as argument:


/var/lib/kubelet/pki# ls
kubelet-client.crt  kubelet-client.key  kubelet.crt  kubelet.key

You can regenerate the certs manually using the root CA of your cluster, however, starting from Kubernetes 1.8 there is an automated approach at your disposal.

You can instruct your kubelets to renew their certificates automatically as the expiration date approaches using the config flags:

  • --rotate-certificates


  • --feature-gates=RotateKubeletClientCertificate=true

By default, the kubelet certificates expire in one year, you can tune this parameter passing the flag --experimental-cluster-signing-duration to the kube-controller-manager binary.

Kubernetes ServiceAccount token rotation

Every time you create a serviceAccount, a Kubernetes secret storing its auth token is automatically generated.

$ kubectl get serviceaccounts
NAME             SECRETS   AGE
default          1         26d
falco-account    1         18d
sysdig-account   1         12d

$ kubectl get secrets
NAME                         TYPE                                  DATA      AGE
default-token-f2lmn          kubernetes.io/service-account-token   3         26d
falco-account-token-jvgtz    kubernetes.io/service-account-token   3         18d
sysdig-account-token-9sjgd   kubernetes.io/service-account-token   3         12d

You can request new tokens from the API and replace the old ones:

$ kubectl delete secret falco-account-token-jvgtz
$ cat > /tmp/rotate-token.yaml <<EOF
apiVersion: v1
kind: Secret
  name: falco-account-token
    kubernetes.io/service-account.name: falco-account
type: kubernetes.io/service-account-token

If you describe the new secret, you will be able to see the new token string. Note that existing pods using this serviceAccount will continue using the old (invalid) token, you may want to plan a rolling update over the affected pods to start using the new token.

Updating serviceAccount tokens is not as common as updating user certs, passwords, etc. There is no fully automated way of doing this other than using the Kubernetes API at the moment. Consider whether or not this security artifact rotation makes sense for your use cases.

Kubernetes user TLS certificate rotation

As we have seen in the 'How to create a Kubernetes user' example, you can assign a certificate to a user, but there is no User API object per se.

When you sign the user certificate using Kubernetes root CA, you can assign an expiration date using the -days parameter to enforce routinary rotation:

openssl x509 -req -in john.csr -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -days 365 -out john.crt

$ openssl x509 -in john.crt -text
        Version: 1 (0x0)
        Serial Number: 11309651818125161149 (0x9cf3f46850b372bd)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=kubernetes
            Not Before: Apr  3 10:43:25 2018 GMT
            Not After : Apr  3 10:43:25 2019 GMT

Then you can replace the old user certificate using the config set-credentials command:

$ kubectl config set-credentials john --client-certificate=/home/newusers/john.crt --client-key=/home/newusers/john.key

Next steps

In this part of the Kubernetes Security guide we have learned how to enable and configure Kubernetes RBAC permissions, use credentials for users and services and rotate TLS certificates and tokens, covering the authentication and authorization part of Kubernetes security.

Now you are ready to move to the next part in this series, on how to configure Kubernetes Pod Security with Kubernetes Security Context, Kubernetes Security Policy and and Kubernetes Network Policy..

Acknowledgements: Daniel Kerwin contributed to this article.

Eager to learn more? Check out our online session: Kubernetes Security Features

Kubernetes security features focus on infrastructure layer: verifying that configuration is safe, follow the security best practices, the last privileges and your deployment has no known issues. How does that compare with runtime security?

Stay up to date!

Get new articles from this blog (weekly)
Or container ecosystem updates (monthly)

Thanks so much for signing up!
Please check your inbox for a confirmation email.