Kubernetes Security Guide, Chapter 1: Kubernetes RBAC and TLS certificates

By on April 4, 2018
Falco GCSCC Kubernetes

RBAC security context is a fundamental part of your Kubernetes security best practices, as well as rolling out TLS certificates / PKI authentication for the core Kubernetes API server. We will learn how to create Kubernetes users, pod service account, set their permissions using RBAC and rotate their TLS certificates and security tokens.

Kubernetes Role-Based Access Control (RBAC)

RBAC is essentially an authorization and access control specification where you define the actions (GET, UPDATE, DELETE, etc) that Kubernetes subjects (i.e. human users, software, kubelets) are allowed to perform over Kubernetes entities (i.e. pods, secrets, nodes).

RBAC uses the "rbac.authorization.k8s.io" API group to drive authorization decisions.

You need to understand its building blocks first:

  • Namespaces: Logical segmentation and isolation, or "virtual clusters". Correct use of Kubernetes namespaces is fundamental for security, as you can group together users, roles and resources according to business logic without granting global privileges for the cluster. Typically you use a namespace to group a project, an application, a team or a customer.

  • Subjects: The security "actors".

    • Regular users: Humans or other authorized accesses from outside the cluster. Kubernetes delegates the user creation and management to the administrator. In practice, this means that you can "refer" to a user, as we will see on the RBAC examples below, but there is no User API object per se.
    • ServiceAccounts: Used to assign permissions to software entities. Kubernetes creates its own default serviceAccounts, and you can create additional ones for your pods/deployments. Any pod run by Kubernetes gets its own privileges through its serviceAccount, applied to all processes run within the containers of that pod.
    • Groups of users. Kubernetes user groups are not explicitly created, instead, the API can implicitly group users using a common property, like the prefix of a serviceAccount or the organization field of a user certificate. As with regular Linux permissions, you can assign RBAC privileges to entire groups.

  • Resources: The entities that will be accessed by the subjects.

    • Resources can refer to a generic entity ("pod", "deployment", etc), subresources like the logs coming from a pod ("pod/log") or the particular resource name like an Ingress: "ingress-controller-istio", including custom resources your deployment defines.
    • Resources can also refer to Pod Security Policies or PSP, we will talk about them later.

  • Role and ClusterRole: A set of permissions over a group of resources, think of it as a "security profile", a Role is always confined to a single namespace, a ClusterRole is cluster-scoped.

    • Before designing your security policy, take into account that Kubernetes RBAC permissions are explicitly additive, there are no "deny" rules.
    • Some resources only make sense at the cluster level (i.e. nodes), you need to create a ClusterRole to control access in this case.
    • Roles define a list of actions that can be performed over the resources or verbs: GET, WATCH, LIST, CREATE, UPDATE, PATCH, DELETE.

  • RoleBindings and ClusterRoleBindings: Grants the permissions defined in a Role or ClusterRole to a subject or group of subjects. Again, RoleBindings are bounded to a certain namespace, ClusterRoleBindings are cluster-global.

Let's start looking at these Role and RoleBinding definitions:

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: default
  name: pod-reader
rules:
- apiGroups: [""] # "" indicates the core API group
  resources: ["pods"]
  verbs: ["get", "watch", "list"]

# This role binding allows "jane" to read pods in the "default" namespace.
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: read-pods
  namespace: default
subjects:
- kind: User
  name: jane
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

We define a Role that grants the verbs ["get", "watch", list"] over any pod resource, only in the "default" namespace. Then, we create a RoleBinding that grants the permissions defined in "pod-reader" to the user "jane".

Kubernetes RBAC security configuration, authentication and authorization

RBAC configuration: API server flags

First and foremost, make sure that your cluster configuration supports RBAC. The location of the configuration file is your kube-apiserver manifest, and this depends on the deployment method but it's usually inside /etc/kubernetes/manifests in either the master node(s) or the apiserver pod.

You need to look for this flag: --authorization-mode=Node,RBAC

Node authorization is used by the kubelets, we will discuss kubelet permissions in the 'Securing Kubernetes components' chapter of this guide.

The API server has plenty of flag options, some of them that you should avoid from a Kubernetes security best practices point of view:

  • --insecure-port: Opens up access to unauthorized, unauthenticated requests, if this parameter is equal to 0, it means no insecure port.
  • --insecure-bind-address: Ideally, you should avoid insecure connections altogether, but in case you really need them, you can use this parameter to just bind to localhost. Make sure this parameter is not set, or at least not set to a network-reachable IP address.
  • --anonymous-auth: Enables anonymous requests to the secure port of the API server.

How to create Kubernetes users and serviceAccounts

The key here is to always apply the principle of least privilege:

  • Grant the minimum required access privileges for the task that a user or pod need to carry out.
  • Prefer Role and RoleBinding to their cluster counterparts, when possible. Is much easier to control security when is bounded to independent namespaces.
  • Avoid the use of wildcards ["*"] when defining access to resources or verbs over these resources, be specific.

ServiceAccounts are used to provide an identity to the processes that run in your pods (similar concept to the sshd or www-data users in a Linux system). If you don't specify a serviceAccount, these pods will be assigned to the default service account of their namespace.

Using default serviceAccounts can be vague and prone to oversights, try to use service-specific service accounts instead. This way you can granularly control the API access that you grant to any software entity inside your cluster.

How to create a Kubernetes serviceAccount step by step

Imagine, for example, that your app needs to query the Kubernetes API to retrieve pod information and state changes because you want to notify and send some updates using webhooks.

You just need 'read-only' access and just want to monitor one specific namespace. Using a serviceAccount you can grant these specific privileges (and nothing else) to your software agent.

The default serviceAccount (the one you will get if you don't specify any) is not able to retrieve this information:

$ kubectl auth can-i list pods -n default --as=system:serviceaccount:default:default
no

We have created an example deployment to showcase this Kubernetes security feature.

Take a look at the rbac/flask.yaml file:

apiVersion: v1
kind: Namespace
metadata:
  name: flask
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: flask-backend
  namespace: flask
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: flask-backend-role
  namespace: flask
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: flask-backend-role-binding
  namespace: flask
subjects:
  - kind: ServiceAccount
    name: flask-backend
    namespace: flask
roleRef:
  kind: Role
  name: flask-backend-role
  apiGroup: rbac.authorization.k8s.io
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: flask
  namespace: flask
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: flask
    spec:
      serviceAccount: flask-backend
      containers:
      - image: mateobur/flask:latest
        name: flask
        ports:
        - containerPort: 5000

This will create a serviceAccount ("flask backend"), a Role that grants some permissions over the other pods in this "flask" namespace, a RoleBinding associating the serviceAccount and the Role, and finally a deployment of pods that will use the serviceAccount:

$ kubectl create -f flask.yaml

If you query the secrets for the flask namespace, you can verify that an API access token was automatically created for your serviceAccount:

$ kubectl get secrets -n flask
NAME                        TYPE                                  DATA      AGE
default-token-fjfgn         kubernetes.io/service-account-token   3         5m
flask-backend-token-68b6q   kubernetes.io/service-account-token   3         5m

Then, you can check that the permissions are working as you expect with the kubectl authcommand, that can query access for verbs, subjects and impersonate other accounts:

$ kubectl auth can-i list pods -n default --as=system:serviceaccount:flask:flask-backend
no

$ kubectl auth can-i list pods -n flask --as=system:serviceaccount:flask:flask-backend
yes

$ kubectl auth can-i create pods -n flask --as=system:serviceaccount:flask:flask-backend
no

To summarize, you will need to configure a serviceAccount and its related RBAC permissions if your software needs to interact with the hosting Kubernetes cluster. Other examples might include the kubelet agents or a Kubernetes Horizontal Pod Autoscaler.

How to create a Kubernetes user step by step

As we mentioned before, Kubernetes users do not have an explicit API object that you can create, list or modify.

Users are bundled as a parameter of a configuration context that defines the cluster name, (default) namespace and username:

$ kubectl config get-contexts
CURRENT   NAME                          CLUSTER      AUTHINFO           NAMESPACE
*         kubernetes-admin@kubernetes   kubernetes   kubernetes-admin

If you take a look at the current context, you will note that the user has client-certificate-data and client-key-data attributes (omitted in the output by default for security reasons).

$ kubectl config view
...
users:
- name: kubernetes-admin
  user:
    client-certificate-data: REDACTED
    client-key-data: REDACTED

If you have access to the Kubernetes root certification authority, you can generate a new security context that declares a new Kubernetes user.

So, in order to create a new Kubernetes user, let's start creating a new private key:

$ openssl genrsa -out john.key 2048

Then, you need to create a certificate signing request containing the public key and other subject information:

$ openssl req -new -key john.key -out john.csr -subj "/CN=john/O=examplegroup"

Please note that Kubernetes will use the Organization (O=examplegroup) field to determine user group membership for RBAC.

Now, you have to sign this CSR using the root Kubernetes CA, found in /etc/kubernetes/pki for this example, the file location in your deployment may vary:

# openssl x509 -req -in john.csr -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -out john.crt
Signature ok
subject=/CN=john/O=examplegroup
Getting CA Private Key

You can inspect the new certificate:

# openssl x509 -in john.crt -text
Certificate:
    Data:
        Version: 1 (0x0)
        Serial Number: 11309651818125161147 (0x9cf3f46850b372bb)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=kubernetes
        Validity
            Not Before: Apr  2 20:20:54 2018 GMT
            Not After : May  2 20:20:54 2018 GMT
        Subject: CN=john, O=examplegroup
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)

Let's repeat this process for a second user, so we can show how to assign RBAC permissions to a group:

$ openssl genrsa -out mary.key 2048
$ openssl req -new -key mary.key -out mary.csr -subj "/CN=mary/O=examplegroup"
# openssl x509 -req -in mary.csr -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -out mary.crt

You can now register the new credentials and config context:

$ kubectl config set-credentials john --client-certificate=/home/newusers/john.crt --client-key=/home/newusers/john.key
$ kubectl config set-context john@kubernetes --cluster=kubernetes --user=john
Context "john@kubernetes" created.

$ kubectl config get-contexts
CURRENT   NAME                          CLUSTER      AUTHINFO           NAMESPACE
*         kubernetes-admin@kubernetes   kubernetes   kubernetes-admin
          john@kubernetes               kubernetes   john

If you want this file to be portable between hosts you need to embed the certificates inline. You can do this automatically appending the --embed-certs=true parameter to the kubectl config set-credentials command.

Let's use this new user / context:

$ kubectl config use-context john@kubernetes
$ kubectl get pods
Error from server (Forbidden): pods is forbidden: User "john" cannot list pods in the namespace "default"

Ok, this is expected because we haven't assigned any RBAC permissions to our "john" user.

Let's go back to our root admin user and create a new clusterrolebinding:

$ kubectl config use-context kubernetes-admin@kubernetes
$ kubectl create clusterrolebinding examplegroup-admin-binding --clusterrole=cluster-admin --group=examplegroup
clusterrolebinding "examplegroup-admin-binding" created

$ kubectl config use-context john@kubernetes
$ kubectl get pods
NAME        READY     STATUS    RESTARTS   AGE
flask-cap   1/1       Running   0          1m

Please note that we have assigned these credentials to the group rather than the user, so the user 'mary' should have exactly the same access privileges.

Kubernetes TLS certificates rotation and expiration

Modern Kubernetes deployments and managed cloud Kubernetes providers will properly configure TLS so the communication between the API server and the kubelets, users and pods is already secured, that's why we are going to just focus on the maintenance and rotation aspects of these certificates.

Setting a certificate rotation policy from the start will protect you against the usual key mismanagement or leaking that is bound to happen over long periods of time. This is often overlooked and never-expiring tokens are shared between administrators for convenience reasons.

We are going to cover 3 scenarios:

  • kubelet TLS certificate rotation & expiration
  • serviceAccount token rotation
  • Kubernetes user cert rotation & expiration

It is worth mentioning that the current TLS implementation in the Kubernetes API has no way to verify a certificate besides checking the origin. Neither CRL (Certificate Revocation List) nor OCSP (Online Certificate Status Protocol) are implemented. This means that a lost or exposed certificate will be able to authenticate to the API as long as it hasn't expired.

There are a few ways to mitigate the impact:

  • issue (very) short lived certificates to keep the period of potential exposure small
  • remove the permissions in RBAC. You cannot re-use the username until the certificate has expired
  • recreate the certificate authority and issue new certificates to all active users
  • consider OIDC (OpenID Connect) as a alternative authentication method

Kubernetes kubelet TLS certificate rotation

The kubelet is a critical component from the security point of view since it serves as the bridge between the node operating system and the cluster logic. We will discuss specific kubelet security best practices in the next chapters, now we will focus on authentication.

By default the kubelet executable will load its certificates from a regular directory that is passed as argument:

--cert-dir=/var/lib/kubelet/pki/

/var/lib/kubelet/pki# ls
kubelet-client.crt  kubelet-client.key  kubelet.crt  kubelet.key

You can regenerate the certs manually using the root CA of your cluster, however, starting from Kubernetes 1.8 there is an automated approach at your disposal.

You can instruct your kubelets to renew their certificates automatically as the expiration date approaches using the config flags:

  • --rotate-certificates

and

  • --feature-gates=RotateKubeletClientCertificate=true

By default, the kubelet certificates expire in one year, you can tune this parameter passing the flag --experimental-cluster-signing-duration to the kube-controller-manager binary.

Kubernetes ServiceAccount token rotation

Every time you create a serviceAccount, a Kubernetes secret storing its auth token is automatically generated.

$ kubectl get serviceaccounts
NAME             SECRETS   AGE
default          1         26d
falco-account    1         18d
sysdig-account   1         12d

$ kubectl get secrets
NAME                         TYPE                                  DATA      AGE
default-token-f2lmn          kubernetes.io/service-account-token   3         26d
falco-account-token-jvgtz    kubernetes.io/service-account-token   3         18d
sysdig-account-token-9sjgd   kubernetes.io/service-account-token   3         12d

You can request new tokens from the API and replace the old ones:

$ kubectl delete secret falco-account-token-jvgtz
$ cat > /tmp/rotate-token.yaml <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: falco-account-token
  annotations:
    kubernetes.io/service-account.name: falco-account
type: kubernetes.io/service-account-token
EOF

If you describe the new secret, you will be able to see the new token string. Please note that existing pods using this serviceAccount will continue using the old (invalid) token, you may want to plan a rolling update over the affected pods to start using the new token.

Updating serviceAccount tokens is not as common as updating user certs, passwords, etc. There is no fully automated way of doing this other than using the Kubernetes API at the moment. Consider whether or not this security artifact rotation makes sense for your use cases.

Kubernetes user TLS certificate rotation

As we have seen in the 'How to create a Kubernetes user' example, you can assign a certificate to a user, but there is no User API object per se.

When you sign the user certificate using Kubernetes root CA, you can assign an expiration date using the -days parameter to enforce routinary rotation:

openssl x509 -req -in john.csr -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -days 365 -out john.crt

$ openssl x509 -in john.crt -text
Certificate:
    Data:
        Version: 1 (0x0)
        Serial Number: 11309651818125161149 (0x9cf3f46850b372bd)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=kubernetes
        Validity
            Not Before: Apr  3 10:43:25 2018 GMT
            Not After : Apr  3 10:43:25 2019 GMT

Then you can replace the old user certificate using the config set-credentials command:

$ kubectl config set-credentials john --client-certificate=/home/newusers/john.crt --client-key=/home/newusers/john.key

Next steps

In this section we have learned how to enable and configure RBAC permissions in Kubernetes, use credentials for users and services and rotate TLS certificates and tokens, covering the authentication and authorization part of Kubernetes security. Next will continue with Kubernetes Security Context, Pod Security Policies and Kubernetes Network Policy.

Acknowledgements: Daniel Kerwin contributed to this article.




Eager to learn how different security tools compare? Check our webinar Comparing Docker Security tools

We ran a webinar discussing and comparing some of the most interesting container security tools, join this session to learn:

  • How to secure Docker containers and what are the best practices
  • Why you need to both static and dynamic (run-time) scanning or your images
  • What other container security measures and policies we can implement
  • Demo: how configure and implement the basics with the most important tools in this list and how they compare

Join Comparing Docker Security tools webinar


Stay up to date!

Get new articles from this blog (weekly)
Or container ecosystem updates (monthly)

Thanks so much for signing up!
Please check your inbox for a confirmation email.