If you want to keep your Kubernetes environment secure, you need to work with Kubernetes Secrets securely. Improper Secrets management can lead to breaches of your clusters, your workloads, or both.

Keep reading for tips on managing Kubernetes Secrets, including how to encrypt Secrets data, how to create and view Secrets, and when it makes sense to use a third-party vault.

What Are Secrets in Kubernetes?

A Kubernetes Secret is sensitive information – such as a password or access key – that is used within a Kubernetes environment.

Secrets are not unique to Kubernetes, of course. You use secrets data in virtually every type of modern application environment or platform.

However, what makes Secrets in Kubernetes different is that Kubernetes provides some built-in tooling to help admins work with Secrets in a secure way. This tooling makes it possible to store sensitive data independently of pods, while making it accessible to pods when they need to reference it.

Are Kubernetes Secrets Secure?

When properly managed, Kubernetes Secrets are quite secure. That’s the whole point of Secrets, after all: to provide a secure means of managing passwords, access keys, and other sensitive information that applications running inside Kubernetes may require.

Without Kubernetes Secrets management, you’d have to hard-code data like passwords into pod specifications or container images in many cases. Doing so would be pretty insecure because anyone who can read that data (or who can intercept it when it passes over the network) would gain access to sensitive information.

Secrets management addresses this problem by making it possible to store secrets in Kubernetes’s key-value store, Etcd, where it is safer from unauthorized access.

How to Encrypt and Protect Kubernetes Secrets

We said above that Kubernetes Secrets are secure when they are properly managed. The keyword there is “properly managed” – and to manage secrets data properly in Kubernetes, you need to take some extra steps.

By default, Secrets are not especially secure. The fact that they are stored in Etcd makes them somewhat safer than baking them into pod specifications or container images because, in general, fewer human or machine users will have access to Etcd than to pods or images.

However, by default, Secrets stored in Etcd are neither encrypted nor protected with role-based access controls. Thus, anyone who does have access to Etcd – which, in general, means any human or machine entity who can interact with your Kubernetes cluster API – can easily view sensitive Secrets. So, potentially, could anyone who has access to your Kubernetes master nodes at the operating system level, since Etcd runs on those Kubernetes nodes.

To protect against this risk, you should take two key steps: encrypt Etcd Secrets and apply RBAC rules.

Kubernetes Secrets Encryption

When you encrypt Secrets, Kubernetes stores the Secrets in encoded form in Etcd and makes the data available to pods via keys. Thus, anyone who can access Etcd will only be able to view the encoded version of your Secrets, not the actual Secrets. At the same time, though, pods that you authorize to view the Secrets will be supplied with the Secrets data.

To enforce this type of setup, create an encryption configuration file that includes the encoded secrets and corresponding access keys. For example:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: dGhpcyBpcyBwYXNzd29yZA==
      - identity: {}

After creating an encryption configuration file, restart the Kubernetes API server. After that, any new Secrets you create in Kubernetes will be encrypted by default.

Secure Secrets with RBAC Rules

You can add another layer of protection to Kubernetes Secrets by using Kubernetes’s RBAC framework to restrict who can view Secrets in the first place. This isn’t an alternative to encrypting Secrets – you should do that, too – but RBAC restrictions help to ensure that even if a Secret is not encrypted for some reason, unauthorized users can’t view it.

To restrict access to Secrets, create Roles or ClusterRoles for whichever human or machine users you do want to be able to view Secrets. For example:

kubectl create clusterrole secretadmin \
          --verb=get --verb=list --verb=create --verb=update  \
          --resource=secret \
          --namespace=mysuperprojectCode language: PHP (php)

This command creates a ClusterRole for a user called secretadmin, who can view Secrets.

Then create a RoleBinding or ClusterRoleBinding to enforce the rule. For example:

kubectl create clusterrolebinding canadminsecret \
          --role=secretadmin \
          --serviceaccount=mysuperproject:thepowerfulapp \
          --namespace=mysuperprojectCode language: PHP (php)

Working with Kubernetes Secrets

Whether your Secrets are encrypted and secured via RBAC rules or not, the commands for creating, viewing, and managing Secrets are mostly the same. The following are the basics.

Create a New Kubernetes Secret

You can create a new Secret and store it in Etcd with a command such as:

kubectl create secret generic secret-name

This creates what Kubernetes calls an “opaque” Secret, which is a generic user-defined Secret. Kubernetes also defines other types of special Secrets, such as tokens for service accounts.

Alternatively, you can declare secrets using YAML. For example:

apiVersion: v1
kind: Secret
metadata:
name: mysecret
type: Opaque
data:
username: YWRtaW4=
password: MWYyZDFlMmU2N2Rm

View Kubernetes Secrets

To view an existing Secret, use:

kubectl get secret secret-nameCode language: JavaScript (javascript)

Kubectl will print the Secret in base64 encoded format. You’ll need to use an external base64 decoder (like the Linux utility base64) to decode the Secret fully, using a command such as:

echo $encoded_secret | base64 -dCode language: PHP (php)

Note that if you’ve created RBAC restrictions on who can view Secrets, only users with the requisite permissions will be able to access Secrets this way.

Using a Secrets Vault

As an alternative to storing Secrets in Etcd, you can use what’s known as a secrets vault. A secrets vault is a third-party tool that stores Kubernetes Secrets data (and sometimes secrets for other systems, too) and makes it available to pods as needed.

The main reasons to use a secrets vault include:

• Avoid storing secrets in Etcd: While Etcd can effectively store secrets data, that is not the only or primary thing it is designed to do. It’s a generic key-value store that houses all of the data required to manage a cluster. Thus, using a secrets vault lets you separate Secrets from other, unrelated data.
• Store secrets securely by default: Most secrets vaults are configured with strong security controls by default – unlike Etcd, which, as we have noted, does not encrypt data until you tell it to.
• Centralize secrets management: Most secrets vaults can manage not just Kubernetes Secrets, but also secrets for a wide range of other environments. Most vaults that support Kubernetes also let you manage secrets for multiple clusters using one vault instance. Thus, using an external secrets vault helps you to centralize secrets management across your entire IT estate and manage them all in a consistent way.

For relatively small-scale Kubernetes environments, a secrets vault may be overkill. But for large-scale environments where you need to manage hundreds of Secrets, secrets vaults can typically do it more efficiently and securely than Etcd.

Installing a Secrets Vault in Kubernetes

The process for setting up a secrets vault depends on which secrets vault you choose. In most cases, however, the main steps include:

1. Deploy the secrets vault into Kubernetes. Usually, you can do this using a Helm chart.
2. Connect your Kubernetes cluster (or clusters) to the secrets vault.
3. Customize your configuration as desired.

Popular secrets vaults that support Kubernetes include HashiCorp Vault and CyberArk Conjur.

Conclusion

Kubernetes provides relatively robust functionality for managing Secrets securely via Etcd. However, you must be sure to enable encryption and RBAC controls in order to avoid major security pitfalls when relying on Etcd for Secrets management.

For large-scale or complex Kubernetes Secrets management needs, consider using a third-party secrets vault, which provides more security and efficiency than Etcd.