Helm is being used broadly to deploy Kubernetes applications as it is an easy way to publish and consume them via a couple of commands, as well as integrate them in your GitOps pipeline. But is Helm security taken seriously? Can you trust it blindly?
This post explains the benefits of using Helm, the pitfalls, and offers a few recommendations for Helm security. Let’s get started!
Why Helm?
Helm is a graduated open-source CNCF project originally created by DeisLabs. If you want to know more about how it works, we recommend you read the Helm 101 article of our Learning Cloud Native hub.
Let’s face it, managing an application lifecycle on Kubernetes is hard. If you want to just deploy an application, it requires at least a Deployment, usually tweaking the application configuration via modifications performed to a ConfigMap or a Secret, deploying CRDs if needed, and more. Clearly, it’s not really straightforward.
You can create your own deployment files with some environment variables and use envsubst to substitute them at runtime (envsubst < deploy.yml | kubectl apply -f -) to have a “DIY templating engine,” but that is probably not an optimal solution.
Kustomize improves the previous DIY solution but it has some limitations as well (it is mainly focused on templating, not packaging). Jsonnet can also be used for templating.
Helm security is not a priority and it is largely up to the user to make good use of it. Helm is not perfect, but it tries to make that process easier by providing a simple command interface, a repository with more than 9000 charts available called Artifact Hub (and the ability to host your own charts on your own repository), and a templating engine (with over 60 available functions, mostly based on the Go template language). That allows you to package complex applications to make them easily deployable by just providing specific parameters.
For example, you can deploy a whole MySQL cluster with replication enabled (a non-trivial task, let’s be honest) by just using the architecture=replication parameter.
It also has some advanced features, such as hooks (to run specific tasks at specific points on the deployment process such as ‘pre-install’), and can be integrated with GitOps tools, such as ArgoCD or Flux. You can leverage library charts or named templates, and even run post-render tasks (e.g., to run Kustomize).
Helm security – How to secure Helm
We’ve covered a lot of ground, but we didn’t pay any attention to any Helm security aspects and most charts are not secure by default.
There are a few angles to tackle depending on the process we want to cover. Are we just consuming the Helm charts, the Kubernetes objects created by the charts, or are we talking about custom Helm charts?
Custom Helm charts
If writing your own Helm charts, a few general recommendations apply, as well as some security focused ones:
- Store the charts in a Git repository. This may seem obvious in 2022, but Git will give you some benefits just by using it, such as easy rollbacks or the ability to track changes.
- Store the Helm charts in a proper repository. Charts can be served via HTTP but everything is HTTPS these days, right?
- Use
helm lintor any other linter you prefer to verify the Helm charts are properly formed. You don’t want to break the production environment for a silly typo.For example, in this basic Helm chart file without a proper
version, Helm lint complains about it:apiVersion: v2 name: hello-world description: A Helm chart for Kubernetes type: application appVersion: "0.0.1" $ helm lint --strict ==> Linting . [ERROR] Chart.yaml: version is required [INFO] Chart.yaml: icon is recommended [ERROR] templates/: validation: chart.metadata.version is required [ERROR] : unable to load chart validation: chart.metadata.version is required Error: 1 chart(s) linted, 1 chart(s) failedCode language: JavaScript (javascript)
- Use consistent versioning on your charts (Helm follows the SemVer2 standard). It is helpful for reproducibility and to be able to respond quickly in a situation where you need to update your charts because a vulnerability has been found. If your charts are unversioned or using “latest”, which one would you update?
There are two different versions you can use: the version of the chart itself (versionin theChart.yamlfile) and the version of the application (appVersion).$ helm show chart falcosecurity/falco | grep -E '^version|^appVersion' appVersion: 0.33.0 version: 2.2.0Code language: JavaScript (javascript)Don’t forget to Keep a Changelog (like Falco does).
- Create test scenarios for your Helm charts to cover your use cases. The idea is to validate the success of the Helm deployment by creating Kubernetes objects (as in Helm templates) that will test your deployed chart by running
helm test <RELEASE_NAME>. For example, a test can be just a simple pod running on the same namespace where your application has been deployed, that queries your application API to see if it has been deployed properly:
apiVersion: v1
kind: Pod
metadata:
…
annotations:
"helm.sh/hook": test
spec:
containers:
- name: wget
image: busybox
command: ['wget']
args: ['{{ .Values.service.name }}:{{ .Values.service.port }}']
restartPolicy: NeverCode language: JavaScript (javascript)Usually, tests are stored in the templates/tests/ folder and are required to have the “helm.sh/hook": test annotation to identify themselves as tests.
$ helm test hello-world
NAME: hello-world
...
Phase: Succeeded
$ kubectl get po -n hello-world
NAME READY STATUS RESTARTS AGE
hello-world-78b98b4c85-kbt58 1/1 Running 0 91s
hello-world-test 0/1 Completed 0 67sCode language: JavaScript (javascript)- Sign your charts easily with
helm package –sign(and verify them withhelm install --verify). Asserting the integrity of the software components is the most common task when securing the software supply chain. This usually means verifying a digital signature (either included with the software itself or close to it). Helm uses a PGP-based digital signature to create provenance records stored in provenance files (.prov), which are stored alongside a packaged chart. Let’s see an example:
$ helm package --sign --key 'Eduardo Minguez' hello-world --keyring ~/.gnupg/secring.gpg
Password for key "Eduardo Minguez (gpg key) <[email protected]>" >
Successfully packaged chart and saved it to: /home/edu/git/my-awesome-stuff/hello-world-0.0.1.tgzCode language: JavaScript (javascript)And this is what the provenance file looks like:
$ cat hello-world-0.0.1.tgz.prov
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
...
name: hello-world
...
files:
hello-world-0.0.1.tgz: sha256:b3f75d753ffdd7133765c9a26e15b1fa89784e18d9dbd8c0c51037395eeb332e
-----BEGIN PGP SIGNATURE-----
wsFcB…
-----END PGP SIGNATURE-----%If the signature doesn’t match, Helm will complain:
$ helm verify hello-world-0.0.1.tgz
Error: openpgp: invalid signature: hash tag doesn't matchCode language: JavaScript (javascript)Running helm install --verify will automatically check the provenance files:
$ helm install --verify myrepo/mychart-1.2.3Code language: JavaScript (javascript)Or, you can just pull the chart and verify it:
$ helm pull --verify myrepo/mychart-1.2.3Code language: JavaScript (javascript)The public key needs to be trusted beforehand for --verify to work, so you must make it publicly available somewhere, otherwise it will fail:
$ helm pull --verify myrepo/mychart-1.2.3
Error: openpgp: signature made by unknown entity
$ cat security/pubkey.gpg | gpg --import --batch
$ helm pull --verify myrepo/mychart-1.2.3
Signed by:..
Code language: JavaScript (javascript)There is also a sigstore Helm plugin to use Rekor as signature storage, which is even better.
- Automate all the previous steps (testing, versioning, signing, and releasing) in a CI/CD pipeline to make sure they are consistent with the best practices on every change, and to avoid potential problems when doing manual changes.
You can use the helm/charts-repo-actions-demo for inspiration on how to create a GitHub actions workflow to test and release a chart:
- name: Run chart-releaser
uses: helm/chart-releaser-action@v1.4.0
with:
charts_dir: charts
config: cr.yaml
env:
CR_TOKEN: "${{ secrets.GITHUB_TOKEN }}"Code language: JavaScript (javascript)Kubernetes objects
When creating the Kubernetes objects via templates, Helm doesn’t provide any security measures out of the box. You are on your own and can apply any bad practice you want, such as deploying a container with a root user or with full capabilities (OK, you may want to do it). Let’s talk about some recommendations:
- Use Role-based access control (RBAC) to limit the object’s permissions (don’t use
cluster-adminfor everything). For example, the falcosidekick Helm chart creates a Role, ServiceAccount, and RoleBinding to minimize the required permissions used in the K8s Deployment:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: {{ include "falcosidekick.fullname" . }}
…
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: {{ include "falcosidekick.fullname" . }}
…
rules:
- apiGroups:
- ""
resources:
- endpoints
verbs:
- get
…
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
…
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: {{ include "falcosidekick.fullname" . }}
subjects:
- kind: ServiceAccount
name: {{ include "falcosidekick.fullname" . }}
…
Code language: JavaScript (javascript)- Provide sane defaults. For example, if your chart includes a MySQL pod, don’t use a default password so any user could know about it. Instead, generate it randomly or force the user to specify it. However, there are a couple of things to consider, including how to deal with upgrades in this GitHub issue and this blog post. You can use the lookup function and the resource policy annotation to prevent overwriting when upgrading, as follows:
{{- if not (lookup "v1" "Secret" .Release.Namespace "hello-world") }}
apiVersion: v1
kind: Secret
metadata:
name: mysecret
annotations:
"helm.sh/resource-policy": "keep"
type: Opaque
stringData:
password: {{ randAlphaNum 24 }}
{{- end }}
Code language: JavaScript (javascript)- Reduce the attack surface. Make the deployment small and enable components and features using flags or values if needed. For example, the Falco Helm chart doesn’t deploy falcosidekick by default, but you can enable it easily:
falcosidekick:
# -- Enable falcosidekick deployment.
enabled: false
Code language: JavaScript (javascript)Which is then used in the Chart:
dependencies:
- name: falcosidekick
condition: falcosidekick.enabled
Code language: JavaScript (javascript)All the rest of Kubernetes recommendations apply (including the CIS benchmarks, for example), so make sure you scan your Kubernetes object definitions for best practices. If your preferred tool doesn’t support Helm charts, don’t worry. You can always render the Kubernetes objects in a previous step with the Helm template command, as follows:
$ helm template falco falcosecurity/falco --namespace falco --create-namespace --set driver.kind=ebpf > all.yaml
Code language: JavaScript (javascript)And then verify them as:
$ myawesometool --verify all.yaml
Code language: JavaScript (javascript)Using Helm charts
- Don’t trust the Helm charts blindly, especially third-party ones. Fortunately, as we’ve seen, the
helm templatecommand renders and outputs the Kubernetes objects created by the Helm chart, so it is a good practice to at least take a quick look at the results before deploying it in your Kubernetes cluster. You probably don’t want to use joaquinito2051‘s charts. - As explained before, use
helm verifyto check the digital signatures of the charts you use to make sure you are using the charts you are supposed to. - Uninstall unused releases: If you’re no longer using a Helm release, uninstall it to reduce your attack surface.
- Always try to keep the Helm Charts you use updated (as well as the
helmbinary and their plugins!). Let’s face it, mistakes and bugs happen so it is a good idea to always use the latest version with the latest fixes for both the Helm chart itself or for the objects the Helm chart creates (for example, if it uses a container image that has been found vulnerable). This also applies to subcharts. There are a couple of options to verify what an upgrade will change, including using the helm diff plugin:
$ helm diff --install foo --set image.tag=1.14.0 .
Code language: JavaScript (javascript)Or the ability to render the manifests via helm template and use kubectl to make the diffs:
$ helm template --is-upgrade --no-hooks --skip-crds foo --set image.tag=1.14.0 . | kubectl diff --server-side=false -f -
Code language: JavaScript (javascript)However, there are some corner cases when using both approaches, and ideally you should check both to cover all the scenarios. See the kubectl and Helm diff challenges article for more details.
- Store Kubernetes secrets encrypted. Base64 is an encoding algorithm, not an encryption one, and despite its name, Kubernetes secrets are not secret. This is a convoluted topic to discuss because some folks prefer to store encrypted secrets along with the code, while others prefer to keep them in a different location. There are a few alternatives worth mentioning, including helm-secrets plugin, Hashicorp Vault, Bitnami’s Sealed secrets, Mozilla’s sops, DIY solutions, and more. Let’s see a simple example of helm-secrets with AWS SSM via vals.
Create an AWS SSM SecureString object:
$ aws ssm put-parameter --name mysecret --value "secret0" --type SecureString
Code language: JavaScript (javascript)Check the Helm parameter required. In this example, “secretdata“:
$ cat hello-world/templates/secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: mysecret
type: Opaque
stringData:
password: {{ .Values.secretdata }}
Code language: JavaScript (javascript)Verify it:
$ helm secrets --backend vals template hello-world -s templates/secret.yaml --set secretdata="ref+awsssm://mysecret"
---
# Source: hello-world/templates/secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: mysecret
type: Opaque
stringData:
password: secret0
Code language: JavaScript (javascript)- Finally, you can even run arbitrary commands right after rendering the chart, like:
$ helm install mychart my-chart --post-renderer my-script.sh
Code language: JavaScript (javascript)Where the my-script.sh script can be mostly everything, including running kustomize to apply environment variables, verifying a specific parameter has not been used, a script that calls a webhook to get some data, Windows batch scripts, and more! Your imagination is the limit!
Helm security conclusion
Helm is a useful tool to manage the Kubernetes applications lifecycle. While the security aspect is not enforced by default, this article has covered some best practices and Helm security recommendations for consuming and creating Helm charts.