Helm security and best practices

By Eduardo Mínguez - NOVEMBER 15, 2022

Helm is being used broadly to deploy Kubernetes applications as it is an easy way to publish and consume them via a couple of commands, as well as integrate them in your GitOps pipeline. But is Helm security taken seriously? Can you trust it blindly?

This post explains the benefits of using Helm, the pitfalls, and offers a few recommendations for Helm security. Let’s get started!

Why Helm?

Helm is a graduated open-source CNCF project originally created by DeisLabs. If you want to know more about how it works, we recommend you read the Helm 101 article of our Learning Cloud Native hub.

Let’s face it, managing an application lifecycle on Kubernetes is hard. If you want to just deploy an application, it requires at least a Deployment, usually tweaking the application configuration via modifications performed to a ConfigMap or a Secret, deploying CRDs if needed, and more. Clearly, it’s not really straightforward.

You can create your own deployment files with some environment variables and use envsubst to substitute them at runtime (envsubst < deploy.yml | kubectl apply -f -) to have a “DIY templating engine,” but that is probably not an optimal solution.

Kustomize improves the previous DIY solution but it has some limitations as well (it is mainly focused on templating, not packaging). Jsonnet can also be used for templating.

Helm security is not a priority and it is largely up to the user to make good use of it. Helm is not perfect, but it tries to make that process easier by providing a simple command interface, a repository with more than 9000 charts available called Artifact Hub (and the ability to host your own charts on your own repository), and a templating engine (with over 60 available functions, mostly based on the Go template language). That allows you to package complex applications to make them easily deployable by just providing specific parameters.

For example, you can deploy a whole MySQL cluster with replication enabled (a non-trivial task, let’s be honest) by just using the architecture=replication parameter.

It also has some advanced features, such as hooks (to run specific tasks at specific points on the deployment process such as ‘pre-install’), and can be integrated with GitOps tools, such as ArgoCD or Flux. You can leverage library charts or named templates, and even run post-render tasks (e.g., to run Kustomize).

Helm security – How to secure Helm

We’ve covered a lot of ground, but we didn’t pay any attention to any Helm security aspects and most charts are not secure by default.

Securing Helm

There are a few angles to tackle depending on the process we want to cover. Are we just consuming the Helm charts, the Kubernetes objects created by the charts, or are we talking about custom Helm charts?

Custom Helm charts

If writing your own Helm charts, a few general recommendations apply, as well as some security focused ones:

  • Store the charts in a Git repository. This may seem obvious in 2022, but Git will give you some benefits just by using it, such as easy rollbacks or the ability to track changes.
  • Store the Helm charts in a proper repository. Charts can be served via HTTP but everything is HTTPS these days, right?
  • Use helm lint or any other linter you prefer to verify the Helm charts are properly formed. You don’t want to break the production environment for a silly typo.

    For example, in this basic Helm chart file without a proper version, Helm lint complains about it:

    apiVersion: v2 name: hello-world description: A Helm chart for Kubernetes type: application appVersion: "0.0.1" $ helm lint --strict ==> Linting . [ERROR] Chart.yaml: version is required [INFO] Chart.yaml: icon is recommended [ERROR] templates/: validation: chart.metadata.version is required [ERROR] : unable to load chart validation: chart.metadata.version is required Error: 1 chart(s) linted, 1 chart(s) failed
    Code language: JavaScript (javascript)
  • Use consistent versioning on your charts (Helm follows the SemVer2 standard). It is helpful for reproducibility and to be able to respond quickly in a situation where you need to update your charts because a vulnerability has been found. If your charts are unversioned or using “latest”, which one would you update?
    There are two different versions you can use: the version of the chart itself (version in the Chart.yaml file) and the version of the application (appVersion).
    $ helm show chart falcosecurity/falco | grep -E '^version|^appVersion' appVersion: 0.33.0 version: 2.2.0
    Code language: JavaScript (javascript)

    Don’t forget to Keep a Changelog (like Falco does).

  • Create test scenarios for your Helm charts to cover your use cases. The idea is to validate the success of the Helm deployment by creating Kubernetes objects (as in Helm templates) that will test your deployed chart by running helm test <RELEASE_NAME>. For example, a test can be just a simple pod running on the same namespace where your application has been deployed, that queries your application API to see if it has been deployed properly:
apiVersion: v1 kind: Pod metadata: … annotations: "helm.sh/hook": test spec: containers: - name: wget image: busybox command: ['wget'] args: ['{{ .Values.service.name }}:{{ .Values.service.port }}'] restartPolicy: Never
Code language: JavaScript (javascript)

Usually, tests are stored in the templates/tests/ folder and are required to have the “helm.sh/hook": test annotation to identify themselves as tests.

$ helm test hello-world NAME: hello-world ... Phase: Succeeded $ kubectl get po -n hello-world NAME READY STATUS RESTARTS AGE hello-world-78b98b4c85-kbt58 1/1 Running 0 91s hello-world-test 0/1 Completed 0 67s
Code language: JavaScript (javascript)
  • Sign your charts easily with helm package –sign (and verify them with helm install --verify). Asserting the integrity of the software components is the most common task when securing the software supply chain. This usually means verifying a digital signature (either included with the software itself or close to it). Helm uses a PGP-based digital signature to create provenance records stored in provenance files (.prov), which are stored alongside a packaged chart. Let’s see an example:
$ helm package --sign --key 'Eduardo Minguez' hello-world --keyring ~/.gnupg/secring.gpg Password for key "Eduardo Minguez (gpg key) <[email protected]>" > Successfully packaged chart and saved it to: /home/edu/git/my-awesome-stuff/hello-world-0.0.1.tgz
Code language: JavaScript (javascript)

And this is what the provenance file looks like:

$ cat hello-world-0.0.1.tgz.prov -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 ... name: hello-world ... files: hello-world-0.0.1.tgz: sha256:b3f75d753ffdd7133765c9a26e15b1fa89784e18d9dbd8c0c51037395eeb332e -----BEGIN PGP SIGNATURE----- wsFcB… -----END PGP SIGNATURE-----%

If the signature doesn’t match, Helm will complain:

$ helm verify hello-world-0.0.1.tgz Error: openpgp: invalid signature: hash tag doesn't match
Code language: JavaScript (javascript)

Running helm install --verify will automatically check the provenance files:

$ helm install --verify myrepo/mychart-1.2.3
Code language: JavaScript (javascript)

Or, you can just pull the chart and verify it:

$ helm pull --verify myrepo/mychart-1.2.3
Code language: JavaScript (javascript)

The public key needs to be trusted beforehand for --verify to work, so you must make it publicly available somewhere, otherwise it will fail:

$ helm pull --verify myrepo/mychart-1.2.3 Error: openpgp: signature made by unknown entity $ cat security/pubkey.gpg | gpg --import --batch $ helm pull --verify myrepo/mychart-1.2.3 Signed by:..
Code language: JavaScript (javascript)

There is also a sigstore Helm plugin to use Rekor as signature storage, which is even better.

  • Automate all the previous steps (testing, versioning, signing, and releasing) in a CI/CD pipeline to make sure they are consistent with the best practices on every change, and to avoid potential problems when doing manual changes.
    You can use the helm/charts-repo-actions-demo for inspiration on how to create a GitHub actions workflow to test and release a chart:
- name: Run chart-releaser uses: helm/[email protected].4.0 with: charts_dir: charts config: cr.yaml env: CR_TOKEN: "${{ secrets.GITHUB_TOKEN }}"
Code language: JavaScript (javascript)

Kubernetes objects

When creating the Kubernetes objects via templates, Helm doesn’t provide any security measures out of the box. You are on your own and can apply any bad practice you want, such as deploying a container with a root user or with full capabilities (OK, you may want to do it). Let’s talk about some recommendations:

  • Use Role-based access control (RBAC) to limit the object’s permissions (don’t use cluster-admin for everything). For example, the falcosidekick Helm chart creates a Role, ServiceAccount, and RoleBinding to minimize the required permissions used in the K8s Deployment:
--- apiVersion: v1 kind: ServiceAccount metadata: name: {{ include "falcosidekick.fullname" . }} … --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: {{ include "falcosidekick.fullname" . }} … rules: - apiGroups: - "" resources: - endpoints verbs: - get … --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding … roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: {{ include "falcosidekick.fullname" . }} subjects: - kind: ServiceAccount name: {{ include "falcosidekick.fullname" . }} …
Code language: JavaScript (javascript)
  • Provide sane defaults. For example, if your chart includes a MySQL pod, don’t use a default password so any user could know about it. Instead, generate it randomly or force the user to specify it. However, there are a couple of things to consider, including how to deal with upgrades in this GitHub issue and this blog post. You can use the lookup function and the resource policy annotation to prevent overwriting when upgrading, as follows:
{{- if not (lookup "v1" "Secret" .Release.Namespace "hello-world") }} apiVersion: v1 kind: Secret metadata: name: mysecret annotations: "helm.sh/resource-policy": "keep" type: Opaque stringData: password: {{ randAlphaNum 24 }} {{- end }}
Code language: JavaScript (javascript)
falcosidekick: # -- Enable falcosidekick deployment. enabled: false
Code language: JavaScript (javascript)

Which is then used in the Chart:

dependencies: - name: falcosidekick condition: falcosidekick.enabled
Code language: JavaScript (javascript)

All the rest of Kubernetes recommendations apply (including the CIS benchmarks, for example), so make sure you scan your Kubernetes object definitions for best practices. If your preferred tool doesn’t support Helm charts, don’t worry. You can always render the Kubernetes objects in a previous step with the Helm template command, as follows:

$ helm template falco falcosecurity/falco --namespace falco --create-namespace --set driver.kind=ebpf > all.yaml
Code language: JavaScript (javascript)

And then verify them as:

$ myawesometool --verify all.yaml
Code language: JavaScript (javascript)

Using Helm charts

  • Don’t trust the Helm charts blindly, especially third-party ones. Fortunately, as we’ve seen, the helm template command renders and outputs the Kubernetes objects created by the Helm chart, so it is a good practice to at least take a quick look at the results before deploying it in your Kubernetes cluster. You probably don’t want to use joaquinito2051‘s charts.
  • As explained before, use helm verify to check the digital signatures of the charts you use to make sure you are using the charts you are supposed to.
  • Uninstall unused releases: If you’re no longer using a Helm release, uninstall it to reduce your attack surface.
  • Always try to keep the Helm Charts you use updated (as well as the helm binary and their plugins!). Let’s face it, mistakes and bugs happen so it is a good idea to always use the latest version with the latest fixes for both the Helm chart itself or for the objects the Helm chart creates (for example, if it uses a container image that has been found vulnerable). This also applies to subcharts. There are a couple of options to verify what an upgrade will change, including using the helm diff plugin:
$ helm diff --install foo --set image.tag=1.14.0 .
Code language: JavaScript (javascript)

Or the ability to render the manifests via helm template and use kubectl to make the diffs:

$ helm template --is-upgrade --no-hooks --skip-crds foo --set image.tag=1.14.0 . | kubectl diff --server-side=false -f -
Code language: JavaScript (javascript)

However, there are some corner cases when using both approaches, and ideally you should check both to cover all the scenarios. See the kubectl and Helm diff challenges article for more details.

Create an AWS SSM SecureString object:

$ aws ssm put-parameter --name mysecret --value "secret0" --type SecureString
Code language: JavaScript (javascript)

Check the Helm parameter required. In this example, “secretdata“:

$ cat hello-world/templates/secret.yaml apiVersion: v1 kind: Secret metadata: name: mysecret type: Opaque stringData: password: {{ .Values.secretdata }}
Code language: JavaScript (javascript)

Verify it:

$ helm secrets --backend vals template hello-world -s templates/secret.yaml --set secretdata="ref+awsssm://mysecret" --- # Source: hello-world/templates/secret.yaml apiVersion: v1 kind: Secret metadata: name: mysecret type: Opaque stringData: password: secret0
Code language: JavaScript (javascript)
$ helm install mychart my-chart --post-renderer my-script.sh
Code language: JavaScript (javascript)

Where the my-script.sh script can be mostly everything, including running kustomize to apply environment variables, verifying a specific parameter has not been used, a script that calls a webhook to get some data, Windows batch scripts, and more! Your imagination is the limit!

Helm security conclusion

Helm is a useful tool to manage the Kubernetes applications lifecycle. While the security aspect is not enforced by default, this article has covered some best practices and Helm security recommendations for consuming and creating Helm charts.