Helm is being used broadly to deploy Kubernetes applications as it is an easy way to publish and consume them via a couple of commands, as well as integrate them in your GitOps pipeline. But is Helm security taken seriously? Can you trust it blindly?
This post explains the benefits of using Helm, the pitfalls, and offers a few recommendations for Helm security. Let’s get started!
Let’s face it, managing an application lifecycle on Kubernetes is hard. If you want to just deploy an application, it requires at least a Deployment, usually tweaking the application configuration via modifications performed to a ConfigMap or a Secret, deploying CRDs if needed, and more. Clearly, it’s not really straightforward.
You can create your own deployment files with some environment variables and use
envsubst to substitute them at runtime (
envsubst < deploy.yml | kubectl apply -f -) to have a “DIY templating engine,” but that is probably not an optimal solution.
Helm security is not a priority and it is largely up to the user to make good use of it. Helm is not perfect, but it tries to make that process easier by providing a simple command interface, a repository with more than 9000 charts available called Artifact Hub (and the ability to host your own charts on your own repository), and a templating engine (with over 60 available functions, mostly based on the Go template language). That allows you to package complex applications to make them easily deployable by just providing specific parameters.
For example, you can deploy a whole MySQL cluster with replication enabled (a non-trivial task, let’s be honest) by just using the
It also has some advanced features, such as hooks (to run specific tasks at specific points on the deployment process such as ‘pre-install’), and can be integrated with GitOps tools, such as ArgoCD or Flux. You can leverage library charts or named templates, and even run post-render tasks (e.g., to run Kustomize).
Helm security – How to secure Helm
We’ve covered a lot of ground, but we didn’t pay any attention to any Helm security aspects and most charts are not secure by default.
There are a few angles to tackle depending on the process we want to cover. Are we just consuming the Helm charts, the Kubernetes objects created by the charts, or are we talking about custom Helm charts?
Custom Helm charts
If writing your own Helm charts, a few general recommendations apply, as well as some security focused ones:
- Store the charts in a Git repository. This may seem obvious in 2022, but Git will give you some benefits just by using it, such as easy rollbacks or the ability to track changes.
- Store the Helm charts in a proper repository. Charts can be served via HTTP but everything is HTTPS these days, right?
helm lintor any other linter you prefer to verify the Helm charts are properly formed. You don’t want to break the production environment for a silly typo.
For example, in this basic Helm chart file without a proper
version, Helm lint complains about it:
- Use consistent versioning on your charts (Helm follows the SemVer2 standard). It is helpful for reproducibility and to be able to respond quickly in a situation where you need to update your charts because a vulnerability has been found. If your charts are unversioned or using “latest”, which one would you update?
There are two different versions you can use: the version of the chart itself (
Chart.yamlfile) and the version of the application (
- Create test scenarios for your Helm charts to cover your use cases. The idea is to validate the success of the Helm deployment by creating Kubernetes objects (as in Helm templates) that will test your deployed chart by running
helm test <RELEASE_NAME>. For example, a test can be just a simple pod running on the same namespace where your application has been deployed, that queries your application API to see if it has been deployed properly:
Usually, tests are stored in the
templates/tests/ folder and are required to have the “
helm.sh/hook": test annotation to identify themselves as tests.
- Sign your charts easily with
helm package –sign(and verify them with
helm install --verify). Asserting the integrity of the software components is the most common task when securing the software supply chain. This usually means verifying a digital signature (either included with the software itself or close to it). Helm uses a PGP-based digital signature to create provenance records stored in provenance files (
.prov), which are stored alongside a packaged chart. Let’s see an example:
And this is what the provenance file looks like:
$ cat hello-world-0.0.1.tgz.prov -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 ... name: hello-world ... files: hello-world-0.0.1.tgz: sha256:b3f75d753ffdd7133765c9a26e15b1fa89784e18d9dbd8c0c51037395eeb332e -----BEGIN PGP SIGNATURE----- wsFcB… -----END PGP SIGNATURE-----%
If the signature doesn’t match, Helm will complain:
helm install --verify will automatically check the provenance files:
Or, you can just pull the chart and verify it:
The public key needs to be trusted beforehand for
--verify to work, so you must make it publicly available somewhere, otherwise it will fail:
- Automate all the previous steps (testing, versioning, signing, and releasing) in a CI/CD pipeline to make sure they are consistent with the best practices on every change, and to avoid potential problems when doing manual changes.
You can use the helm/charts-repo-actions-demo for inspiration on how to create a GitHub actions workflow to test and release a chart:
When creating the Kubernetes objects via templates, Helm doesn’t provide any security measures out of the box. You are on your own and can apply any bad practice you want, such as deploying a container with a root user or with full capabilities (OK, you may want to do it). Let’s talk about some recommendations:
- Use Role-based access control (RBAC) to limit the object’s permissions (don’t use
cluster-adminfor everything). For example, the falcosidekick Helm chart creates a Role, ServiceAccount, and RoleBinding to minimize the required permissions used in the K8s Deployment:
- Provide sane defaults. For example, if your chart includes a MySQL pod, don’t use a default password so any user could know about it. Instead, generate it randomly or force the user to specify it. However, there are a couple of things to consider, including how to deal with upgrades in this GitHub issue and this blog post. You can use the lookup function and the resource policy annotation to prevent overwriting when upgrading, as follows:
- Reduce the attack surface. Make the deployment small and enable components and features using flags or values if needed. For example, the Falco Helm chart doesn’t deploy falcosidekick by default, but you can enable it easily:
Which is then used in the Chart:
All the rest of Kubernetes recommendations apply (including the CIS benchmarks, for example), so make sure you scan your Kubernetes object definitions for best practices. If your preferred tool doesn’t support Helm charts, don’t worry. You can always render the Kubernetes objects in a previous step with the Helm template command, as follows:
And then verify them as:
Using Helm charts
- Don’t trust the Helm charts blindly, especially third-party ones. Fortunately, as we’ve seen, the
helm templatecommand renders and outputs the Kubernetes objects created by the Helm chart, so it is a good practice to at least take a quick look at the results before deploying it in your Kubernetes cluster. You probably don’t want to use joaquinito2051‘s charts.
- As explained before, use
helm verifyto check the digital signatures of the charts you use to make sure you are using the charts you are supposed to.
- Uninstall unused releases: If you’re no longer using a Helm release, uninstall it to reduce your attack surface.
- Always try to keep the Helm Charts you use updated (as well as the
helmbinary and their plugins!). Let’s face it, mistakes and bugs happen so it is a good idea to always use the latest version with the latest fixes for both the Helm chart itself or for the objects the Helm chart creates (for example, if it uses a container image that has been found vulnerable). This also applies to subcharts. There are a couple of options to verify what an upgrade will change, including using the helm diff plugin:
Or the ability to render the manifests via
helm template and use
kubectl to make the diffs:
However, there are some corner cases when using both approaches, and ideally you should check both to cover all the scenarios. See the kubectl and Helm diff challenges article for more details.
- Store Kubernetes secrets encrypted. Base64 is an encoding algorithm, not an encryption one, and despite its name, Kubernetes secrets are not secret. This is a convoluted topic to discuss because some folks prefer to store encrypted secrets along with the code, while others prefer to keep them in a different location. There are a few alternatives worth mentioning, including helm-secrets plugin, Hashicorp Vault, Bitnami’s Sealed secrets, Mozilla’s sops, DIY solutions, and more. Let’s see a simple example of helm-secrets with AWS SSM via vals.
Create an AWS SSM SecureString object:
Check the Helm parameter required. In this example, “
- Finally, you can even run arbitrary commands right after rendering the chart, like:
my-script.sh script can be mostly everything, including running kustomize to apply environment variables, verifying a specific parameter has not been used, a script that calls a webhook to get some data, Windows batch scripts, and more! Your imagination is the limit!
Helm security conclusion
Helm is a useful tool to manage the Kubernetes applications lifecycle. While the security aspect is not enforced by default, this article has covered some best practices and Helm security recommendations for consuming and creating Helm charts.