As Kubernetes is eating the world, discover an alternative certified Kubernetes offering called K3s, made by the wizards at Rancher.
K3s is gaining a lot of interest in the community for its easy deployment, low footprint binary, and its ability to be used for specific use cases that the full Kubernetes may be too advanced for. K3s is a fully CNCF (Cloud Native Computing Foundation) certified Kubernetes offering. This means that you can write your YAML to operate against a regular Kubernetes, and they will work exactly the same on a k3s cluster.
In this blog post, we walk through a deployment of k3s. We will then install Falco OSS (for those who want to understand the Open source runtime engine and how its rules and alerting work in k3s) and Sysdig Essentials to walk through security and visibility of k3s quickly, and enhance k3s’ awesome with more awesomeness. We will provide steps and scripts to show you how to create a secure and fully observable cluster within eight minutes.
But first….
What is k3s?
Lightweight Kubernetes. Production ready, easy to install, half the memory, all in a binary less than 100 MB. Useful for Edge, IOT, CI, Development, and many other ways to embed k8s capabilities in a small and powerful package.
In Ranchers own gitpage words:
K3s is a fully conformant production-ready Kubernetes distribution with the following changes:
- It is packaged as a single binary.
- It adds support for sqlite3 as the default storage backend. Etcd3, MySQL, and Postgres are also supported.
- It wraps Kubernetes and other components in a single, simple launcher.
- It is secure by default with reasonable defaults for lightweight environments.
- It has minimal to no OS dependencies (just a sane kernel and cgroup mounts needed).
- It eliminates the need to expose a port on Kubernetes worker nodes for the kubelet API by exposing this API to the Kubernetes control plane nodes over a websocket tunnel.
How does it work? (Diagram, because sometimes a picture can speak 1000 words) This diagram shows that the k3s components are similar to the ones on the vanilla Kubernetes distribution, but only to its bare essentials.
Lets deploy our k3s cluster! (minutes 1-3)
We provide three options below to quickly deploy a k3s cluster with three nodes. You can add more nodes by updating the Terraform config or by adding them in your provider script.
A. Rancher Default Script
1. Download k3s latest release. x86_64, ARMv7, and ARM64 are supported. Disclaimer: ARM isn’t supported yet for Sysdig/Falco.
2. Run the server:
sudo k3s server & # Kubeconfig is written to /etc/rancher/k3s/k3s.yaml sudo k3s kubectl get node
On a different node, run the following. NODE_TOKEN comes from /var/lib/rancher/k3s/server/node-token.
# on your server
sudo k3s agent --server https://myserver:6443 --token ${NODE_TOKEN}
B. k3sup
k3sup is a light-weight utility (from Alex Ellis) to get from zero to KUBECONFIG with k3s on any local or remote VM. All you need is ssh access to run the k3sup binary and get kubectl access immediately.
We prepared some Terraform scripts that provision three nodes on GKE or AWS, and then deploy k3s on them with k3sup. We also created a k3sup.sh script to make things even easier, although you will need to tweak it with the nodes’ IPs and your ssh keys.
C. Managed k3s Cluster
Civo Cloud is the world’s first k3s-powered, managed Kubernetes service. With a managed service, you’ll have your cluster ready in just a few clicks.
Lets install Falco! (Minutes 3-4) (OSS)
Falco is an open source runtime security tool. It was originally built by Sysdig, Inc, was donated to the CNCF, and is now a CNCF incubating project. Falco parses Linux system calls from the kernel at runtime, and asserts the stream against a powerful rules engine. If a rule is violated, a Falco alert is triggered and can be sent to many convenient mechanisms (listed later on in this blog).
Here’s a great diagram @krisnova put together to show how Falco works, which we modified to show k3s:
Falco default rulesets
By default, Falco ships with a mature set of rules that will check the kernel for unusual behavior, such as:
- Privilege escalation using privileged containers
- Namespace changes using tools like
setns - Read/Writes to well-known directories such as
/etc,/usr/bin,/usr/sbin,etc - Creating symlinks
- Ownership and mode changes
- Unexpected network connections or socket mutations
- Spawned processes using
execve - Executing shell binaries such as
sh,bash,csh,zsh, etc. - Executing SSH binaries such as
ssh,scp,sftp, etc. - Mutating Linux coreutils executables
- Mutating login binaries
- Mutating
shadowutilorpasswdexecutablesshadowconfigpwckchpasswdgetpasswdchangeuseraddEtc
Falco alerts
Falco can send alerts to one or more channels:
- Standard Output
- A file
- Syslog
- A spawned program
- A HTTP[s] end point
- A client via the gRPC API
Installing Falco
Now that we know what Falco is and what it does, here are the steps to run Falco on your k3s cluster:
- Deploy your cluster with one of the methods suggested above: k3s script, k3sup, or civo cloud.
- Optionally, create a namespace with:
kubectl create ns falco. - Run
helm install falco falcosecurity/falco. If you are using GCP or later kernels, try this instead:helm install falco falcosecurity/falco --set ebpf.enabled=true.helm repo add falcosecurity https://falcosecurity.github.io/charts helm repo update helm install falco falcosecurity/falco
Falco in action
Here’s an example of a rule violation and the resulting output from Falco (standard output):
The Falco Rule: Network tool launched in a container.
The Attack:
The Result: A notification in STDOUT at the speed of the kernel that can be sent as JSON and other formats!
For more information on Falco rules and installation details, check out falco.org.
Lets install Sysdig! (Commercial Tool, Minutes 3-8)
With the new Sysdig onboarding, you’ll be set in less than five mins.
- Sign up for a free trial.
- Install via our “Get Started” curl scripts.
- You will be provided with an access key and a curl statement that requires a cluster name. In this case, we used
sysdg-k3s. - K3s uses a non-standard place for the containerd runtime. You will need to add this to the end of your curl statement:
-cd unix:///run/k3s/containerd/containerd.sock -cv /run/k3s/containerd
- You will be provided with an access key and a curl statement that requires a cluster name. In this case, we used
Sysdig for k3s in action: The 5 essential workflows
Now that we are installed, let’s see what Sysdig brings to the table.
Earlier this year we launched Sysdig Essentials to provide your organization with an easy way to run containers in production with confidence. We provided a Sysdig Essentials pricing tier aimed for organizations looking to start with the essential use cases; this gives a simplified on-ramp to a Secure DevOps approach.
Let’s now see the five essential workflows in action on our recently created k3s cluster. We’ll start from image scanning, and we’ll go all the way to overall visibility. It’s vital to understand what’s occurring during a security event.
1. Image scanning
In this example, we have deployed a busybox image as a k3s pod. The Sysdig image scanner can analyze the image both inline (locally in your CI/CD pipeline), and by fetching it from the registry.
Sysdig helps you follow the image scanning best practices. For example, you can shift left security with inline scanning, detecting security issues earlier on the pipeline and blocking vulnerable images from being pushed to a registry.
All you need is an image … We got the rest.
2. Runtime security: Falco rules engine… PLUS+
The same rule we used with the Falco example can be used in Sysdig Secure. However, this time we were able to stop the container and leave an audit trail.
The Sysdig Activity Audit shows the ncat command execution, along with all of the other commands that were run in the k3s cluster/pod/process/container.
All of the rich context you can see in our UI can be extracted to a downstream SIEM, to power your Kubernetes pursuits with embedded security … Powerful!
3. Compliance
Although there currently aren’t CIS benchmarks for k3s, the normal kubernetes host or linux host conventions would work. This will allow you to adjust and ensure your k3s cluster compliance at the instance level.
You can visualize the compliance score across your environments as a dashboard within Sysdig.
4. Kubernetes and container monitoring
It’s super important to understand the utilization, availability, and overall capacity of your k3s cluster. That key information will help you either add or remove resources, and size accordingly. A better management of your resources will save you money.
5. Application and cloud service monitoring
The out-of the box dashboards provided by Sysdig will help you monitor the most common cloud services. They follow the Golden Signals principles to give you a very good idea of the health and performance of your application as seen by your final users.
Sysdig is also fully compatible with Prometheus, the de-facto standard for Kubernetes monitoring. This means that your developers can keep using the tools they know and love while accessing a broad variety of resources from repositories like PromCat.
Summary
What did we learn today? K3s is excellent, and you get the ease of deployment and capabilities of Kubernetes in a simple binary.
Add Sysdig, and it’s a no brainer. Running K3s on your instances, along with the five essential workflows for DevOps, can all be done within eight minutes.
Deploy a k3s cluster today and secure it with Falco and Sysdig now!
Useful Links
https://k3s.io/ – K3s
https://github.com/alexellis/k3sup – Alex Ellis’ K3sup tool
https://falco.org – Falco project page
https://sysdig.com/company/free-trial/ – Sysdig Essentials 14 day trial
https://github.com/danpopSD/sysdig-k3s – POP’s Script Repo (Terraform and k3sup scripts)
https://civo.com – managed k3s clusters.
Thank you to Darren Shepherd and the Rancher team for your out-of-the-box thinking and creating k3s, as well as Alex Ellis, Saiyam Pathak, and the Civo Cloud team for the assist on this blog!