Intezer and Microsoft reported on Sept. 9 that TeamTNT hackers are deploying Weave Scope in compromised systems as an auxiliary tool in their intrusions. Weave Scope is a legitimate and powerful tool to manage server infrastructure that, once deployed, makes it easy to control all resources.
In this article, we will describe how this tool can be used maliciously, and how to add specific checks in your security set up to look for it.
Vulnerability Context
The vulnerability that TeamTNT used to originally gain entry was an unsecured Docker API endpoint exposed to the internet. After exploiting it, they then proceeded to deploy Weave Scope as their tool to scan the whole system.
The attack flow
- The attacker identified an exposed unprotected Docker API.
- They spun up a new Ubuntu container on the host running the exposed Docker daemon.
- The new container mounted the host filesystem, allowing access to sensitive information.
- They created a new privileged user they can use to SSH into the server.
- After connecting to the host, they downloaded and installed the Weave Scope application.
- They connected to the Weave Scope console over port 4040.
- This gave the attackers full visibility into the Kubernetes environment, allowing them to identify the sensitive assets and take over the environment.
The impact of exposed Docker API and Weave Scope reach
An exposed Docker API allows any attacker to execute workloads in your infrastructure as root users. This way, an attacker can gain access to the host operating system, all of the running containers, and possibly all of the cluster’s infrastructure.
From there, Weave Scope is going to make it easier to scan all workloads to search for further vulnerabilities, exfiltrate any unsecured information, and execute additional workloads like cryptominers. They can also plant additional exploits, ready to be used in case Weave Scope is terminated and the Docker API endpoint gets secured.
Mitigating exposed Docker API port and Weave Scope usage
Exposed Docker API port
A full blown security audit of all of your infrastructure is the recommended action if you suspect you may be compromised, as no one knows what could have been installed with this exploit.
But for a quick check, first make sure the Docker API port is well protected and not accessible from a public IP. You can check it by reaching port 2376 (default Docker port) from outside with these commands:
export public_ip=<your_infra_public_ip> nc -z $public_ip 2367 ; if [ "$?" = "1" ]; then echo "Port not exposed" ; else echo "Port exposed" ; fi
Any cloud managed container platform is going to use a secure by default approach. Make sure you follow their recommendations on how to connect to your workloads in a secure way.
If you have deployed Docker or Kubernetes by yourself, you must know that they follow the opposite paradigm. In them, everything is unsecured by default, and you have to make sure you add your own security to it. A secure Kubernetes deployment or Docker installation aren’t trivial tasks. Although Docker and Kubernetes automatically provide many management mechanisms out of the box, you still have to take care of their security and authentication configuration.
Falco, a CNCF incubating project, can help detect anomalous activities in cloud native environments. As an extra precaution, you can use a Falco rule like this one to detect any connection to the port 2376 from outside of the local network:
- rule: Connection to exposed Docker API on port 2376
desc: Detected connection to exposed Docker API on port 2376
condition: >
(inbound_outbound) and fd.sport=2376
and not fd.snet in (rfc_1918_addresses))
output: >
Detected connection to exposed Docker API on port 2376
(command=%proc.cmdline connection=%fd.name user=%user.name
container_id=%container.id image=%container.image.repository)
priority: CRITICAL
tags: [network]
Detect Weave Scope deployment
You should check if Weave Scope is already deployed in your Docker or Kubernetes infrastructure.
Checking all running images on Docker is easy with docker ps:
docker ps | grep weavescope
To get a nice list of all running images on Kubernetes, execute:
$ kubectl get pods -o jsonpath="{..image}" --all-namespaces |\
tr -s '[[:space:]]' '\n' |\
sort |\
uniq -c
An image scanner can also help you detect the Weave Scope image, adding weaveworks/scope to a global block/disallow list.
You can also leverage Falco to make sure you prevent the image from running unnoticed on Docker or Kubernetes, with a rule like this:
- rule: Launch of Weave Scope network tool container
desc: Detect launching Weave Scope network tool container
condition: >
container and
container.image.repository startswith 'weaveworks/scope'
output: >
Detect launching Weave Scope network tool binary process running
(command=%proc.cmdline connection=%fd.name user=%user.name
user_loginuid=%user.loginuid container_id=%container.id
image=%container.image.repository)
priority: WARNING
tags: [network]
Keep in mind, the image could be deployed with a different registry/image name combination than the original Weave Scope one. So you should add some checks that don’t depend on the image name.
For example, some image scanners allow you to detect specific files and folders. Remember, Weave Scope is a legitimate tool, so it’s not going to appear in a database of vulnerabilities. If you visit the Weave Scope image entry in Docker Hub, you can see that its entrypoint is:
ENTRYPOINT ["/home/weave/scope" "--mode=probe" "--no-app" "--probe.docker=true"]
So if we create an image scanning policy to check for that specific endpoint, we are going to detect this tool even if the name or source of the image is a different one.
The best way to do is to use a “like” comparison against “*/home/weave/scope*,” in case additional parameters change.
You will ideally set up this policy to be used with an admission controller in Kubernetes, preventing matching images from being deployed. You should also include an inline scan step in your CI pipelines, so non-compliant images are blocked from reaching your registry.
The last line of defense is to add a Falco rule to detect if the directory/binary combination is running in the kernel of the Docker host, or Kubernetes nodes, no matter the nature of the method used to deploy it:
- rule: Weave Scope network tool binary process
desc: Detect the Weave Scope network tool binary process running
condition: spawned_process and proc.cmdline contains 'weave/scope'
output: >
Detect launching Weave Scope network tool binary process running
(command=%proc.cmdline connection=%fd.name user=%user.name
user_loginuid=%user.loginuid container_id=%container.id
image=%container.image.repository)
priority: CRITICAL
tags: [network, weave_scope]
Conclusions
An exposed Docker API entrypoint is a very dangerous threat vector, and combined with the powerful Weave Scope tool, it can deeply compromise your whole infrastructure very quickly.
Sysdig Secure can help you with its inline scanning, runtime security using Falco, admission controller, and image analyzer features. Try Sysdig today!