ISC Diary

There are a lot of discussions at the moment about a SSHD rootkit hitting mainly RPM based Linux distributions.
Thanks to our reader unSpawn, we received a bunch of samples of the rootkit. The rootkit is actually a trojanized library that links with SSHD and does *a lot* of nasty things to the system.

At this point in time we still do not know what the initial attack vector is – it is unknown how the attackers get root access on the compromised servers that is needed to change the legitimate libkeyutils library with a trojanized one. We are, of course, keeping an eye on the development and will post a new diary or update this one if we receive more information about the attack vectors.

The trojanized library is very, very nasty. Upon execution it performs a number of actions, as described below.

The code first deobfuscates the text strings needed for it to work. The original text is only XORed so this is very easy to retrieve and the deobfuscated strings have already been posted on a lot of sites.

Once that has been done, the library sets up everything needed for it to work. It resolves symbols for the following functions which are used later: PEM_write_RSAPrivateKey, PEM_write_DSAPrivateKey, MD5_Init, MD5_Update, and MD5_Final. As you can already see, it is definitely messing up with the authentication mechanism.

Besides resolving the symbols, the library also hooks the following functions: pam_authenticate, pam_start and crypt as well as audit_log_user_message and audit_log_acct_message. By hooking these functions, the rootkit can modify the flow of the SSHD – as you can see, this is a user-mode rootkit, as it does not affect the kernel.

The main activity of the rootkit consists in collection of credentials of authenticated users. Notice that the rootkit can steal username and password pairs as well as RSA and DSA private keys, so no matter which authentication mechanism you use, if the target host is infected it will successfully steal your information. The hooking of audit_log* functions was done to allow the attacker to stay as low profile as possible – if the attacker uses the hardcoded backdoor password to issue any commands to the rootkit, no logs will be created.

The current version of the rootkit supports three commands: Xver, Xcat and Xbnd. The first command just prints the rootkit’s version; the Xcat commands print the collected information back in the session for the attacker while the Xbnd command allows the attacker to setup a listener.

Besides this, the rootkit can automatically send collected credentials to the attacker. In order to do this the rootkit has a DGA (Domain Generation Algorithm) implemented that will create random looking domain names in the .biz, .info and .net domains (in that order). It will then send a DNS packet containing collected credentials to the target IP address, if it was able to resolve it (meaning the attacker has registered that day’s domain). If no domains have been resolved, the DNS packet is sent to the hard-coded IP address, which in all samples we received was 78.47.139.110.

The rootkit itself looks very similar to the Ebury trojan which was detected back in 2011. In fact, I’m pretty sure that a lot of the code has been directly copied, however, the Ebury trojan patched the whole SSHD and required the attacker to change it.

This was easier to detect and prone to being overwritten with patching. The libkeyutils library, which comes as part of the keyutils-libs package is not changed that often so the chance of it being overwritten automatically is much lower.

If you run a RPM based system you can check the integrity of the file with the rpm command:

# rpm -Vv keyutils-libs-1.2-1.el5
........    /lib/libkeyutils-1.2.so
S.5.....    /lib/libkeyutils.so.1
........    /usr/share/doc/keyutils-libs-1.2
........  d /usr/share/doc/keyutils-libs-1.2/LICENCE.LGPL

This will check a lot of things, the most important being the MD5 checksum so if you see the output as one above you have a trojanized library. Proper output should have all (and only) dots. Keep in mind that the RPM’s verification, of course, depends on the integrity of its database and the kernel itself.

We will keep an eye on the development and will update the diary accordingly – if you have samples or more information, especially on what the initial attack vector is please let us know.

I’d like to thanks again to unSpawn for supporting the SANS ISC.

UPDATE:

Over the night (depending on where you live :), a lot of things happened. Thanks to one of our readers, Steve, I got access to a compromised server and hopefully might shed more light on what is going on here.

cPanel also posted a notice to their users that they have been compromised. What’s even worse, it appears that one of the main support servers in cPanel has been compromised which again means that the attackers got *loads* of passwords. cPanel recommends that passwords are changed, but keep in mind – if your servers are infected with the SSHD rootkit, the attackers will get your passwords/keys *anyway* (see below). So make sure that you check if your server has been compromised and that you clean it accordingly.

While analyzing the rootkit I noticed that, similarly to the Ebury trojan, it uses shared memory for communication between processes (and Steve e-mailed about that too). On Linuxes, you can check the status of shared memory with the ipcs command:

# ipcs -m

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status
0x7400845c 1769472    root      600        4          0
0x00000000 2129921    root      644        52         2
0x7400845b 1736706    root      600        4          0
0x00000000 2162691    root      644        16384      2
0x00000000 2195460    root      644        268        2
0x0052e2c1 2228229    postgres  600        10469376   16

In order to check the owner, use the –p flag:

# ipcs -mp

------ Shared Memory Creator/Last-op --------
shmid      owner      cpid       lpid
1769472    root       1975       1975
2129921    root       2931       2940
1736706    root       1965       1965
2162691    root       2931       2940
2195460    root       2931       2940
2228229    postgres   4011       6813

Now you can verify if SSHD owns any of the shared memory segments – normally it shouldn’t, so if it does you can try to inspect the system further.

Finally, unSpawn also wrote a ClamAV signature (a logical signature – so store this in a .ldb file):

RKH_libkeyutils.so.1.9;Target:6;(((0)&(1)&(2))&(((3)&(4)&(5))|((6)&(7)&(8))));636f6e6e656374;73656e64;736f636b6574;62696e64;746d7066696c65;77616974706964;646c636c6f7365;737472636174;737472637079

It should help detect malicious libraries. Since the attackers can store it in a different directory, make sure to use the find command to locate all potential copies. Something like this will work:

# find / -name libkeyutils*

UPDATE2:

As you can see in the comments below, SJFriedl is right that the private key cannot be stolen on the server since effectively it is never sent.

The rootkit does, however, hook the PEM_write_RSAPrivateKey and PEM_write_DSAPrivateKey functions, which are used to dump the private key into a file. I'm not sure when this is used yet, but it is possible that they maybe also hook libraries of the ssh-keygen utility, which is used for generation of RSA/DSA keys, and is also linked with the libkeyutils library.

--
Bojan (@bojanz)
INFIGO IS