Recently I’ve been working on several incidents that included attackers getting shell access to the compromised host and somehow elevating their privileges to root. Of course, once they have access to the box, one of the first things they want to do is to be able to establish that same level of access again.

While there are many, many ways for an attacker to plant a backdoor that will allow him access to the server later, the easiest way is, of course, to create a new, privileged account that the attacker can use to access the server.
Now, when analyzing what happened during the attack we figured that this was exactly what the attacker did, however there were no logs on the system and subsequently the central logging system and SIEM implemented by the victim were of no use.

As this was a Linux server, you can probably already guess what the attacker did: they simply opened the /etc/passwd and /etc/shadow files and added their backdoor accounts (with an UID of 0). Of course, since they did this directly by modifying the system files there were absolutely no logs.

Very simple, yet also very effective!

So, what can we do against this? One obvious answer is to monitor any changes to these two and some other important files (for example, /etc/hosts, /etc/sudoers .. there are actually many). On Linux servers it is actually really easy to do this thanks to auditd – the Linux Audit daemon.

Auditd is the userspace part of the Linux Auditing system, which integrates deeply with the kernel. Being integrated with the kernel allows it to inspect every little detail of what’s happening on the system. In fact, many administrators turn auditd off due to huge amounts of logs it can create, and potential performance impact on the system. However, it is a true gem in auditing, if used correctly.

In our example on monitoring /etc/passwd we just need to add one rule to the /etc/audit/audit.rules file:

-w /etc/passwd -p wa

This tells auditd to monitor and log any changes (trigger on write and attribute change of the file). So when our attacker tries to modify this file directly, we will get log similar to the following:

type=SYSCALL msg=audit(1360781410.961:24122): arch=40000003 syscall=15 success=yes exit=0 a0=8357590 a1=81a4 a2=1 a3=1 items=1 ppid=17480 pid=8437 auid=510 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=210 comm="vim" exe="/usr/bin/vim" key=(null)
type=CWD msg=audit(1360781410.961:24122):  cwd="/etc/audit"
type=PATH msg=audit(1360781410.961:24122): item=0 name="/etc/passwd" inode=4786344 dev=fd:00 mode=0100644 ouid=0 ogid=0 rdev=00:00

Pretty cool! So we had the UID and GID of the user who modified the /etc/passwd file, as well as the full path to the process (command) that was used to edit it.

Since auditd creates logs in /var/log/audit/audit.log, we can now send the same file to our SIEM and create a rule to trigger an alert on such actions, so we can catch any modifications to this (and other) system critical files.
For the bonus part, we can even setup a simple cron job (it will suffice in most cases) that will calculate an SHA1 sum of the /etc/passwd file, trigger on any changes and do a diff on the old file (saved before) and the new one.

What other things do you do to monitor your critical files? Let us know!

--
Bojan
INFIGO IS