Auditd is your friend

Published: 2013-02-14
Last Updated: 2013-02-14 07:10:50 UTC
by Bojan Zdrnja (Version: 1)
9 comment(s)

Recently I’ve been working on several incidents that included attackers getting shell access to the compromised host and somehow elevating their privileges to root. Of course, once they have access to the box, one of the first things they want to do is to be able to establish that same level of access again.

While there are many, many ways for an attacker to plant a backdoor that will allow him access to the server later, the easiest way is, of course, to create a new, privileged account that the attacker can use to access the server.
Now, when analyzing what happened during the attack we figured that this was exactly what the attacker did, however there were no logs on the system and subsequently the central logging system and SIEM implemented by the victim were of no use.

As this was a Linux server, you can probably already guess what the attacker did: they simply opened the /etc/passwd and /etc/shadow files and added their backdoor accounts (with an UID of 0). Of course, since they did this directly by modifying the system files there were absolutely no logs.

Very simple, yet also very effective!

So, what can we do against this? One obvious answer is to monitor any changes to these two and some other important files (for example, /etc/hosts, /etc/sudoers .. there are actually many). On Linux servers it is actually really easy to do this thanks to auditd – the Linux Audit daemon.

Auditd is the userspace part of the Linux Auditing system, which integrates deeply with the kernel. Being integrated with the kernel allows it to inspect every little detail of what’s happening on the system. In fact, many administrators turn auditd off due to huge amounts of logs it can create, and potential performance impact on the system. However, it is a true gem in auditing, if used correctly.

In our example on monitoring /etc/passwd we just need to add one rule to the /etc/audit/audit.rules file:

-w /etc/passwd -p wa

This tells auditd to monitor and log any changes (trigger on write and attribute change of the file). So when our attacker tries to modify this file directly, we will get log similar to the following:

type=SYSCALL msg=audit(1360781410.961:24122): arch=40000003 syscall=15 success=yes exit=0 a0=8357590 a1=81a4 a2=1 a3=1 items=1 ppid=17480 pid=8437 auid=510 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=210 comm="vim" exe="/usr/bin/vim" key=(null)
type=CWD msg=audit(1360781410.961:24122):  cwd="/etc/audit"
type=PATH msg=audit(1360781410.961:24122): item=0 name="/etc/passwd" inode=4786344 dev=fd:00 mode=0100644 ouid=0 ogid=0 rdev=00:00

Pretty cool! So we had the UID and GID of the user who modified the /etc/passwd file, as well as the full path to the process (command) that was used to edit it.

Since auditd creates logs in /var/log/audit/audit.log, we can now send the same file to our SIEM and create a rule to trigger an alert on such actions, so we can catch any modifications to this (and other) system critical files.
For the bonus part, we can even setup a simple cron job (it will suffice in most cases) that will calculate an SHA1 sum of the /etc/passwd file, trigger on any changes and do a diff on the old file (saved before) and the new one.

What other things do you do to monitor your critical files? Let us know!


Keywords: Audit
9 comment(s)


OSSEC does that and many more on more platforms. Cheers!
To monitor critical files, I use an Intrusion Detection System (IDS).

Actually, at the moment I'm using two: Samhain and Tripwire. Both have their individual strengths & weaknesses, and I keep meaning to explore another competitor - AIDE - but never seem to get the time.

At this first cursory glance, I'd say that the main difference between these tools and auditd is that auditd is better at telling you what happened and who did it.
But a system like Samhain/Tripwire/AIDE is better at the actual forensics of what happened - their DB is like a snapshot of the system state, so is much more useful after the fact when doing forensics.

It's a subtle distinction, and it doesn't mean that either one is better than the other.
In fact, there is no reason not to run both...

Which is probably what I'll be doing shortly. :-)
fcheck has nice fire&forget strategy - by default it sends email once an hour with detected changes and updates DB

samhain can be coupled with auditd to check for files modification (via inotify) and report changes immediately. distributed installation of samhain/yule is nice because central server can keep both config and database
etckeeper makes it easy to track your /etc directory in Git, so you have full change history and can detect/diff changes. Git repositories are easily replicated to another machine, you can tag and cryptographically sign it at any point, etc. It's useful in far more situations than intrusions.
If the attacker can modify passwd/shadow, though, you already lost. Any local logs you create, they can erase. Any monitoring daemons killed. Files can potentially be altered via raw disk blocks, avoiding detection by inotify etc. and maybe the changes won't become 'visible' until after a reboot. FreeBSD's 'securelevel' addresses each of these things with some amount of success.
I started looking at auditd when the other types of logs on a system just weren't enough to truly grasp what the potential attacker did. Personally I think auditd is one of those obscure tools that needs more exposure because it's great. Once you have it configured, you have it send the logs to the SIEM and you have an awesome security tool.

Too bad more SIEM vendors don't support auditd logs as of yet.
I have deployed ossec and audit but also have tried to record the commands executed by users and potential attackers. I'm testing snoopy and forwarding of history (bash_history) to a remote syslog.

Snoopy generates a lot of information and can be bypassed. The forwarding of history in real time can also be easily bypassed using another shell (sh, ksh, etc.) or for example launching a connection from metasploit.

Does anyone know a better alternative?

The attacker has to elevate privileges to edit /etc/passwd and /etc/shadow from a local account. RHEL does a poor job in logging privilege elevations. Auditd may catch it. but the event will be lost in a lot of noise. Debian/Ubuntu offer ninja which is a HIDS that monitors privilege escalation/elevation events and can even stop them. It is relatively easy to port to RHEL. This program has advantages over auditd which just tells you you've been compromised.

A simple cron script can be written to monitor the addition of UID0 accounts to /etc/passwd and send an alert via email or logger to /var/log/messages, but it has the same drawback as auditd. It's a passive reporting tool.
I highly recommend this framework. Its got a lot of moving parts but is well worth the work.

Diary Archives