Behavioral Analysis of Rootkit Malware

Published: 2006-07-16
Last Updated: 2006-07-17 23:36:24 UTC
by Lenny Zeltser (Version: 3)
0 comment(s)

Those who've taken my Reverse-Engineering Malware class know that I am a fan of a two-phased approach to malware analysis:

  1. The behavioral analysis phase examines how the malicious program interacts with its environment: the file system, the registry (if it's a Windows program), and the network.
  2. The code analysis phase examines the code of the malicious program to understand what capabilities are built into it.
Each phase produces findings that reinforce findings from the other phase, resulting in a comprehensive understanding of the malicious program that would be harder to obtain via a single phase. The analyst typically starts with the phase that he or she is most comfortable with.

The behavioral analysis phase can be tricky when the malicious specimen exhibits rootkit tendencies--hiding its processes or files, for instance. One way to deal with this is to patch the specimen so that the concealing subroutine never executes. This is not always easy. To ease the challenge of monitoring rootkit-concealed processes, we can employ programs that can detect concealment mechanisms such as function hooking. I'd like to describe two such programs: Helios and IceSword.

The authors of Helios call it an advanced malware detection system. It attempts to tackle the task of heuristically detecting and blocking malicious programs, even rootkits, before they can embed themselves deep in the system. The program is still in the alpha development phase, but it is available as a free download for those who want to experiment with it in a laboratory environment.

I took Helios for a spin today to see whether it could help with malware analysis. I think it can be a helpful addition to the reverse-engineer's toolkit, because it can detect when the malicious program attempts to hide itself via rootkit techniques. Helios can also unhide the malicious process to to make behavioral analysis a bit easier.

For example, consider a malicious program called malware.exe. Executing normally, it is visible in the task list, as you can see in the following Task Manager screen shot:

If this program had exhibited rootkit behavior, its process would be hidden. We can simulate this by hiding the  malware.exe process using a rootkit, such as one called FU. By executing the command "fu -ph" and then supplying the process ID of the malicious program, we can hide the process from Task Manager.

If Helios is running in the background, it can detect the attempt to hide this process and alert you about it:

Helios can also allow you to unhide the concealed process with a click of a button:

Once unhidden, the malicious process is visible in Task Manager again.

Although Helios shows promise, it is still clearly a work a progress. For instance, I was unable to use Helios to detect a process concealed with FUTo, a newer version of the FU rootkit. Also, activating some features of Helios crashed my VMware virtual machine. I hope the program's authors continue their efforts to make it production-ready.

Another program that is definitely worth mentioning is IceSword, which offers a collection of utilities that can help locate rootkit-concealed programs. For example, even after I hid malware.exe using FU rootkit, IceSword listed the malicious process among the processes running on the infected system:

If you know of other helpful tools for analyzing rootkit-like malicious software, please let us know. We'll be glad to hear from you.

Lenny Zeltser
0 comment(s)


Diary Archives