[Guest Diary: Didier Stevens] [Shellcode Detection with XORSearch]

Frank Boldewin (http://www.reconstructer.org/) developed a shellcode detection method to find shellcode in Microsoft Office files, like .doc and .xls files. He released this as a feature of his OfficeMalScanner tool (http://www.reconstructer.org/code.html).

I consider this a very interesting detection method, and wanted to use this method on other file types like pictures. That’s what motivated to integrate this in my XORSearch tool.

XORSearch has been presented here before. It’s a string search tool that brute-forces the content of the searched file with simple encoding methods like XOR, ROL, … Say that you have a malware sample that downloads a file. You want to know the download URL, but the strings command will not find the URL, because it is encoded with XOR key 0xD1. XORSearch will find the URL like this: xorsearch malware.exe http

At the beginning of this year, I extended XORSearch beyond string searching: with option –p, it will find embedded PE-files (executables).

And now, shellcode is the next target.

Frank was kind enough to share his shellcode detector’s source code with me. But I wanted a flexible detector, one that can be tailored by the user without coding. So I developed a syntax for Frank’s shellcode detection rules and converted his source code with this new syntax. Let me explain with an example.

32-bit shellcode needs to establish its position in memory. A common method is known as “Get EIP” and uses these 2 instructions:

        call label

label:

        pop eax

The hex code for these instructions is: E80000000058

But another register than eax can be used, and then the hex code is not 58 but something else, for example 59. To handle different registers, my syntax supports bit wildcards: E800000000(B;01011???)

This will match E80000000058, E80000000059, …, E8000000005F. All with one pattern.

A pattern has a name and a score, this makes up a rule:

GetEIP method 1:10:E800000000(B;01011???)

The name of the rule is “GetEIP method 1”, the score is 10. Each time a match is found, the rule’s score is added to the total score.

To use XORSearch’s shellcode detector with Frank’s rules, you use option –W (wildcard).

(option –d 3 disables ROT encoding brute-forcing: ROT generates too much false positives with shellcode detection)

You can see from the screenshot that many detection rules triggered on this sample, and that the total score is 136.

To view all the rules I embedded in XORSearch, issue command xorsearch –L.

And if you want to provide your own rules, use option –w. I explain the rule syntax in detail in this blogpost:

http://blog.didierstevens.com/2014/09/29/update-xorsearch-with-shellcode-detector/

XORSearch is open source written in C, without OS-specific calls. I publish the source code and binaries for Windows, OSX and Linux.

Download XORSearch: http://blog.didierstevens.com/programs/xorsearch/

-- 
Alex Stanford - GIAC GWEB & GSEC,
Research Operations Manager,
SANS Internet Storm Center