Web Traffic Analysis with httpry - SANS Internet Storm Center

Web Traffic Analysis with httpry
httpry is a tool specialized for the analysis of web traffic. The tool itself can be used to capture traffic (httpry -o file) but other other tools are better suited for that such as tcpdump, Snort, Sguil. When it comes to finding out if certain types of files were downloaded via http, this tool does a super job. It can be used in combination with regular expressions (Regex) to find if a file, a script or a malware was downloaded from site or by a host and will ignore everything else. Whether the http traffic is using port 80, 443, 8080, etc, it will parse and display all the web traffic using this simple command: httpry -i eth0 If you are working with a large pcap file and want to filter on a particular IP or network, httpry support libpcap filters to zoom in on the web traffic you want to analyze. This libpcap filter will show all the web traffic associated with host 192.168.5.25 using this filter: httpry -r file 'host 192.168.5.25' 07/28/2010 18:00:02 192.168.5.25 216.66.8.10 > GET www.symantec.com /enterprise/security_response/threatexplorer/threats.jsp HTTP/1.0 - - 07/28/2010 18:00:02 216.66.8.10 192.168.5.25 < - - - HTTP/1.0 301 Moved Permanently 07/28/2010 18:00:02 192.168.5.25 216.66.8.16 > GET www.symantec.com /business/security_response/threatexplorer/threats.jsp HTTP/1.0 - - 07/28/2010 18:00:03 216.66.8.16 192.168.5.25 < - - - HTTP/1.0 200 OK 07/28/2010 18:00:03 192.168.5.25 67.97.80.71 > GET vil.nai.com /VIL/newly_discovered_viruses.aspx HTTP/1.0 - - 07/28/2010 18:00:03 192.168.5.25 67.97.80.71 > GET vil.nai.com /VIL/newly_discovered_viruses.aspx HTTP/1.0 - - 07/28/2010 18:00:03 67.97.80.71 192.168.5.25 < - - - HTTP/1.1 200 OK 07/28/2010 18:01:48 74.125.157.101 192.168.5.25 < - - - HTTP/1.1 200 OK 07/28/2010 18:01:48 192.168.5.25 173.194.15.95 > GET safebrowsing-cache.google.com /safebrowsing/rd/ChNnb29nLW1hbHdhcmUtc2hhdmFyEAEYlZQCIJaUAioFFooAAAEyBRWKAAAB HTTP/1.1 - - 07/28/2010 18:01:48 173.194.15.95 192.168.5.25 < - - - HTTP/1.1 200 OK If you are checking for a particular file extension such as.exe, .js, .msi, .jpg, etc, if you combined your search with grep, httpry can be used to find if any binaries (i.e. malware) were downloaded from a certain site or by a particular client using a pcap captured files. In this example we grep for all the JavaScript transffered by host 192.168.5.25. httpry -r file 'host 192.168.5.25' \| grep "\.js" 07/28/2010 10:57:08 192.168.5.25 69.192.143.238 > GET www.quickquote.lincoln.com /static/com/forddirect/presentation/constants/SkinConstants_lincoln.js HTTP/1.1 - - 07/28/2010 10:57:08 192.168.5.25 69.192.143.238 > GET www.quickquote.lincoln.com /yui/yahoo-dom-event/yahoo-dom-event.js HTTP/1.1 - - 07/28/2010 10:57:08 192.168.5.25 69.192.143.238 > GET www.quickquote.lincoln.com /static/com/forddirect/application/bp20/metrics/s_code.js HTTP/1.1 - - The httpry website is here. The tarball can be download here and a freeBSD port here. ----------- Guy Bruneau IPSS Inc. gbruneau at isc dot sans dot org	Guy 439 Posts ISC Handler
Subscribe	Jul 30th 2010 8 years ago
How is this tool different from wireshark ? We can capture do the same in wireshark right ? Educate me if i'm missing something here.	Anonymous
Quote	Jul 30th 2010 8 years ago
httpry is a command line tool and will only shows web links (i.e. GET, POST, HEAD, etc) and the answer from the web site when the traffic is replayed which cannot be done using Wireshark. While it is possible to do something similar with tshark (tshark -T text -R "tcp.port == 80" -r file.pcap), it is not as clean and you would have to include all the ports you suspect might be transferring web traffic. httpry doesn't require that. If you are working with large pcap files, tshark also has limitation on the size of the file it is willing to work with.	Guy 439 Posts ISC Handler
Quote	Jul 30th 2010 8 years ago
@Guy: You've been missing out on some of the best tshark features: use tshark -r file.pcap -R "http" and it will decode GET/POST on any port (if memory serves) but will also put in MIME types as well as return codes. Best of all, you can code LUA scripts to automate various things. Unfortunately, the coolest feature, HTTP object export, is not possible via tshark or the LUA interface when last I checked. So, though httpry is cool, I think I agree with charlie: how is this any better?	Guy 13 Posts
Quote	Jul 31st 2010 8 years ago
the difference between tshark and a tool like httpry is performance; on a 1 Gig pipe network with high web traffic, httpry-like tool (and snort for sure) will outperform tshark.	Guy 3 Posts
Quote	Aug 1st 2010 8 years ago
Actually, tshark will certainly outperform Snort for straight URL logging. Even when using unified2, Snort still has trouble keeping up logging more than a few hundred alerts per second. I typically use urlsnarf from the dsniff suite if I want to do high-bandwidth URL logging. It will handle around 1000 URL's per second on older AMD64 hardware. Does anyone have experience comparing urlsnarf and httpry for speed?	Guy 13 Posts
Quote	Aug 2nd 2010 8 years ago
Actually, I have ran several test between tshark and httpry on large files (several Gig) using a shell scripts and httpry easily outperform tshark. This is simple binary to compile and very compact (~200K) and quite fast.	Guy 439 Posts ISC Handler
Quote	Aug 2nd 2010 8 years ago
Yes, I've just run tests of my own and they show that httpry is in fact very fast. (My original comment was not about speed, just functionality.) In fact, httpry easily outperforms urlsnarf in every way, so I think it's safe to say now that httpry is the fastest possible way to log URL's.	Guy 13 Posts
Quote	Aug 2nd 2010 8 years ago
I want to combine with grep: ... -m get ...\| grep "[\.js\|\.zip\|\.arj\|\.tar]" this will not work perfectly. why? can anybody help. thanks a lot. yours klaus	Klaus 1 Posts
Quote	Aug 4th 2010 8 years ago
Klaus, You need to escape the pipe for that to work -m get ...\| grep "\.js\\|\.zip\\|\.arj\\|\.tar"	Guy 439 Posts ISC Handler
Quote	Aug 29th 2010 8 years ago