Web Traffic Analysis with httpry

Published: 2010-07-30
Last Updated: 2010-07-30 01:00:12 UTC
by Guy Bruneau (Version: 1)
9 comment(s)

httpry is a tool specialized for the analysis of web traffic. The tool itself can be used to capture traffic (httpry -o file) but other other tools are better suited for that such as tcpdump, Snort, Sguil. When it comes to finding out if certain types of files were downloaded via http, this tool does a super job. It can be used in combination with regular expressions (Regex) to find if a file, a script or a malware was downloaded from site or by a host and will ignore everything else. Whether the http traffic is using port 80, 443, 8080, etc, it will parse and display all the web traffic using this simple command:

httpry -i eth0

If you are working with a large pcap file and want to filter on a particular IP or network, httpry support libpcap filters to zoom in on the web traffic you want to analyze. This libpcap filter will show all the web traffic associated with host using this filter:

httpry -r file 'host'

07/28/2010 18:00:02 > GET www.symantec.com /enterprise/security_response/threatexplorer/threats.jsp HTTP/1.0 - -
07/28/2010 18:00:02 < - - - HTTP/1.0 301 Moved Permanently
07/28/2010 18:00:02 > GET www.symantec.com /business/security_response/threatexplorer/threats.jsp HTTP/1.0 - -
07/28/2010 18:00:03 < - - - HTTP/1.0 200 OK
07/28/2010 18:00:03 > GET vil.nai.com /VIL/newly_discovered_viruses.aspx HTTP/1.0 - -
07/28/2010 18:00:03 > GET vil.nai.com /VIL/newly_discovered_viruses.aspx HTTP/1.0 - -
07/28/2010 18:00:03 < - - - HTTP/1.1 200 OK
07/28/2010 18:01:48 < - - - HTTP/1.1 200 OK
07/28/2010 18:01:48 > GET safebrowsing-cache.google.com /safebrowsing/rd/ChNnb29nLW1hbHdhcmUtc2hhdmFyEAEYlZQCIJaUAioFFooAAAEyBRWKAAAB HTTP/1.1 - -
07/28/2010 18:01:48 < - - - HTTP/1.1 200 OK

If you are checking for a particular file extension such as.exe, .js, .msi, .jpg, etc, if you combined your search with grep, httpry can be used to find if any binaries (i.e. malware) were downloaded from a certain site or by a particular client using a pcap captured files. In this example we grep for all the JavaScript transffered by host

httpry -r file 'host' | grep "\.js"

07/28/2010 10:57:08 > GET www.quickquote.lincoln.com /static/com/forddirect/presentation/constants/SkinConstants_lincoln.js HTTP/1.1 - -
07/28/2010 10:57:08 > GET www.quickquote.lincoln.com /yui/yahoo-dom-event/yahoo-dom-event.js HTTP/1.1 - -
07/28/2010 10:57:08 > GET www.quickquote.lincoln.com /static/com/forddirect/application/bp20/metrics/s_code.js HTTP/1.1 - -

The httpry website is here. The tarball can be download here and a freeBSD port here.


Guy Bruneau IPSS Inc. gbruneau at isc dot sans dot org

Keywords: httpry
9 comment(s)


How is this tool different from wireshark ? We can capture do the same in wireshark right ? Educate me if i'm missing something here.
httpry is a command line tool and will only shows web links (i.e. GET, POST, HEAD, etc) and the answer from the web site when the traffic is replayed which cannot be done using Wireshark. While it is possible to do something similar with tshark (tshark -T text -R "tcp.port == 80" -r file.pcap), it is not as clean and you would have to include all the ports you suspect might be transferring web traffic. httpry doesn't require that. If you are working with large pcap files, tshark also has limitation on the size of the file it is willing to work with.
@Guy: You've been missing out on some of the best tshark features: use tshark -r file.pcap -R "http" and it will decode GET/POST on any port (if memory serves) but will also put in MIME types as well as return codes. Best of all, you can code LUA scripts to automate various things. Unfortunately, the coolest feature, HTTP object export, is not possible via tshark or the LUA interface when last I checked. So, though httpry is cool, I think I agree with charlie: how is this any better?
the difference between tshark and a tool like httpry is performance; on a 1 Gig pipe network with high web traffic, httpry-like tool (and snort for sure) will outperform tshark.
Actually, tshark will certainly outperform Snort for straight URL logging. Even when using unified2, Snort still has trouble keeping up logging more than a few hundred alerts per second. I typically use urlsnarf from the dsniff suite if I want to do high-bandwidth URL logging. It will handle around 1000 URL's per second on older AMD64 hardware. Does anyone have experience comparing urlsnarf and httpry for speed?
Actually, I have ran several test between tshark and httpry on large files (several Gig) using a shell scripts and httpry easily outperform tshark. This is simple binary to compile and very compact (~200K) and quite fast.
Yes, I've just run tests of my own and they show that httpry is in fact very fast. (My original comment was not about speed, just functionality.) In fact, httpry easily outperforms urlsnarf in every way, so I think it's safe to say now that httpry is the fastest possible way to log URL's.
I want to combine with grep:
... -m get ...| grep "[\.js|\.zip|\.arj|\.tar]"
this will not work perfectly. why? can anybody help. thanks a lot. yours klaus

You need to escape the pipe for that to work

-m get ...| grep "\.js\|\.zip\|\.arj\|\.tar"

Diary Archives