Threat Level: green Handler on Duty: Brad Duncan

SANS ISC: Domain Whitelisting With Alexa and Umbrella Lists - update - SANS Internet Storm Center SANS ISC InfoSec Forums


Sign Up for Free!   Forgot Password?
Log In or Sign Up for Free!
Domain Whitelisting With Alexa and Umbrella Lists - update

A was asked if I could share the files of my last diary entry: Domain Whitelisting With Alexa and Umbrella Lists.

You can find the files on my site here. And to teach you how to fish :-), here are the commands I used to produce these lists:

 

csv-cut.py -s "\t" 1 emd.txt > blacklist.txt
csv-lookup.py -s , -e blacklist.txt 0 top-1m-umbrella.csv 1 0 blacklist-umbrella.csv
csv-lookup.py -s , -e blacklist.txt 0 top-1m-alexa.csv 1 0 blacklist-alexa.csv

My csv tools can be found on my Beta GitHub repository.

My assumption when I read this blog post, was that the blacklisted domains would rank low in the Alexa and Umbrella lists. They don't, look at the histograms of the rankings.

Blacklisted domains with Alexa rank:

Blacklisted domains with Umbrella rank:

These long tail distributions indicate that blacklisted domains with higher ranks are more prevalent than those with lower ranks. This is also reflected in the ranking median (287,251 for Alexa and 393,879 for Umbrella) and average (350,553 for Alexa and 420,846 for Umbrella).

Conclusion: don't use Alexa and Umbrella top 1,000,000 lists as whitelists blindly, even if you just use the top 1000 or 10000.

Didier Stevens
Microsoft MVP Consumer Security
blog.DidierStevens.com DidierStevensLabs.com

DidierStevens

144 Posts
ISC Handler

Sign Up for Free or Log In to start participating in the conversation!