Read This If You Are Using a Script to Pull Data From This Site

Published: 2017-05-10
Last Updated: 2017-05-10 14:05:53 UTC
by Johannes Ullrich (Version: 1)
2 comment(s)

I love it when people write tools to pull data from this site, and we try to accommodate automated tools like this with our API. but sometimes, scripts go bad and we keep having cases were scripts pull the same data several times a second. I would love to let the owner of the script know, but often this is hard.

To prevent some of these issues, I am going to enforce a new rule going forward: Your User-Agent has to include a contact for the script. I prefer a simple e-mail address. A URL will do if that is easier for you. The data will exclusively be used to contact you in case of a problem.

To enforce this, generic user agents will be blocked (like "Python-urllib/2.7", "Wget/1.12 (linux-gnu)", "curl/7.38.0"). I will start doing so with older pages that should no longer be used by automated scripts anyway (as they are not designed for automation like our API), and initially only block specific User Agents.

If you hit the page with a blocked User Agent, a "403" error will be returned (Forbidden) and a simple text message pointing to this post [1]. 


Johannes B. Ullrich, Ph.D., Dean of Research, SANS Technology Institute

Keywords: api rate limit
2 comment(s)

OAuth, and It's High Time for Some Personal "Security-Scaping" Today

Published: 2017-05-10
Last Updated: 2017-05-10 02:16:35 UTC
by Rob VandenBrink (Version: 1)
1 comment(s)

After Bojan's recent story on the short-lived Google Docs OAuth issues last week (, I got to thinking.  The compromise didn't affect too many people, but it got me thinking about OAuth.  The piece of OAuth that I focused on is the series of permisssions and tokens that allow interaction between applications, which is what the recent compromise took advantage of. 

My personal mantra is "the best day to change the password for "X" is today", and as part of this I've expanded that proverb to include looking at application permissions and privacy settings!

For instance, using Google’s “Security Checkup” at , I found that at some point in the past, I granted TripAdvisor access to my Gmail account. This wasn’t intentional, it was probably an “OK” prompt during an install or update process – you know, the ones you sometimes just click quickly / accidentally without paying attention to?  Then wonder if you just clicked something dumb right after? Anyway, yes, one of those - *click* - gone now!

I moved on to Facebook - application settings are here:

and privacy settings are here:

Really, everything in that page needs to be looked at!. Me, I was surprised to find that I was using an older email address for my Facebook login (oops) –with the login buried in my iPad app, it wasn’t something I had thought about (plus I’m not in facebook too much lately)

Other sites of interest:


In particular:



Really, most apps that you run have a privacy or a security page – it never seems to be front-and-center though, in fact for many of the apps I access primarily from a dedicated app on my phone or tablet, I needed to go to the “real” application in my browser to find these settings.

As you go, be sure to translate the security questions to plain English. For instance, from Google’s “privacy checkup”, you’ll see:

From another perspective this can mean “do you want to give Google access to your telephone number and link that back to your identity?”  Since that information is likely in the phone book (and the online version of the phone book), the answer might very well be yes, but that’s not how it was asked ..

The Google privacy checkup is a really good one to run through. I found my Youtube history (now deleted, thanks!), a map of my travels from Google Maps and my Chrome history – all gone now that I've seen it and clicked that handy *delete* button. In an academic way we all know that “Google knows all”, but is creepy to find all in one place like that. If you found an actual person tracking that information on you it’d probably be something you’d go to a lawyer about!

Kudos to Google for giving you access to that info and control over sharing it, but it certainly isn’t front-and-center by any stretch!

Shifting gears (and away form OAuth a bit), the application settings on your phone is another place to find stuff leaking that you haven't been thinking of.  That silly app you installed 2 or 3 years ago?  Maybe it's got background access to your location and contacts (I can't think why a mirror app has a legit need for that info, except to sell it) - today is a good day to dig into these settings too.

On my iPad / iPhone, some of those settings are in  Settings / Privacy, and others are in Settings directly (look for the applicatio name) – here you are looking for oddball permissions. For instance, apps that over time you have granted access to your location or contacts that don’t need that information. You’ll want to go over all of your access - - Your fitness app for instance likely needs your location info but not your contacts. In my case, my hotel “loyalty app” had access to my location – this sort-of makes sense if you're looking for a hotel "right now", but it’s not something I wanted them to have.

If you’ve had your phone for a while, you’ll likely be surprised to see who and what applications have access to your location, your contacts, inbox and calendar, your camera and microphone – this really is a good thing to revisit periodically, maybe when you change your smoke alarm batteries? (and today of course)

While you’re in there, if you’ve got embedded passwords how about maybe changing those today too? There are a large number of folks who still use the same passwords for everything, and over the years we’ve seen compromises at Yahoo, at Facebook and lots of other major (and minor) services. If you haven’t changed your Gmail (or whatever) password in a year or two, today is an EXCELLENT day to do this. Pick something long for a password (longer than 15 characters, but really the longer the better). The key is to use different, complex passwords for as many things as possible. Especially if your approach is to bury the password in the app settings on your phone, there's no reason it needs to be easy for you to type or remember.  If you don’t already use a password manager to keep track of these, today is a good day to consider that as well.  Many password managers will do a good bit of this password change stuff for you!

Better yet, investigate changing as many services as possible to two factor authentication.

I think I’ve just taken up half of your day with the list above, but while we’re on a roll, what else should we be looking at? What have I missed?  By all means use our comment form and add to the list!

Rob VandenBrink

1 comment(s)


Diary Archives