Experimental New Domain / Domain Age API

Published: 2022-06-21
Last Updated: 2022-06-21 14:41:50 UTC
by Johannes Ullrich (Version: 1)
4 comment(s)

One of our goals is to provide data to "color your logs" (or "Augment" them, as vendors may say). I have been experimenting with various ways to get simplified access to "domain age" data for a while now. This means not just data about new domains but how old a particular domain is. It may be an interesting parameter to add to when investigating.

To make it easier to retrieve this data, we now have two new API functions, and I may finally document them properly at https://isc.sans.edu/api (where you will find all the other random data we make available). I have been playing with this for a while and may have posted about it, but now it is as ready as it will be for a while.

Lookups are simple:

curl --user-agent 'this is Dr. J' 'https://isc.sans.edu/api/domainage/sans.org?json'

Just replace "sans.org" with the domain you are interested in.

For domains "first seen" on a particular date, try:

curl --user-agent 'this is Dr. J' 'https://isc.sans.edu/api/recentdomains/2022-06-01?json'

if you omit the date, the last date ("today") is returned. This only works for dates one month back.

Quick FAQ:

  • Where does the data come from?
    Multiple sources. Some domains we discover by seeing them in our web/ssh/firewall log data. Some comes from registrars, some from certificate transparency logs. Some of the old domain data comes from "whois" lookups.
  • How "good" is the "firstseen" date?
    We call it "firstseen" for a reason. This is the first time we have seen the domain. It may be older. Sometimes this is based on whois data, but not always.
  • What is the rate limit / SLA for this API:
    Right now, we do not have a strict rate limit. But this is meant for occasional, not bulk, lookup. One lookup a second, maybe a thousand or so a day, should be good. We do not do API keys or authentication. But please add some information to the user agent that allows us to reach out in case of a problem. Some default user agents may get blocked, so customize your user agent (we want to get at least rid of requests that are too lazy to alter their user agent)
  • Are there any restrictions on usage?
    Do not resell. Other than that, you are OK to use it. Please attribute. Our standard "creative commons" license applies if you are interested in details. Please ask us if you have questions.
  • What is the data quality?
    That is what I want you to tell me? See errors/omissions? Let us know. The data is provided "as-is" (but you will get the money back that you didn't pay if something is wrong)
  • What is the "type" about?
    Treat it as a "comment," but it is still being developed.
  • What output formats do you support
    RTFM at isc.sans.edu/api 

---
Johannes B. Ullrich, Ph.D. , Dean of Research, SANS.edu
Twitter|

Keywords: domains api
4 comment(s)

Comments

I tried a few old domains (cisco,ibm, mit etc.) and the firstseen is always 2011-01-31.

It seems that is the earliest date for old domains or database was built then.

Thanks,
Rock
yes, old domains have this "artificial" first seen date. I started working on this last year and imported a list of old and popular domains and just set the first seen date to Jan 31st, 2011. A bit of a random date, but essentially just means: This domain is old enough where the exact registration date probably doesn't matter.
Is this somehow related to domain_stats / domain_stats2 by Mark Beggett, with the ISC integration described here? https://github.com/MarkBaggett/domain_stats2
that was the original motivation for this feature. But the complete integration as outlined in the GitHub readme is not ready yet.

Diary Archives