Thanks to Gebhard for pointing out the article by Heise about a new spider focusing on finding web application vulnerabilities [1]. "Punkspider" runs essentially a vulnerabiliy scan on random web sites. The results are then searchable. I am not sure about the quality about the results (it doesn't find anything for isc.sans.edu ... ) but you may want to check your own site. There is also a simple, non documented at this point, json API:

http://punkspider.hyperiongray.com/service/search/domain/

Which accepts the following GET parameters:

searchkey: url|title
searchvalue: the url or title you would like to search for
pages: 0
pagesize: how many results (10 by default)
pagenumber: which page (1 by default)

For example:

http://punkspider.hyperiongray.com/service/search/domain/?searchkey=url&searchvalue=isc.sans.edu&pages=0&pagesize=10&pagenumber=1

The Heise article below has more details. Evidentially it is possible to block the spider via robots.txt but I haven't seen the user agent documented. (need to check my logs). Of course, you could block it in robots.txt, or return overly large, or wrong results based on the user agent. Maybe some fake vulnerabilities to see who is exploiting them later.

[1] http://www.h-online.com/security/news/item/Vulnerabilities-served-up-1810524.html

------
Johannes B. Ullrich, Ph.D.
SANS Technology Institute
Twitter