Threat Level: green Handler on Duty: Rick Wanner

SANS ISC: Announcing: The "404 Project" - SANS Internet Storm Center SANS ISC InfoSec Forums


Sign Up for Free!   Forgot Password?
Log In or Sign Up for Free!
Announcing: The "404 Project"

We all know that web applications are the new firewall. However, so far we had a hard time collecting web application logs. The hard part is to balance ease of install of a sensor (without disrupting the web application), fidelity of the log information and privacy.

With firewall logs, it is pretty simple. A rejected packet in a firewall has very little information and privacy isn't a big issue. Web application are different as the actual "meat" of the log event is in the request content, which may contain personal information. Parsing web logs isn't so easy either. Administrators frequently customize log formats for special purposes.

To balance these different issues we decided to focus on errors, but instead of parsing logs, we set up a little php script that you can add to your error page. In its current form, the script will work with PHP web servers (tested with Apache) that support the curl extension. Curl is installed by default in current versions of PHP.

Now all you need is an "error page". In Apache, just use the ErrorDocument configuration directive. For example:

ErrorDocument 404 /error.html

Will redirect users to "/error.html" in case of a 404 error [1].  You may already have a page like that configured. All you need to do is add the php snippet to the end, sending us the intended URL, the user agent and the IP address of the client access the missing page.

The hope is to collect data from automated probes, similar in how DShield's firewall logs reflect portscan activity.

In particular if you are running a personal / home web server: Please consider adding the collector script.

Once we get a few submitters, we will start adding continuously updated reports to the site, just like we do for the DShield data. However, we can't do this until we have at least a dozen submitters (better 100 or more) . We can not publish "one off" errors as they will likely be specific to your site and again could cause privacy issues.

Why do we only support PHP? Well, that's the language I know. Feel free to submit a .Net/Java/Ruby/Perl or whatever version of the script.

Simple steps to sign up:

  1. Login to retrieve your authentication key here https://isc.sans.edu/myinfo.html
  2. Download the php snippet here https://isc.sans.edu/tools/404project.html
  3. paste it into your Error Document
  4. test...

Please contact us if you have any questions.

[1] http://httpd.apache.org/docs/2.0/mod/core.html#errordocument

------
Johannes B. Ullrich, Ph.D.
SANS Technology Institute
Twitter

I will be teaching next: Defending Web Applications Security Essentials - SANS Security West 2019

Johannes

3508 Posts
ISC Handler
I have a better idea! If you are using a home server to run PHP based web access, TURN OFF PHP!

PHP is very dangerous if you do not configure it correctly, and if you do not understand EXACTLY what your applications are capable of doing!

A home user??? PHP??? What were you thinking!

Sorry but that is just crazy. Please change your article to warn would-be home server people of the dangers. For professionals, yes fine, but I for one will not run PHP do to it's hack history.

PHP does not even exist on my servers. I like it that way! I take the time to code in alternate languages. It is worth the time to learn, and the extra time to code.

PHP is also the language choice of hackers. Probably for two reasons..

1 - It is easy to learn.

2 - It is easy for the server administrator or the web application developer to make critical mistakes that can open the door to your data or your entire network.

Write that RUBY, PERL or APACHE Module in C now :-)
Al of Your Data Center

80 Posts
C ... a language that brought us fun features like buffer overflows and format string errors ;-)

If you don't need php, turn it off like any feature you don't need. But having a home web server to experiment is perfectly fine. Manage it well, monitor it, and make your mistakes with it before you start coding a real site with real customer data. DShield.org started out as an experiment like that. Just having it hosted in a "real datacenter" didn't make it any more secure.
Johannes

3508 Posts
ISC Handler
As far as C goes, Microsoft was guilty of sloppy coding. When you compile a program you get warnings. You can set the severity of warnings to extreme and produce clean compiled code with some effort, or you can ignore the errors and release the project anyway as Microsoft and many others did. There is no excuse for releasing "features" like they did :-) Today we still are cleaning up those "features".

In the average home environment (there are many exceptions of course) you do not usually have all the technology you have in a data center to control flow, monitor traffic patterns, limit applications by their signature, etc. It is a risk, but true, the data the "visitor" gets will most likely be worthless to them.

The problem is when a machine gets owned. It can become a menace, like those IP addresses in China that attack regularly. Chances are the owners don't even know it is happening and are just pawns in many cases. Not all cases of course, but many.

So, yes I agree that it can be beneficial to test in a non-critical environment, but you also miss out on all of the great tools data center engineers have in place to control events.
Al of Your Data Center

80 Posts
On the positive side, well over 90% of the alerts from our web app firewall are for scans and attacks against PHP code. So you should get a lot of data. :-)

If we were running PHP I would already have recommended we re-write everything based just on the web app firewall data.

Although we whitelist everything to minimize our attack surface, we did make an exception and blacklisted requests looking for .php files and set an automatic IP Block on the source regardless of what the request contained. It's that prevalent.
Anonymous
Well, can anybody offer up versions of the script in any other languages?

I know how to write perl script but not perl script to run on a webserver and not perl that I'd want to be probed by everybody and their uncle from the Internet.

I looked at the PHP and have about half of an idea on how to convert it to Perl. It would require libcurl and at least two modules WWW::Curl and MIME::Base64, but that's as far as I got.

I don't know how to pull the variables out of Apache to get User Agent, Redirect URL, or Remote URL.

Also, I'm not sure if perl has built-ins that would be better suited than calling out to libcurl.
Jasey

93 Posts
I run php only because there is an application that is open sourced that we need. I do not write in php. All my web stuff is in perl.

Nonetheless, mt apache web server writes a log file to /var/log/apache2/errors that can be post-processed to generate a report, similar to the dshield firewall logs stuff. Why do you need a script at all to run when the web server runs? Just batch process the logs in a cron job and ship them off to isc.
Moriah

133 Posts
PERL-101 ...

Create a list of valid environment variables. These are the ones you will want.

@valid_ENV = ('REMOTE_HOST','REMOTE_ADDR','REMOTE_USER','HTTP_USER_AGENT');

..then filter upon them to make sure nothing else gets by.

Example of how to use the variables..

$host = $ENV{'REMOTE_ADDR'};
Al of Your Data Center

80 Posts
BTW the variables are from APACHE, not PERL. PERL is just the interface to them. You can get them all at apache.org. They would work the same in PHP by the way.
Al of Your Data Center

80 Posts
"Al of Your Data Center" you need to chill. Who makes you the authority on what Home users do with their Internet?

And while you are chilling...read a book on Netiquette. All the unnecessary capitalization and punctuation marks are how 9 year olds type.
HackDefendr

65 Posts
Too funny! Glad you enjoyed reading my posts. I always find it interesting when someone takes a post I make to heart. I don't know why you would be defensive here, but no matter. Flames are fun too. They expose human vulnerabilities, which are just like those on the Internet... open to exploitation :-)
Al of Your Data Center

80 Posts
Oh, and while I am at it.. those capital letters are because they are abbreviations for larger words and are supposed to be capitalized. All except for the "TURN OFF PHP" that is :-)
Al of Your Data Center

80 Posts
I implemented the code, but (luckily) my hosting company doesn't allow outgoing connections from within PHP.
I can imagine that many companies also don't allow their web servers to make direct connections to the Internet.
Lode

5 Posts
We have a fairly liberal comment policy. But let's be careful not to cross the line between spirited discussion and personal attacks.
Rick

290 Posts
ISC Handler
I'm game. You should get some cool stuff from my website (gets scanned a lot). It's been online for probably 10+ years.
Ryan Greenier

3 Posts
It's all set up on our public site. According to OSSEC, we get a lot of scans and probes. By the way, the account confirmation email and account setup email were both blank. I seem to have been able to work around it.
Anonymous
"Step 4: test..."

Other than the code returning an "OK", is there any way to confirm that data is making to isc?
RTAdams89

2 Posts
What the heck - I am in on one of my sites. Cannot wait to see the information. Not getting an error, so I presume its working.
Anonymous
if you log in to your account, you should now is a link to a page that summarizes the logs you submitted today. (isc.sans.edu/my404.html)
Johannes

3508 Posts
ISC Handler

Sign Up for Free or Log In to start participating in the conversation!