Apache HTTP DoS tool mitigation - SANS Internet Storm Center

Apache HTTP DoS tool mitigation
If you've been following our diaries or any other IT Security related news, you probably know about the Apache HTTP DoS tool that was released on Thursday (see the original diary at http://isc.sans.org/diary.html?storyid=6601). We received some information from our readers and I did a bit more testing so I decided to put up a diary with possible mitigation options – I hope to update the diary as we receive more stuff. So, to recap the previous diary: the tool works by exhausting Apache processes; this is done by sending incomplete request headers so Apache keeps waiting for the final header line to arrive, the tool instead just sends a bogus header to keep the connection open. Besides Apache (both versions 1.x and 2.x), Squid is also affected. Knowing how many servers running on Apache there are, this makes the tool very dangerous since it doesn't require absolutely any knowledge from the attacker – all he/she has to do is run the tool and the target site goes down. So, what mitigation options (if any) we have? First of all, those running some load balancers or reverse proxies in front of their Apache installations should check if they are really vulnerable or not – it's possible that the proxy is not affected by this vulnerability. If you are running a standalone Apache server (as probably most web hosters are), you still have some mitigation options, but you should be very careful about them. NOTE: Test all these options thoroughly first as they might impact legitimate access to your server! The TimeOut Directive, described at http://httpd.apache.org/docs/2.2/mod/core.html#timeout is the most important option here. It defines how long Apache will wait to receive a GET request, or amount of time to wait between receiving packets on POST requests. This is particularly interesting for POST requests and warrants additional research to see if the tool can be modified further to attack such, hardened servers. So, the default for TimeOut is 300 seconds – 5 minutes – for a GET request to arrive. This means that the DoS tool can keep one connection tied up for up to 5 minutes, after that it has to open another connection. Since the default number of connections is 256 the tool can easily exhaust this. I did some testing with lowering this value down to 5 seconds – this might be too low because some slow clients will not be able to get the web page so don't blindly put this in your configuration files. Anyway, since the tool has the option for specifying timeout, I put it to 4 seconds to see if the attack still worked. And sure, it worked fine, but the tool had to send much more traffic. My measurements showed that in order to DoS an Apache server with TimeOut set to 5 seconds the tool needed about 2MB of traffic per minute. That is only ~45 kb/sec which makes this attack even more dangerous – a lone DSL user might DoS a big site. However, the TimeOut directive is still probably the best mitigation option. Another possibility is also to use a module, such as mod_limitipconn, which limits the number of connections from one IP address. Again, use this with caution since you might prevent legitimate visitors if they are coming from behind a proxy. Also, this option is only partially good – sure, it will stop a script kiddie without other resources from DoS-ing your web site, but an advanced attacker can easily run the tool from dozen of machines. And this is, unfortunately, more or less what we have at the moment. I will do more research on the scenario with the POST request and TimeOut directive – will post an update if I find something interesting. If you did some research and have workarounds please let us know. -- Bojan I will be teaching next: Web App Penetration Testing and Ethical Hacking - SANS Prague August 2019	Bojan 379 Posts ISC Handler
Subscribe	Jun 21st 2009 1 decade ago
I'm glad to have already adopted nginx as a reverse proxy. I'd experienced 'DoS'-like situations caused by legitimate traffic on a busy site where a server-side script had stalled waiting for something and the number of open connections exceeded Apache's maximum. Large PHP scripts cause Apache to use a lot of RAM per thread/process so I couldn't increase MaxClients further. But nginx can handle a lot of clients with minimal RAM, so it's possible to even have large KeepAliveTimeout's for better performance without risk of having too many open connections. I limit connections per IP and the speed of connection attempts per IP, to protect against large DoSes.	Steven C. 171 Posts
Quote	Jun 21st 2009 1 decade ago
thats the problem, apache will reset the timer if at least one byte is received before timer expiration. lighttpd suffers the same symptoms as well as at least two other closed source http servers. steven's adoption seems the best quick fix for the moment but if a better one should come it will be written following freebsd's accf filters philosophy, altho that will not fix the case of ssl or big post data.	Steven C. 7 Posts
Quote	Jun 22nd 2009 1 decade ago
Robert Hansen keeps a blog entry: http://ha.ckers.org/blog/20090617/slowloris-http-dos/ on this topic that you may want to check	Steven C. 7 Posts
Quote	Jun 22nd 2009 1 decade ago
anyway, this is not a bug, neither i can call it a flaw, it is just a friendly manner to serve the requests that can be exploited by the bad guy. i don't see anything wrong (ethically speaking) in resetting that timer the way it is (think congestions and the fact that there MAY be real slow clients) but ... one should be prepared to say "I'm sorry but i have to drop you" to the most lazy connection. care should be taken when computing laziness factor on big posts.	Steven C. 7 Posts
Quote	Jun 22nd 2009 1 decade ago
Adrian, agree with everything you wrote. Your post, as well as Slowloris can indeed make an Apache server suffer which can be particularly nasty for web hosting companies (for example, not making their SLAs). There is an unofficial patch for Apache available, I'll check it to see if it helps.	Bojan 379 Posts ISC Handler
Quote	Jun 22nd 2009 1 decade ago
there are more tools that you may use to test for vulns without attacking the target and a tool that one could use to do a ddos. namely: ap_finger - to guess the timeouts chinese_death - to check if the server is resetting the read timeout killap - ddos tool all may be found at http://pub.mud.ro/~cia/computing/apache-httpd-denial-of-service-example.html but you need to fix the compile-time errors that keep away bad guys without brains. or i can email to sans working versions	Bojan 7 Posts
Quote	Jun 22nd 2009 1 decade ago
http://pub.mud.ro/~cia/files/deadsnail/ the fingerprint and vulcheck programs as done 2 years ago (works the same :P )	Bojan 7 Posts
Quote	Jun 22nd 2009 1 decade ago

If you've been following our diaries or any other IT Security related news, you probably know about the Apache HTTP DoS tool that was released on Thursday (see the original diary at http://isc.sans.org/diary.html?storyid=6601).

We received some information from our readers and I did a bit more testing so I decided to put up a diary with possible mitigation options – I hope to update the diary as we receive more stuff.

So, to recap the previous diary: the tool works by exhausting Apache processes; this is done by sending incomplete request headers so Apache keeps waiting for the final header line to arrive, the tool instead just sends a bogus header to keep the connection open. Besides Apache (both versions 1.x and 2.x), Squid is also affected. Knowing how many servers running on Apache there are, this makes the tool very dangerous since it doesn't require absolutely any knowledge from the attacker – all he/she has to do is run the tool and the target site goes down. So, what mitigation options (if any) we have?

First of all, those running some load balancers or reverse proxies in front of their Apache installations should check if they are really vulnerable or not – it's possible that the proxy is not affected by this vulnerability.

If you are running a standalone Apache server (as probably most web hosters are), you still have some mitigation options, but you should be very careful about them. NOTE: Test all these options thoroughly first as they might impact legitimate access to your server!

The TimeOut Directive, described at http://httpd.apache.org/docs/2.2/mod/core.html#timeout is the most important option here. It defines how long Apache will wait to receive a GET request, or amount of time to wait between receiving packets on POST requests. This is particularly interesting for POST requests and warrants additional research to see if the tool can be modified further to attack such, hardened servers.

So, the default for TimeOut is 300 seconds – 5 minutes – for a GET request to arrive. This means that the DoS tool can keep one connection tied up for up to 5 minutes, after that it has to open another connection. Since the default number of connections is 256 the tool can easily exhaust this. I did some testing with lowering this value down to 5 seconds – this might be too low because some slow clients will not be able to get the web page so don't blindly put this in your configuration files. Anyway, since the tool has the option for specifying timeout, I put it to 4 seconds to see if the attack still worked. And sure, it worked fine, but the tool had to send much more traffic. My measurements showed that in order to DoS an Apache server with TimeOut set to 5 seconds the tool needed about 2MB of traffic per minute. That is only ~45 kb/sec which makes this attack even more dangerous – a lone DSL user might DoS a big site. However, the TimeOut directive is still probably the best mitigation option.

Another possibility is also to use a module, such as mod_limitipconn, which limits the number of connections from one IP address. Again, use this with caution since you might prevent legitimate visitors if they are coming from behind a proxy. Also, this option is only partially good – sure, it will stop a script kiddie without other resources from DoS-ing your web site, but an advanced attacker can easily run the tool from dozen of machines.

And this is, unfortunately, more or less what we have at the moment. I will do more research on the scenario with the POST request and TimeOut directive – will post an update if I find something interesting. If you did some research and have workarounds please let us know.

--
Bojan

I will be teaching next: Web App Penetration Testing and Ethical Hacking - SANS Prague August 2019

Bojan

379 Posts
ISC Handler

I'm glad to have already adopted nginx as a reverse proxy. I'd experienced 'DoS'-like situations caused by legitimate traffic on a busy site where a server-side script had stalled waiting for something and the number of open connections exceeded Apache's maximum. Large PHP scripts cause Apache to use a lot of RAM per thread/process so I couldn't increase MaxClients further. But nginx can handle a *lot* of clients with minimal RAM, so it's possible to even have large KeepAliveTimeout's for better performance without risk of having too many open connections. I limit connections per IP and the speed of connection attempts per IP, to protect against large DoSes.

Steven C.

171 Posts

thats the problem, apache will reset the timer if at least one byte is received before timer expiration. lighttpd suffers the same symptoms as well as at least two other closed source http servers. steven's adoption seems the best quick fix for the moment but if a better one should come it will be written following freebsd's accf filters philosophy, altho that will not fix the case of ssl or big post data.

Steven C.
7 Posts

Robert Hansen keeps a blog entry: http://ha.ckers.org/blog/20090617/slowloris-http-dos/ on this topic that you may want to check

Steven C.
7 Posts

anyway, this is not a bug, neither i can call it a flaw, it is just a friendly manner to serve the requests that can be exploited by the bad guy. i don't see anything wrong (ethically speaking) in resetting that timer the way it is (think congestions and the fact that there MAY be real slow clients) but ... one should be prepared to say "I'm sorry but i have to drop you" to the most lazy connection. care should be taken when computing laziness factor on big posts.

Steven C.
7 Posts

Adrian, agree with everything you wrote. Your post, as well as Slowloris can indeed make an Apache server suffer which can be particularly nasty for web hosting companies (for example, not making their SLAs).
There is an unofficial patch for Apache available, I'll check it to see if it helps.

Bojan

379 Posts
ISC Handler

there are more tools that you may use to test for vulns without attacking the target and a tool that one could use to do a ddos.
namely:
ap_finger - to guess the timeouts
chinese_death - to check if the server is resetting the read timeout
killap - ddos tool
all may be found at http://pub.mud.ro/~cia/computing/apache-httpd-denial-of-service-example.html but you need to fix the compile-time errors that keep away bad guys without brains. or i can email to sans working versions

Bojan
7 Posts

http://pub.mud.ro/~cia/files/deadsnail/
the fingerprint and vulcheck programs as done 2 years ago (works the same :P )

Bojan
7 Posts