Ubuntu - protecting against "slow" DoS attack on apache2 web-server (defend from Slowloris)

Yesterday I've discovered some sort of attack on our web-site.
Some sort - because most likely it was caused by misconfigured client, but nevertheless it effectively put the web server down.



Effect
Ubuntu server 8.04 LTS, Apache 2 webserver.
Server works without any problems - you can ping, connect via ssh, FTP, receive or send email, etc, CPU load is minimal and absolutely not suspicious, but webpages are extremely slow and most of the time will not open at all.


Log ( /var/log/apache2/error.log ) shows single line:
[Wed Feb 03 12:44:00 2010] [error] server reached MaxClients setting, consider raising the MaxClients setting

Indeed, there were 150 (default maximum limit in Ubuntu) apache2 processes started, so no new connections can be made.

Site access-log shows frequent requests every few seconds from the single IP:

[03/Feb/2010:12:46:01 -0500] "GET /download/timeleft.exe HTTP/1.1" 206 ..
[03/Feb/2010:12:46:03 -0500] "GET /download/timeleft.exe HTTP/1.1" 206 ..
[03/Feb/2010:12:46:03 -0500] "GET /download/timeleft.exe HTTP/1.1" 206 ..
[03/Feb/2010:12:46:05 -0500] "GET /download/timeleft.exe HTTP/1.1" 206 ..
...



What happens
Most likely client was trying to download EXE file using some misconfigured downloading tool over the very slow network - it starts downloading, then abandons the thread and starts next.
On the server side - new process is started to serve client request, and pretty soon client request count exceeds maximum. Default client count is 150 and timeout is 300 seconds, so even 1 request at second will completely overload the server in less than 3 minutes.

This is absolutely the same tactics which is used in the "Slowloris" HTTP DoS script (I will not publish a link on it here) which is used to slow down Apache web server (IIS and ASP are not affected - which is quite a surprise, but it's only because of different design - non-threading). You can find more information about Slowloris in the Google. In short - this is very easy way to disable a webserver, and quite dangerous because it doesn't require powerful computer and fast connection (which are needed for the "ordinary" flood attack) - even started from inside virtual machine over quite average cable internet connection - it locks quad-core Ubuntu server with 4Gb or RAM!



Monitoring
Quite simple - monitor number of running apache2 processes and trigger some action (send email, etc if it reaches your apache's MaxClients value).
This command returns number of active connection (OK, number+1, grep is counted as well ;) )
ps aux grep apache2 wc -l



Prevention
1. Short-term prevention - restricting client using firewall. Example using iptables:
iptables -I INPUT 1 -s 55.55.55.55 -j DROP
Where 55.55.55.55 - is attacker IP address.
This adds first rule into the chain to drop all packets from the given IP. Just don't put your IP there - you won't like the result.

2. Tuning MaxClients value. In general, do not increase it to more than:
(Total Memory - OS memory) / Apache process size.
Check apache2 process memory with ps aux. See column RSS (size in kilobytes). So if you have 2Gb of memory and average apache2 process (without much modules, php, mysql, etc) takes 5Kb, then maximum client count should be ~ (2048-250)/5 = 360.
Increasing is more will cause going into the swap space, and performance will degrade dramatically.
However keep in mind that increasing MaxClients will only postpone, not fix the issue.

3. Restricting number of connects from one IP address using firewall - for example using recent module. I really could not make it working, and playing with the firewall on production system is not the best idea, so I skipped this part. (If you have the solution - please let me know)

4. Restricting number of connects from one IP address using mod_qos module for Apache.
Install prerequisites:
apt-get install apache2-threaded-dev gcc
Get mod_qos from sourceforge:
wget http://downloads.sourceforge.net/project/mod-qos/mod-qos/9.7/mod_qos-9.7.tar.gz?use_mirror=iweb
Unpack:
tar xvfz mod_qos-9.7.tar.gz
Compile and install:
cd mod_qos-9.7/apache2/
apxs2 -i -c mod_qos.c
When compiled - new library should be created in /usr/lib/apache2/modules/mod_qos.so
Make sure pemisions are rw-r-r: chmod 644 mod_qos.so

Now setup this module in Apache:
Create two files in /etc/apache2/mods-available/directory:
qos.load:
LoadModule qos_module /usr/lib/apache2/modules/mod_qos.so

qos.conf:
## QoS Settings
<ifmodule mod_qos.c>
# handles connections from up to 100000 different IPs
QS_ClientEntries 100000
# will allow only 50 connections per IP
QS_SrvMaxConnPerIP 50
# maximum number of active TCP connections is limited to 256
MaxClients 256
# disables keep-alive when 70% of the TCP connections are occupied:
QS_SrvMaxConnClose 180
# minimum request/response speed (deny slow clients blocking the server,
# ie. slowloris keeping connections open without requesting anything):
QS_SrvMinDataRate 150 1000
# and limit request header and body (carefull, that limits uploads and post requests too):
# LimitRequestFields 30
# QS_LimitRequestBody 102400
</ifmodule>
 
Enable module and restart apache:
a2enmod qos
/etc/init.d/apache2 restart

That's it!