Yesterday I've discovered some sort of attack on our web-site.
Some sort - because most likely it was caused by misconfigured client, but nevertheless it effectively put the web server down.
Effect
Ubuntu server 8.04 LTS, Apache 2 webserver.
Server works without any problems - you can ping, connect via ssh, FTP, receive or send email, etc, CPU load is minimal and absolutely not suspicious, but webpages are extremely slow and most of the time will not open at all.
Log ( /var/log/apache2/error.log ) shows single line:
[Wed Feb 03 12:44:00 2010] [error] server reached MaxClients setting, consider raising the MaxClients setting
Indeed, there were 150 (default maximum limit in Ubuntu) apache2 processes started, so no new connections can be made.
Site access-log shows frequent requests every few seconds from the single IP:
[03/Feb/2010:12:46:01 -0500] "GET /download/timeleft.exe HTTP/1.1" 206 ..
[03/Feb/2010:12:46:03 -0500] "GET /download/timeleft.exe HTTP/1.1" 206 ..
[03/Feb/2010:12:46:03 -0500] "GET /download/timeleft.exe HTTP/1.1" 206 ..
[03/Feb/2010:12:46:05 -0500] "GET /download/timeleft.exe HTTP/1.1" 206 ..
...
What happens
Most likely client was trying to download EXE file using some misconfigured downloading tool over the very slow network - it starts downloading, then abandons the thread and starts next.
On the server side - new process is started to serve client request, and pretty soon client request count exceeds maximum. Default client count is 150 and timeout is 300 seconds, so even 1 request at second will completely overload the server in less than 3 minutes.
This is absolutely the same tactics which is used in the "Slowloris" HTTP DoS script (I will not publish a link on it here) which is used to slow down Apache web server (IIS and ASP are not affected - which is quite a surprise, but it's only because of different design - non-threading). You can find more information about Slowloris in the Google. In short - this is very easy way to disable a webserver, and quite dangerous because it doesn't require powerful computer and fast connection (which are needed for the "ordinary" flood attack) - even started from inside virtual machine over quite average cable internet connection - it locks quad-core Ubuntu server with 4Gb or RAM!
Monitoring
Quite simple - monitor number of running apache2 processes and trigger some action (send email, etc if it reaches your apache's MaxClients value).
This command returns number of active connection (OK, number+1, grep is counted as well ;) )
ps aux grep apache2 wc -l
Prevention
1. Short-term prevention - restricting client using firewall. Example using iptables:
iptables -I INPUT 1 -s 55.55.55.55 -j DROP
Where 55.55.55.55 - is attacker IP address.
This adds first rule into the chain to drop all packets from the given IP. Just don't put your IP there - you won't like the result.
2. Tuning MaxClients value. In general, do not increase it to more than:
(Total Memory - OS memory) / Apache process size.
Check apache2 process memory with ps aux. See column RSS (size in kilobytes). So if you have 2Gb of memory and average apache2 process (without much modules, php, mysql, etc) takes 5Kb, then maximum client count should be ~ (2048-250)/5 = 360.
Increasing is more will cause going into the swap space, and performance will degrade dramatically.
However keep in mind that increasing MaxClients will only postpone, not fix the issue.
3. Restricting number of connects from one IP address using firewall - for example using recent module. I really could not make it working, and playing with the firewall on production system is not the best idea, so I skipped this part. (If you have the solution - please let me know)
4. Restricting number of connects from one IP address using mod_qos module for Apache.
Install prerequisites:
apt-get install apache2-threaded-dev gcc
Get mod_qos from sourceforge:
wget http://downloads.sourceforge.net/project/mod-qos/mod-qos/9.7/mod_qos-9.7.tar.gz?use_mirror=iweb
Unpack:
tar xvfz mod_qos-9.7.tar.gz
Compile and install:
cd mod_qos-9.7/apache2/
apxs2 -i -c mod_qos.c
When compiled - new library should be created in /usr/lib/apache2/modules/mod_qos.so
Make sure pemisions are rw-r-r: chmod 644 mod_qos.so
Now setup this module in Apache:
Create two files in /etc/apache2/mods-available/directory:
qos.load:
LoadModule qos_module /usr/lib/apache2/modules/mod_qos.so
qos.conf:
## QoS Settings
<ifmodule mod_qos.c>
# handles connections from up to 100000 different IPs
QS_ClientEntries 100000
# will allow only 50 connections per IP
QS_SrvMaxConnPerIP 50
# maximum number of active TCP connections is limited to 256
MaxClients 256
# disables keep-alive when 70% of the TCP connections are occupied:
QS_SrvMaxConnClose 180
# minimum request/response speed (deny slow clients blocking the server,
# ie. slowloris keeping connections open without requesting anything):
QS_SrvMinDataRate 150 1000
# and limit request header and body (carefull, that limits uploads and post requests too):
# LimitRequestFields 30
# QS_LimitRequestBody 102400
</ifmodule>
Enable module and restart apache:
a2enmod qos
/etc/init.d/apache2 restart
That's it!
Thursday, February 4, 2010
Tuesday, January 19, 2010
How to monitor employees work from home
To monitor your working at home employees, you can use third-party employee monitoring software. Such software automatically records what employees do on a computer during their working time.
The situation of monitoring employees' work from home might look a bit to difficult technically, but it all depends on how you organize it. For example: an employee works on its own home computer, how to track employee' work on it and nothing besides work?
How to organize monitoring:
1. Remote Desktop (Citrix, Terminal Server)
If your employees work at home, connecting to the company's working environment using Citrix/Terminal Server, you can simply install third-party time tracking software on the company's server side. And in this case the software will monitor employees under Citrix only (in company's environment only). So, under this environment employees have to work only and you, as an employer, have your right legally to monitor everything. And at the same time your employees are secured: if they want to access their web-banking from their home computers using their own Internet access - no problems, you will never even see it, as the time tracking software has no access to employees' home computers.
So, the solution in this case is:
- Installing employee monitoring software on the company's Citrix/ Terminal Server;
- Educating your employees, when working from home not to use Citrix environment for their personal needs.
How employee monitoring software works: it launches automatically in the background as an employee connects to the Citrix/Terminal Server environment, it works invisibly to the employee, automatically recording user name, applications, documents, Internet usage and times.
2. Company's laptop
If your employees work at home, using company's laptop, you also, as an employer, have your right to monitor your property usage. Just install third-party employee monitoring software on your laptops.
So, the solution in this case is:
- Installing employee monitoring software on company's laptops;
- Educating your employees when working from home not to use company's laptops for their personal needs.
How employee monitoring software works: it launches automatically and works invisibly to employees, automatically recording user name, applications, documents, monitoring Internet usage and times. When laptop is connected to the company's network, employee monitoring software automatically sends recorded data to the centralized location.
3. Employees' personal computer or laptop.
If your employees work at home using their own home computer or laptop (no Citrix or Terminal Sever), then you can request your employees to install third party time tracking software right on their home computer, laptop. To secure your employees privacy, you can educate employees to setup the software to track selected applications only or to start the software only when employees actually work for the company and stop it when doing other things.
So, the solution in this case is:
- Installing employee monitoring software on employees' personal home computers, laptops;
- Educating your employees as proposed above.
How employee monitoring software works: it automatically records user name, applications, documents, Internet usage and times. Then recorded data can be automatically sent to the centralized location using FTP connection or employees can simply generate reports and submit them to you over email.
NesterSoft Inc., Jan 19, 2010
The situation of monitoring employees' work from home might look a bit to difficult technically, but it all depends on how you organize it. For example: an employee works on its own home computer, how to track employee' work on it and nothing besides work?
How to organize monitoring:
1. Remote Desktop (Citrix, Terminal Server)
If your employees work at home, connecting to the company's working environment using Citrix/Terminal Server, you can simply install third-party time tracking software on the company's server side. And in this case the software will monitor employees under Citrix only (in company's environment only). So, under this environment employees have to work only and you, as an employer, have your right legally to monitor everything. And at the same time your employees are secured: if they want to access their web-banking from their home computers using their own Internet access - no problems, you will never even see it, as the time tracking software has no access to employees' home computers.
So, the solution in this case is:
- Installing employee monitoring software on the company's Citrix/ Terminal Server;
- Educating your employees, when working from home not to use Citrix environment for their personal needs.
How employee monitoring software works: it launches automatically in the background as an employee connects to the Citrix/Terminal Server environment, it works invisibly to the employee, automatically recording user name, applications, documents, Internet usage and times.
2. Company's laptop
If your employees work at home, using company's laptop, you also, as an employer, have your right to monitor your property usage. Just install third-party employee monitoring software on your laptops.
So, the solution in this case is:
- Installing employee monitoring software on company's laptops;
- Educating your employees when working from home not to use company's laptops for their personal needs.
How employee monitoring software works: it launches automatically and works invisibly to employees, automatically recording user name, applications, documents, monitoring Internet usage and times. When laptop is connected to the company's network, employee monitoring software automatically sends recorded data to the centralized location.
3. Employees' personal computer or laptop.
If your employees work at home using their own home computer or laptop (no Citrix or Terminal Sever), then you can request your employees to install third party time tracking software right on their home computer, laptop. To secure your employees privacy, you can educate employees to setup the software to track selected applications only or to start the software only when employees actually work for the company and stop it when doing other things.
So, the solution in this case is:
- Installing employee monitoring software on employees' personal home computers, laptops;
- Educating your employees as proposed above.
How employee monitoring software works: it automatically records user name, applications, documents, Internet usage and times. Then recorded data can be automatically sent to the centralized location using FTP connection or employees can simply generate reports and submit them to you over email.
NesterSoft Inc., Jan 19, 2010
Subscribe to:
Posts (Atom)