Ubuntu - protecting against "slow" DoS attack on apache2 web-server (defend from Slowloris)

Yesterday I've discovered some sort of attack on our web-site.
Some sort - because most likely it was caused by misconfigured client, but nevertheless it effectively put the web server down.



Effect
Ubuntu server 8.04 LTS, Apache 2 webserver.
Server works without any problems - you can ping, connect via ssh, FTP, receive or send email, etc, CPU load is minimal and absolutely not suspicious, but webpages are extremely slow and most of the time will not open at all.


Log ( /var/log/apache2/error.log ) shows single line:
[Wed Feb 03 12:44:00 2010] [error] server reached MaxClients setting, consider raising the MaxClients setting

Indeed, there were 150 (default maximum limit in Ubuntu) apache2 processes started, so no new connections can be made.

Site access-log shows frequent requests every few seconds from the single IP:

[03/Feb/2010:12:46:01 -0500] "GET /download/timeleft.exe HTTP/1.1" 206 ..
[03/Feb/2010:12:46:03 -0500] "GET /download/timeleft.exe HTTP/1.1" 206 ..
[03/Feb/2010:12:46:03 -0500] "GET /download/timeleft.exe HTTP/1.1" 206 ..
[03/Feb/2010:12:46:05 -0500] "GET /download/timeleft.exe HTTP/1.1" 206 ..
...



What happens
Most likely client was trying to download EXE file using some misconfigured downloading tool over the very slow network - it starts downloading, then abandons the thread and starts next.
On the server side - new process is started to serve client request, and pretty soon client request count exceeds maximum. Default client count is 150 and timeout is 300 seconds, so even 1 request at second will completely overload the server in less than 3 minutes.

This is absolutely the same tactics which is used in the "Slowloris" HTTP DoS script (I will not publish a link on it here) which is used to slow down Apache web server (IIS and ASP are not affected - which is quite a surprise, but it's only because of different design - non-threading). You can find more information about Slowloris in the Google. In short - this is very easy way to disable a webserver, and quite dangerous because it doesn't require powerful computer and fast connection (which are needed for the "ordinary" flood attack) - even started from inside virtual machine over quite average cable internet connection - it locks quad-core Ubuntu server with 4Gb or RAM!



Monitoring
Quite simple - monitor number of running apache2 processes and trigger some action (send email, etc if it reaches your apache's MaxClients value).
This command returns number of active connection (OK, number+1, grep is counted as well ;) )
ps aux grep apache2 wc -l



Prevention
1. Short-term prevention - restricting client using firewall. Example using iptables:
iptables -I INPUT 1 -s 55.55.55.55 -j DROP
Where 55.55.55.55 - is attacker IP address.
This adds first rule into the chain to drop all packets from the given IP. Just don't put your IP there - you won't like the result.

2. Tuning MaxClients value. In general, do not increase it to more than:
(Total Memory - OS memory) / Apache process size.
Check apache2 process memory with ps aux. See column RSS (size in kilobytes). So if you have 2Gb of memory and average apache2 process (without much modules, php, mysql, etc) takes 5Kb, then maximum client count should be ~ (2048-250)/5 = 360.
Increasing is more will cause going into the swap space, and performance will degrade dramatically.
However keep in mind that increasing MaxClients will only postpone, not fix the issue.

3. Restricting number of connects from one IP address using firewall - for example using recent module. I really could not make it working, and playing with the firewall on production system is not the best idea, so I skipped this part. (If you have the solution - please let me know)

4. Restricting number of connects from one IP address using mod_qos module for Apache.
Install prerequisites:
apt-get install apache2-threaded-dev gcc
Get mod_qos from sourceforge:
wget http://downloads.sourceforge.net/project/mod-qos/mod-qos/9.7/mod_qos-9.7.tar.gz?use_mirror=iweb
Unpack:
tar xvfz mod_qos-9.7.tar.gz
Compile and install:
cd mod_qos-9.7/apache2/
apxs2 -i -c mod_qos.c
When compiled - new library should be created in /usr/lib/apache2/modules/mod_qos.so
Make sure pemisions are rw-r-r: chmod 644 mod_qos.so

Now setup this module in Apache:
Create two files in /etc/apache2/mods-available/directory:
qos.load:
LoadModule qos_module /usr/lib/apache2/modules/mod_qos.so

qos.conf:
## QoS Settings
<ifmodule mod_qos.c>
# handles connections from up to 100000 different IPs
QS_ClientEntries 100000
# will allow only 50 connections per IP
QS_SrvMaxConnPerIP 50
# maximum number of active TCP connections is limited to 256
MaxClients 256
# disables keep-alive when 70% of the TCP connections are occupied:
QS_SrvMaxConnClose 180
# minimum request/response speed (deny slow clients blocking the server,
# ie. slowloris keeping connections open without requesting anything):
QS_SrvMinDataRate 150 1000
# and limit request header and body (carefull, that limits uploads and post requests too):
# LimitRequestFields 30
# QS_LimitRequestBody 102400
</ifmodule>
 
Enable module and restart apache:
a2enmod qos
/etc/init.d/apache2 restart

That's it!

How to monitor employees work from home

To monitor your working at home employees, you can use third-party employee monitoring software. Such software automatically records what employees do on a computer during their working time.

The situation of monitoring employees' work from home might look a bit to difficult technically, but it all depends on how you organize it. For example: an employee works on its own home computer, how to track employee' work on it and nothing besides work?

How to organize monitoring:
1. Remote Desktop (Citrix, Terminal Server)
If your employees work at home, connecting to the company's working environment using Citrix/Terminal Server, you can simply install third-party time tracking software on the company's server side. And in this case the software will monitor employees under Citrix only (in company's environment only). So, under this environment employees have to work only and you, as an employer, have your right legally to monitor everything. And at the same time your employees are secured: if they want to access their web-banking from their home computers using their own Internet access - no problems, you will never even see it, as the time tracking software has no access to employees' home computers.

So, the solution in this case is:
- Installing employee monitoring software on the company's Citrix/ Terminal Server;
- Educating your employees, when working from home not to use Citrix environment for their personal needs.

How employee monitoring software works: it launches automatically in the background as an employee connects to the Citrix/Terminal Server environment, it works invisibly to the employee, automatically recording user name, applications, documents, Internet usage and times.

2. Company's laptop
If your employees work at home, using company's laptop, you also, as an employer, have your right to monitor your property usage. Just install third-party employee monitoring software on your laptops.

So, the solution in this case is:
- Installing employee monitoring software on company's laptops;
- Educating your employees when working from home not to use company's laptops for their personal needs.

How employee monitoring software works: it launches automatically and works invisibly to employees, automatically recording user name, applications, documents, monitoring Internet usage and times. When laptop is connected to the company's network, employee monitoring software automatically sends recorded data to the centralized location.


3. Employees' personal computer or laptop.
If your employees work at home using their own home computer or laptop (no Citrix or Terminal Sever), then you can request your employees to install third party time tracking software right on their home computer, laptop. To secure your employees privacy, you can educate employees to setup the software to track selected applications only or to start the software only when employees actually work for the company and stop it when doing other things.

So, the solution in this case is:
- Installing employee monitoring software on employees' personal home computers, laptops;
- Educating your employees as proposed above.

How employee monitoring software works: it automatically records user name, applications, documents, Internet usage and times. Then recorded data can be automatically sent to the centralized location using FTP connection or employees can simply generate reports and submit them to you over email.

NesterSoft Inc., Jan 19, 2010

Is it possible to monitor employee’s laptop while working from home?

I work on company's laptop. Can my company track time on it?

The situation
You work on a company's laptop. You have this laptop with you at home and in your business trips. And of course, you do not always work on it. When you do not work, you might want to use it for your personal needs, like browsing the Internet, accessing your online banking or your personal emails, chatting, playing games. This is all OK if you are done with your work duties for today. But there is a big concern that you have.

Your concerns
So, the concern is: you use company's laptop for your personal needs; can your company track time, Internet usage, software usage, basically anything on its laptop?

The answer
Yes, your company can track time, Internet usage, software and documents usage on its own laptop – see monitor laptop employees

At first, your company is legally allowed to do it.

Your company can technically track its laptop using third-party software in the following cases:

1. If at the moment when you receive your laptop there is software installed already. There already can be some time tracking software installed.
2. If you ever physically connect your laptop to the company's local network. When you login to your company's server, some software can be installed automatically; many companies have this process centralized. You might not even notice as installation can be "quiet".
3. If you ever connect to the company's network using VPN. This is just another way to connect to the company's local network. So, see the item 2.

Using time tracking software your company can record the following information:

1. Computer monitoring – Computer related activities – login, logout time, usage interval;
   
2. Software usage monitoring - software usage, documents accessed;
   
3. Internet monitoring – web-sites visited.
   
4. Time spent on each of these tasks.
   
5. User login names.


6. Screens, keystrokes, emails or chats content.

Your company cannot technically track its laptop using third-party software in the following case:
If you install only Microsoft software, if you do not install any company's or third-party software, if you connect to your company's work environment using Terminal Server, Citrix server, Remote Desktop or VNC connection – then it will be not possible to monitor your laptop using some third-party software.

Your company can monitor Internet usage with some hardware, if you use company's Internet access. Just in this case it is not possible to see time spent on each web-page.

NesterSoft Inc., Nov 06, 2009

Toronto Swine Flu (H1N1) hysteria

Outside Etobicoke City Hall, Sat Oct 31, 2009, 11:20am

Public is really risking to catch cold (Weather: Overcast, +12C, Wind 41km/h, Wind gusts: 65km/h, officially issued Wind warning for the City of Toronto) in a several-hours wait hoping to get a Swine Flu vaccination.

Following broad media coverage of the alleged “H1N1 pandemia” people line up for the vaccination. Crowd looks upset and scared, general atmosphere is near-panic. There are more than a dozen security guards, 2 police cruisers, and two police officers are walking around. CTV news is conducting interviews.

Some statistics: Previous SARS epidemic was a “probable case“ of 44 deaths in 2002 - 2003. At the same time traffic accidents in 2003 caused 2,778 deaths.

NesterSoft WorkTime coupon

There is a 20% off coupon for WorkTime: D109

With this discount get WorkTime Professional for $39.96, WorkTime Home for $23.96, and it applies to multiple licenses purchase as well.
Expires: Oct 31, 2009
Direct link: http://www.nestersoft.com/worktime/register.shtml?coupon=D109

WorkTime – time tracking software tracks computer-related activities: projects and tasks, applications, documents, web-sites and games.

Monitor internet usage

Monitor internet usage - protect your company, your money and build a good working environment.

Internet usage monitoring - how can it help you.

Most of employees know how difficult it is to find a good job. But what employers know - this is even more difficult to find a good employee.
What is a good job? It is simple - good salary, stable company, nice and friendly environment, interesting tasks.
And who is a good employee? Let's try to define - a good employee is professional, cooperative and enthusiastic, it is someone who you can rely to, who understands everything from one word, who can discuss problems and propose solutions, who can recognize its own mistakes, oh and also has a good common sense and intuition! I hope I did not forget anything important :-) Of course this is not a complete definition, I am sure employers might expect more.
And yes, it is not easy to find such an employee. Many people look very enthusiastic on a job interview, but after they simply lose their interest to the job they where looking for so hard. And everything looks good - the workplace, the salary. The most common issue - employees became not cooperative with time - they don't really work fast, the don't really want to understand tasks, they simply lose interest to the working process. Why? Because everything is good! Every next day looks like the previous one. And besides, now there is an Internet access available. And in Internet - there is so much interesting you can find. Some people can spend the whole days browsing. What is Internet for them? Communication, interesting information, games! Internet is a big issue now. Before, for example, when there was no TV, people would read more books. Same with Internet: if there was no Internet access available, employees would spend more time working.
And now you as an employer, you have the task to solve. You see the problem - there are employees who spend too much time in the Internet or doing other things not related to their work. You start thinking if you should monitor internet usage in your company. But you don't want your company to turn into a strict military organization. Why? Because, for example, enthusiastic employees will lose their enthusiasm and will simply run away from you. Because, in general, you understand that there are good professionals in your company, there are less professional but cooperative employees and there are employees who don't really work well - not everyone is so bad, right? Besides, you understand that people need their time to relax and if they want to browse Internet a bit - it is OK. Also you don't want to break anyone's privacy. But still - you need to protect your company and your money somehow.
So, what all the above means? It means that you still need to monitor internet usage, but you need such a solution that shows you the main negative trends in the company. Like, for example, you might want to find out, who in your company spends too much time browsing the Internet or who spends more than 2 hours in the Internet. Or you might want to define, what are the most popular Internet resources not related to the working process (some employees still might need Internet to solve work related tasks, right?). You don't need to monitor every keystroke or screens. All you need - is to be able to find out trends that affect productivity in the company. Then you might want to generate reports and to send them out to all departments. This will be a good stimulus for all your employees, who think that all days are the same and boring.

WorkTime - monitor internet usage in your company and create a better working environment!
NesterSoft Inc., Sep 25, 2009

time tracking software - higher productivity

How can time tracking software help you to keep higher productivity

Either you need some time tracking software for yourself or for employee monitoring - WorkTime can suit all your needs in time tracking.

If you are looking for a productivity monitoring software, with WorkTime you can easily solve many important tasks, like, for example:
- Monitoring computer: with WorkTime you can setup a constant invisible automatic computer monitoring that will help you in tracking employee productivity level.
- Monitoring Internet usage: as for today this is one of the biggest issues, especially in large companies, where a lot of time is simply wasted on Internet related activities. With WorkTime you can monitor Internet usage automatically. WorkTime reports will show you how much time is actually spent working or browsing the Internet. Also, for example, you can find out, what are the most popular Internet resources employees use or who uses Internet the most.
- Employee time tracking: with WorkTime you can automate employee time tracking process. WorkTime will record automatically for you all computer related activities by the employees: applications, documents used, WEB sites visited etc.
- Employee time clock software: with WorkTime you will always know when your employees start and finish working.

Time tracking software market proposes you a lot of time tracking applications, but not very often you can find an application that covers all your needs in time tracking. As per our customers (we get such feedbacks very often), WorkTime is one of the most intuitive and powerful monitoring software and they were looking for something like this for years!

NesterSoft
WorkTime® - Time Tracking Software
http://www.nestersoft.com/worktime