Linux Magazine | July 1999 | BEST DEFENSE | Network-Based Process Table Attacks

The attack reached its peak at approximately 4:30pm. I was sitting in my office at Vineyard. NET, an Internet service provider on Martha's Vineyard, typing at a shell window connected to my ISP's primary Web and mail server. Suddenly, the computer printed something on my screen that was tremendously disturbing. I had asked the computer to list the files in the current directory. The computer had told me that it was unable to do so:

% ls
No more processes
%

This error message is printed by the C-shell. It means that the UNIX operating system has so many processes running that it cannot create a new process for the ls command to execute. This is profoundly bad: practically everything on UNIX -- from starting up new login sessions, to receiving mail, to the hourly maintenance programs -- requires that a new process be created. I hadn't seen this particular error message in nearly ten years. Back in 1988, when I started writing my book Practical UNIX Security (co-authored with Purdue University professor Gene Spafford), I mounted a bunch of attacks against my UNIX system. We called one a process overload attack -- a simple programmed attack in which one program repeatedly spawns off new processes. The program that executes the attack is just five lines long:

main()
{
 while(1)
   fork();
}

Process table attacks can be deadly to a UNIX system. The attack saps the computer's CPU power by constantly creating more processes, each of which is, itself, trying to create more processes. At some point, all of the slots in the process table are filled up, and no more processes can be created. Process table attacks can be very difficult to fight while they are in progress. That's because in order to kill a process, you need to know its process ID, and the only way to find a process ID is by using the ps command -- which itself requires the ability to spawn off yet another process. Instead, the standard procedure is to reboot the computer and find the attacker.

But the attack on Vineyard.NET didn't look like the attacks that I had launched against myself back in 1988. For starters, the main Web server had plenty of CPU cycles to spare. It just didn't have any processes left. Something else was going on.

I typed ls a few more times; one out of four times the command actually executed properly. New processes are created and destroyed all the time on a UNIX system. What was happening, I realized, was that every few times I typed the ls command I was fortunate enough to have my ls process start up before the attacker grabbed the slot that was just freed.

I decided against rebooting. Though that would solve the problem in the short run, it wouldn't tell me the source of the attack, or how it was being mounted. No, I needed to figure out where the attacker was coming from, how the attack was being launched. Only after learning this could I then patch the hole.

I used the ps command to see what processes the computer was actually running. The processes created by the attack were obvious: our Web server was running hundreds of copies of our finger daemon. And the timing was interesting too. Each process had been started exactly 10 minutes after the previous one. Closer examination of our finger daemon revealed that the program had no time-out -- the attacker was simply opening up connections to port 79 of our Web server and then letting those connections sit. Each connection created a new process. Indeed, the attack had been going on all day; we had just noticed it when our process table had been exhausted.

Next I used the netstat command to see where the attack was coming from. All of the TCP/IP connections to port 79 were coming from the same IP address. As it turned out, the IP address was from one of our own users. We checked with our terminal server and found the name of the account that was being used. For the purpose of this article, we'll call the account "Fred".

I called my partner Eric Bates, the co-owner of Vineyard.NET, into my office. Eric was immediately suspicious. Eric knew Fred -- Fred was a novice user who had no interest in attacks. Perhaps Fred's account had been stolen by someone else. Or perhaps it was Fred's teenage son. So we tried calling Fred, but the phone was busy. (Fred, like most people on Martha's Vineyard, has just one phone line.) With the problem diagnosed, we decided to remediate: we killed all of the finger processes and disabled the finger daemon until we could find the source of the problem. Finally, we killed Fred's dialup connection and telephoned his home.

Suffice it to say, Fred was confused when we called him. He didn't realize that his connection to the Internet had been blown away. He didn't understand how we could be calling him while he was online. No, he said, his son hadn't been using the Internet. Instead, he had been trying to use some Web site's chat service -- and the chat service had been giving him problems.

We never were able to figure out how Fred had managed to attack our system and bring it to its virtual knees. Perhaps a piece of buggy chat-system shareware was opening up connections. Or perhaps something about Fred's computer was just batty. In any event, we couldn't ask Fred not to go to that site anymore. We had to fix the problem ourselves.