This system has acted slow the last few days due to DDOS attacks at our ISP

From Friday to Monday (7/8 to 7/11) this system seemed to be acting up.

Networking was very slow. Pings were returning times up to 5000ms (you have to use the -w arg to ping if you want to see the slow responses) — well the ones that got thru, as at times there was 90% packet loss. Strangely for several of us, SSH was popping up a message about the systems host key having changed, even though it hadn’t, nor had the sshd software been updated.

Briefly, our ISP, ServerBeach, said that other hosts in their network were infected with evil hacker code, and some of the hosts were even trying to steal IP addresses, and because they were launching DDOS attacks, the switch got hosed, and at times it may have misdirected packets for our host to one of the evil hosts.

I’m pretty sure we were not broken into – I ran chkrootkit and rkhunter to check for signs of a breakin, and chkrootkit was happy except for 1 false positive due to a bug in it (invoked PS with an invalid arg), and rkhunter was happy.

I used this opportunity to update some software on the box — I brought openssh, openssl, apache, and php up to date. I’ve also installed a bit more system monitoring software, disabled some obsolete accounts, diabled some servers that were not needed, and am looking into chroot and other such things.

One security expert told me the following, in response to ssh complaining about the host key changing and ssh warning of a possible man in the middle attack:

sshv2 man in the middle attacks are difficult, however if you are on
the same switch as and attacker it could be done see: ettercap

But, on the other hand, someone else knowledgable told me:

interesting.. although i disagree with the comment of it not being possible to sniff traffic on a switched network.. if that was what he was saying.. it’s pretty easy

me:: but I thought you get no traffic except for yourself?

right, but say you and i are on a switched network, i can find your mac by pinging you, then with your mac i can man in the middle all traffic sent to you by assuming your mac and forwarding traffic on to you. ettercap is a good example of this, i’ve used it before and it does in fact work on a switched network

if the switches are super intelligent, then i imagine it should detect the mac flopping..

A kind of summary is:

  • We were probably not broken into
  • The DDOS and network slowdown at the ISP is resolved
  • I’m not sure why we got that ssh warning – I even got it again today from a system I haven’t used since before this happened. I think if we start using ssh public keys then things will be safer – I’ll notify the users soon on how to do this.

I received the following info from the ISP in response to support mails. These are in reverse cronological order with only the highlights shown:

The DoS attack that took place this weekend was substantiated by a
flood condition that occurred due to a sustained attack coupled with the loss of the mac-address of the target server. This attack was intermitted, so finding the culprit was not easy. Once the offending server was discovered, the IP was quickly null-routed to prevent further abuse. Thanks for your patience in this matter. If you still have any problems or questions please let us know.

As to your being affected, as long as your keys went back to the old
key you should be in good shape. Checking the MAC address on your server when I ssh in it matches the one in our records so at this time the correct server is answering your IP.

We KNOW we have the IP issue under control, that box has been reformated and is in burnin at this time, so its not going back online any time soon. Unless there is a second box stealing your IP you are in the clear on that one at this time.

What we are watching is for the DDOS attack. In this attack it has faded away to nothing for many hours at a time and then come right back, we are watching to make sure it is NOT coming back this time.

This question I can give you a difinitive answer on, You are on a switched network, so they can not sniff your traffic.

Normaly those who snag IP usualy ping around to find unused ones in our class C licenses, and use them. Taking IPs that are in use usualy does not do anything but waste their time. But between the DDOS and the person who stole the IPs sending data, while yours was waiting for a request, the switch got confused and decided the correct owner of that IP was the wrong box. By shutting him down the switch reverted back to the way it is supposed to be.

Unfortunatly there is not a lot we can do about that behavior in switches, that is what they are designed to do

While investigating the DDOS attack we discovered one server that had hijacked a number of other servers IP addresses. I do not yet have a list of all the IP’s this ex-customer had taken, but he has been removed from the network.

It is my understanding that the AUP people will notify those who’s IP’s were spoofed once we finish figuring all that has gone on here in the last several days.

At this time your server should have the old keys active, if not please let us know and we will see what we can find.

Unfortunatly in a self managed hosting enviroment it is possible for people to add IP’s to their server that they are not supposed to be using. When we catch them they are removed from the network.

The VLAN that your server is on has been subjected to an intermittent DDOS attack. At this time we believe we have identified the targets of these attacks and have removed them from our network to protect the other servers on the network. At this time three servers have been removed from the network due to this attack. Our AUP department is working with those customers to see what can be done to get them backup without affecting the rest of the network again.

Due to the nature of a DDOS attack the only way to protect the network is to identify and block the target server’s IP address. The normal firewall on your server will not alleviate the effects of a swamped network due to a DDOS attack.

We are continuing to monitor the situation, and will update you as the situation changes. This ticket will remain open as long as we feel there is a chance of the issue coming back.

The problem with the network is a DDOS attack. Every time we get it it blocked it starts comming in from a new source. The attack is not aimed at your server.

2 Responses to This system has acted slow the last few days due to DDOS attacks at our ISP »»


Trackbacks & Pingbacks »»

  1. [...] DDOS that was affecting us resolved, see: This system has acted slow the last few days due to DDOS attacks at our ISP [...]

  2. Pingback by StockMorph » Blog Archive » Site slowness fixed | 2005/07/12 at 14:13:48

    [...] DDOS that was affecting us resolved, see: site slowness fixed. [...]

Leave a Reply »»