.
![[ Powered by Fedora Core ]](/icons/powered_by_fedora.png){kind=icon}
A few days ago, at 3:30 PM on Feb, 10, 1998, I noticed that my main machine at wsrcc.com was wedged. This machine is a multi-homed ppro-150 that is on both Best.com's network via a frame-56k link and Athome's network via a 10Mbit/sec cable modem. The machine is currently running Netbsd-1.3, and has been pretty stable. On the other hand Fremont California was having wind, rain, mud, and power problems at the time. I shrugged and figured it might have been a momentary power outage too short to cause the machine to reboot, but long enough to cause havoc. After trying to reboot the machine gracefully via network logins, terminal logins, the X console, alternate virtual terminals I decided it was really dead and used the ultimate reboot switch, the power switch.
The machine rebooted, did what was its usual infinite fsck as it checked all the dirty disks and tantalized me with the inode numbers of files that got lost in the shuffle. All was well enough for a few minutes and then tcp_wrappers started kicking out a few errors about reverse DNS lookups failing. A machine without a valid name was attempting to ftp here. This happens every once in a while. Usually the person gets the hint the second or third time his ftp prints out:
421-DNS config error for HOSTNAME [IPADDRESS].
421-Please have your system admin check the forward and reverse DNS
421 mappings for your machine.
Every once in a while I get a really dim bulb and he tries it for a few dozen times. This appeared to be one of these times.
Then I noticed that the IP address of the unnamed machine was changing. Hmm. Curious. I slapped a tcpdump on that interface and watched. To my surprise I was getting tons of tcp syn's to the ftp port. Even more curious they were all different. At this time the load average was climbing dangerously, so I turned off anon-ftp from the password file by changing user "ftp" to user "noftp". It wasn't long before inetd decided that ftp was looping, since there were so many ftp's coming and going. I removed all mention of ftp from inetd.conf and rebooted. At this point inetd was sending out lots of rejects. I didn't want some poor unwitting system bombarded with rejects for the spoofed packets so I slapped a "ipfilter" filter on that port blocking any ftp TCP packet. I just continued to record all the packets and scratched my head. It certainly looked like a TCP SYN denial-of-service attack. (I was seeing what averaged out to 33,000 syns in 6 hours.) I fired off a message to my ISP to see if they could trace back this flood.
From wolfgang Tue Feb 10 16:13:32 -0800 1998
From: Wolfgang S. Rupprecht <wolfgang.rupprecht+web@gmail.com>
To: XXX@corp.home.net (XXX XXX)
Subject: I'm under attack from spoofed IP packets.
X-Mailer: VM 6.35 under Emacs 19.34.2
Organization: W S Rupprecht Computer Consulting, Fremont CA
Mime-Version: 1.0 (generated by tm-edit 7.106)
Content-Type: text/plain; charset=US-ASCII
XXX, my machine is currently under attack by somebody that either
is able to co-ordinate a large number of hosts, or someone that is
using spoofed src addresses. Could you do me a big favor and see
where the packets are coming from? The attack is on my ftp server's
port.
Name: cXXXX-a.frmt1.sfba.home.com
Address: 24.1.XX.XX
Tel: (510) XXX-XXXX
-wolfgang
--
Wolfgang S. Rupprecht <wolfgang.rupprecht+web@gmail.com> http://www.wsrcc.com/wolfgang/
Never trust a program you don't have sources for.
They were seeing it, but were stumped. This was a new type of attack. Tracing it back through the routers was very slow going.
I spend a few hours assuming it was just a normal SYN denial of service attack. Then curiosity got the better of me. Let's see if the hosts all exist. Yup almost all of them are in DNS. Are they all routable? Yes. Someone is doing a bit homework. I wonder if there is anything else I know about the perpetrator. I wonder how many hops he is away from here. Let's see what is the TTL? Since most OS's pick one of a small number of starting TTL's, I can probably make some good guesses as to how many hops the guy is from here. I restart the tcpdump in a more verbose mode. Well, look at that, the TTL is randomized. Ok, they thought of that one. I wonder, what the TTL really is to one of these hosts picked at random. Sheesh, look at that. The return TCP packet has exactly the same TTL as the spoofed packet. The spoofed packet is spot on. Let's try another. Ditto. Holy crow. How did they do that? They either have access to a core router between me and the spoofed hosts or they have a very clever way of working out the TTL.
At this point I began to notice a very curious phenomenon. Remember I was now blocking ftp/tcp with a silent blackhole filter. If I looked back in my logs I'd see the exact same tcp initial sequence number, at roughly the tcp retransmission intervals, coming from the same hosts. Very weird. Could they really be retransmitting? How would the spoofer know if I was replying or not? At this point I was starting to really fear some large distributed hack. I was starting to curse the kind folks that put out java and other trojan-horse delivery systems. I was wondering could some java trojanlet allow a focused tcp attack from thousands of hosts? I assumed probably. I wonder, is this really from a large number of machines? What other tests can I perform? Scratch head. Think. I don't exactly have much feedback to them. How can I see if it is one host controlling a large network of unwitting dupes?
A thought hit me. Since they are hitting on the ftp port, perhaps they are looking up ftp.wsrcc.com . I wonder what would happen if I changed the dns zone file to point to a different machine on my net. Now my DNS zone files have a cache time of 28 days. A host that looks up the name to ip address mapping is very unlikely to change its notion of the IP address any time soon. I change the zone file to point ftp.wsrcc.com to a powered down host. No sooner had I hit carriage return to load the zone file, I saw the first arp request for the powered down machine. Mind you, the zone file hadn't even loaded into the DNS secondaries yet! We are talking about a few second delay here. It became quite clear now that this was the result of a massive swarm of independent hosts. After all, how else would "fresh" hosts that hadn't looked up the IP address of ftp.wsrcc.com be waiting in the wings, ready to grab the new address within seconds of it being loaded?
Its now getting on towards 10:30 pm, I have to roll over my tcpdump log file since its getting to be several megs. I'm getting somewhat punchy after 7 hours of thinking about this crap. I wonder what else I can do anyway. Why me? Could some spammer be mad at me? I don't go out of my way to be unpleasant to them. Oh well, whatever they are doing is probably too much for me to unravel at this point. I'm thinking that some of the real security places would know what to do. But I'm small fries. I'd have to do the same thing the dishonest companies do and claim to have lost "a million dollars". I'd lost an evening and a about a hundred inodes in the crash. Unless I claimed that the inodes and my time to recover them were worth 10 grand a piece, I'd just get some desk jock to fill out a form and catalog the incident. BFD.
Time to read a bit of netnews and turn in.
The next day I notice that I'm at 98% on my /u partition. Curious. I run a du on some suspects. Lo and behold my ftp incoming has developed a few subdirectories with the curious names of:
" "
" "
Inside were subdirectories of zip-ed files (the PC kind of zip, not gzip). The readme-like *.nfo files claimed to be for "cracked" pc games. I assume they are talking about games that used to have copy protections but have been altered to allow simple copying. These folks had no doubt noticed that one of the interfaces to my machine is on a nice 10Mbit/sec connection. The problem was that the machine normally receives one or two ftp's per *week*. The demand for 10-year old emacs *.el hacks just isn't what it used to be. These cracked games on the other hand, probably put ftp.wsrcc.com on the top 10 most wanted ftp sites. For the milliseconds before the machine crashed that is.
So how did this happen? Two or three weeks before this I needed to allow someone to dump a bunch of tar files off here. I changed the mode of the ~ftp/incoming from mode 0 to write-only. It didn't take long for the folks to spot a writable directory. Back when I was running wu-ftp I buggered it all up to disallow anything other than reading or writing of simple files for the anonymous ftp. A few years ago I started using krb5 and installed it's ftpd. The problem is that krb5's ftpd is pretty broken. 1) it doesn't syslog the anon user's commands. 2) it allows anon-ftp to "mkdir" "delete" "rmdir" etc. The directories made via mkdir are made with normal permissions, and one can cd into them and use them as drop boxes.
So if you are running krb5 ftp and still haven't turned off your incoming directory, run, don't walk to the machine and do so now.
-wolfgang
Wolfgang S. Rupprecht 2/24/98
.
wolfgang.rupprecht+web@gmail.com
(Wolfgang S. Rupprecht)
WSRCC Home Page || Up One
Level
..