Barry's blog Ampersand was the target of a denial of service attack this morning, starting at about 6am (EST). I started working on the problem, narrowing down the universe of causation for disfunction, stopping and restarting the apache server, watching i/o, mem, proc, and of course hoping that the drones would detect a "win" and go elsewhere. I'm lazy. It began pre-coffee. Now I'm surrounded by 4 hyper kids on cartoons and stim sources...
The universe of POST method sources (attack node set a) is 121 nodes (log a). The universe of premature exit from POST method sources (attack node set b) is 193 nodes (log b). Significant overlap of the two sets is presumed, but not yet checked.
The top 10 POST method attackers were:
025 80.58.11.107.proxycache.rima-tde.net
028 200.31.79.214 (ns: admindns.diveo.net.co)
029 80.58.39.107.proxycache.rima-tde.net
030 c-24-21-136-225.client.comcast.net
032 200-168-19-55.dsl.telesp.net.br
046 80.58.3.172.proxycache.rima-tde.net
052 cache4.att.sch.gr
059 80.58.41.44.proxycache.rima-tde.net
082 c-67-162-221-234.client.comcast.net
338 wcs1-mcpherson.nipr.mil
The top 10 premature exit (POST) method attackers were:
046 194.63.235.166 cache4.att.sch.gr
048 163.28.48.69 proxy.tyrc.edu.tw
056 200.31.79.214 diveo.net
078 217.23.230.4 ethicsgradient.derwentside.net
084 203.195.201.29 203-195-201-29.now-india.net.in
111 212.0.138.14 ripe.net
207 66.221.193.1 propagation.net
337 198.26.120.12 WCS1-MCPHERSON.NIPR.MIL
544 24.21.136.225 c-24-21-136-225.client.comcast.net
692 66.144.4.2 State.OH.US
The 100+ fleet of attack machines are able to exploit the design weakness of the blog meme, a store managed by a writer process (perl in this case, but that is an implementation issue), and accessed by a POST method. There are "fixes", but they seem to fall into the problem space defined by abusive data mining (by spammers, who else) of whois:43 services -- throttle the reader (here writer) or place an ACL scheme in place (whitelists or authentication tokens) or play whack-a-mole (blacklists) or heuristics on the temporal properties (too many per unit of time) or content (viagra et seq) properties.
The blogware vendor approach (protect your blog) isn't useful in a production vhost cluster, so I'll have to give this some thought.
Incidently, the box owned by the state of Ohio inserted 1196 spam ads for drugs at wampum on 24 November. Email to the domain contact has gone unanswered. I noticed without taking note that in the past few days a blog went commentless after a 1k drug ad insert campaign. I recall it was running mt3.121, which I'm using on a test blog, and the blogger mentioned that bl didn't work (for him, at least not right away).
I'm not expecting any checks from a state IT mismanagement office, or the brass at nipr.mil. Pity. Spammers should pay for their litter.
Update: I posted a short note to an operator's list. This morning someone wrote to me privately that she'd observed a universe of POST method 150 sources attack a mt blog with the load average consequence I observed.
Posted by EBW at December 4, 2004 10:57 AM | TrackBackHi
I'm interested in this denial of service but I'm non-technical designer and can't get much sense from your exclamation.
My site suddenly experienced an incre3ase in bandwidth over three weeks, rising towards the end 1Gb a day over the limit, when it got to 16 Gb over the half gig allotted, the ISP pulled it, but isn't as dedicated as you so just brushed it off as me uploading large files, I wasn't.
Is this a denial of service? I guess you only have to piss one person off.
I should add my blog isn't a 'normal' blog, not done with blog software, but just an html pages[s] which is now php after a redesign.
One more thing, at the same time it became unavailable to a friend in Florida with Bellsouth. Still is. But loads from New York and as far as I know anywhere else.
Howdy Petet,
I don't know how M$ server-side logs xfrs, but in the real world figuring out the directionality of the xfr initiation on current log data is about a 1min task for a 1st year admin.
The class of attack you experienced wasn't necessarily distributed, though it could have been implemented that way. You have a finite resource (your isp's b/w allocation for the Service Level Agreement product you bought, and more generally, their b/w allocations by their up-streams, and some other resources operators really care about) and it has been exhausted other than how you anticipated it would be. Could be bad designs based on a poor understanding of the technology like the guy who appears to think I invented both dhcp (I didn't, Ralph Droms did, from the SunOS bootp probem space, which I certainly worked on) and the http state management mechanism (I didn't, Dave Kristol and Lou Montulli did, about two years before I started working on putting additional semantics on cookies).
Yup. I'm afraid all it takes is one person, but if you use a M$ product (no process isolation in memory in the operating system) you are part of the problem too.
The inconsistency of view (SBC vs j-random-NYNY-isp) could be several things. Is one view seeing a cache? Are the dns resolutions identical? Is one view encountering an administrative (intentional) filter?
Posted by: Eric at December 5, 2004 05:51 PM