As many of you know, Revision3’s servers were brought down over the Memorial Day weekend by a denial of service attack. It’s an all too common occurrence these days. But this one wasn’t your normal cybercrime – there’s a chilling twist at the end. Here’s what happened, and why we’re even more concerned today, after it’s over, than we were on Saturday when it started.
It all started with just a simple “hi”. Now “hi” can be the sweetest word in the world, breathlessly whispered into your ear by a long-lost lover, or squealed out by your bouncy toddler at the end of the day. But taken to excess – like by a cranky 3-year old–it gets downright annoying. Now imagine a room full of hyperactive toddlers, hot off of a three hour Juicy-Juice bender, incessantly shrieking “hi” over and over again, and you begin to understand what our poor servers went through this past weekend.
On the internet, computers say hi with a special type of packet, called “SYN”. A conversation between devices typically requires just one short SYN packet exchange, before moving on to larger messages containing real data. And most of the traffic cops on the internet – routers, firewalls and load balancers – are designed to mostly handle those larger messages. So a flood of SYN packets, just like a room full of hyperactive screaming toddlers, can cause all sorts of problems.
For adults, it’s typically an inability to cope, followed either by quickly fleeing the room, or orchestrating a massive Teletubbies intervention. Since they lack both legs and a ready supply of plushies, internet devices usually just shut down.
That’s what happened to us. Another device on the internet flooded one of our servers with an overdose of SYN packets, and it shut down – bringing the rest of Revision3 with it. In webspeak it’s called a Denial of Service attack – aka DoS – and it happens when one machine overwhelms another with too many packets, or messages, too quickly. The receiving machine attempts to deal with all that traffic, but in the end just gives up.
(Note the photo of our server equipment responding to the DoS Attack)
In its coverage Tuesday CNet asked the question, “Now who would want to attack Revision3?” Who indeed? So we set out to find out.
Internet attacks leave lots of evidence. In this case it was pretty easy to see exactly what our shadowy attacker was so upset about. It turns out that those zillions of SYN packets were addressed to one particular port, or doorway, on one of our web servers: 20000. Interestingly enough, that’s the port we use for our Bittorrent tracking server. It seems that someone was trying to destroy our bittorrent distribution network.
Let me take a step back and describe how Revision3 uses Bittorrent, aka BT. The BT protocol is a peer to peer scheme for sharing large files like music, programs and video. By harnessing the peer power of many computers, we can easily and cheaply distribute our huge HD-quality video shows for a lot less money. To get started, the person sharing that large file first creates a small file called a “torrent”, which contains metadata, along with which server will act as the conductor, coordinating the sharing. That server is called the tracking server, or “tracker”. You can read much more about Bittorrent at Wikipedia, if you really want to understand how it works.
Revision3 runs a tracker expressly designed to coordinate the sharing and downloading of our shows. It’s a completely legitimate business practice, similar to how ESPN puts out a guide that tells viewers how to tune into its network on DirecTV, Dish, Comcast and Time Warner, or a mall might publish a map of its stores.
But someone, or some company, apparently took offense to Revision3 using Bittorrent to distribute its own slate of shows. Who could that be?
Along with where it’s bound, every internet packet has a return address. Often, particularly in cases like this, it’s forged – or spoofed. But interestingly enough, whoever was sending these SYN packets wasn’t shy. Far from it: it’s as if they wanted us to know who they were.
A bit of address translation, and we’d discovered our nemesis. But instead of some shadowy underground criminal syndicate, the packets were coming from right in our home state of California. In fact, we traced the vast majority of those packets to a public company called Artistdirect (ARTD.OB). Once we were able to get their internet provider on the line, they verified that yes, indeed, that internet address belonged to a subsidiary of Artist Direct, called MediaDefender.
Now why would MediaDefender be trying to put Revision3 out of business? Heck, we’re one of the biggest defenders of media around. So I stopped by their website and found that MediaDefender provides “anti-piracy solutions in the emerging Internet-Piracy-Prevention industry.” The company aims to “stop the spread of illegally traded copyrighted material over the internet and peer-to-peer networks.” Hmm. We use the internet and peer-to-peer networks to accelerate the spread of legally traded materials that we own. That’s sort of directly opposite to what Media Defender is supposed to be doing.
Who pays MediaDefender to disrupt peer to peer networks? I don’t know who’s ponying up today, but in the past their clients have included Sony, Universal Music, and the central industry groups for both music and movies – the RIAA and MPAA. According to an article by Ars Technica, the company uses “its array of 2,000 servers and a 9GBps dedicated connection to propagate fake files and launch denial of service attacks against distributors.” Another Ars Technica story claims that MediaDefender used a similar denial of service attack to bring down a group critical of its actions.
Hmm. Now this could have been just a huge misunderstanding. Someone could have incorrectly configured a server on Friday, and left it to flood us mercilessly with SYN packets over the long Memorial Day weekend. If so, luckily it was pointed at us, and not, say, at the intensive care unit at Northwest Hospital and Medical Center But Occam’s razor leads to an entirely different conclusion.
So I picked up the phone and tried to get in touch with ArtistDirect interim CEO Dimitri Villard. I eventually had a fascinating phone call with both Dimitri Villard and Ben Grodsky, Vice President of Operations at Media Defender.
First, they willingly admitted to abusing Revision3’s network, over a period of months, by injecting a broad array of torrents into our tracking server. They were able to do this because we configured the server to track hashes only – to improve performance and stability. That, in turn, opened up a back door which allowed their networking experts to exploit its capabilities for their own personal profit.
Second, and here’s where the chain of events come into focus, although not the motive. We’d noticed some unauthorized use of our tracking server, and took steps to de-authorize torrents pointing to non-Revision3 files. That, as it turns out, was exactly the wrong thing to do. MediaDefender’s servers, at that point, initiated a flood of SYN packets attempting to reconnect to the files stored on our server. And that torrential cascade of “Hi”s brought down our network.
Grodsky admits that his computers sent those SYN packets to Revision3, but claims that their servers were each only trying to contact us every three hours. Our own logs show upwards of 8,000 packets a second.
“Media Defender did not do anything specific, targeted at Revision3″, claims Grodsky. “We didn’t do anything to increase the traffic” – beyond what they’d normally be sending us due to the fact that Revision3 was hosting thousands of MediaDefender torrents improperly injected into our corporate server. His claim: that once we turned off MediaDefender’s back-door access to the server, “traffic piled up (to Revision3 from MediaDefender servers because) it didn’t get any acknowledgment back.”
Putting aside the company’s outrageous use of our servers for their own profit, and the large difference between one connection every three hours and 8,000 packets a second, I’m still left to wonder why they didn’t just tell us our basement window was unlocked. A quick call or email and we’d have locked it up tighter than a drum.
It’s as if McGruff the Crime Dog snuck into our basement, enlisted an army of cellar rats to eat up all of our cheese, and then burned the house down when we finally locked him out – instead of just knocking on the front door to tell us the window was open.
In the end, here’s what I know:
- A torrential flood of SYN packets rained down on Revision3’s network over Memorial Day weekend.
- Those packets – up to 8,000 a second – came primarily from computers controlled by MediaDefender, who is in the business of shutting down illegal torrent sites.
- Revision3 suffered measurable harm to its business due to that flood of packets, as the attacks on our legitimate and legal Torrent Tracking server spilled over into our entire internet infrastructure. Thus we were unable to serve videos and advertising through much of the weekend, and into Tuesday – and even our internal email servers were brought down.
- Denial of service attacks are illegal in the US under 12 different statutes, including the Economic Espionage Act and the Computer Fraud and Abuse Act.
Although I can only guess, here’s what I think really happened. Media Defender was abusing one of Revision3’s servers for their own purposes – quite without our approval. When we closed off their backdoor access, MediaDefender’s servers freaked out, and went into attack mode – much like how a petulant toddler will throw an epic tantrum if you take away an ill-gotten Oreo.
That tantrum threw upwards of 8,000 SYN packets a second at our servers. And that was enough to bring down both our public facing site, our RSS server, and even our internal corporate email – basically the entire Revision3 business. Smashing the cookie jar, as it were, so that no one else could have any Oreos either.
Was it malicious? Intentional? Negligent? Spoofed? I can’t say. But what I do know is that the FBI is looking into the matter – and it’s far more serious than toddlers squabbling over broken toys and lost cookies.
MediaDefender claims that they have taken steps to ensure this won’t happen again. “We’ve added a policy that will investigate open public trackers to see if they are associated with other companies”, promised Grodsky, “and first will make a communication that says, hey are you aware of this.”
In the end, I don’t think Media Defender deliberately targeted Revision3 specifically. However, the company has a history of using their servers to, as Ars Technica said, “launch denial of service attacks against distributors.” They saw us as a “distributor” – even though we were using Bittorrent for legitimate reasons. Once we shut them out, their vast network of servers were automatically programmed to implement a scorched earth policy, and shut us down in turn. The long Memorial Day weekend holiday made it impossible for us to contact either Media Defender or their ISP, which only exacerbated the problem.
All I want, for Revision3, is to get our weekend back – both the countless hours spent by our heroic tech staff attempting to unravel the mess, and the revenue, traffic and entertainment that we didn’t deliver.
If it can happen to Revision3, it could happen to your business too. We’re simply in the business of delivering entertainment and information – that’s not life or death stuff. But what if MediaDefender discovers a tracker inside a hospital, fire department or 911 center? If it happened to us, it could happen to them too. In my opinion, Media Defender practices risky business, and needs to overhaul how it operates. Because in this country, as far as I know, we’re still innocent until proven guilty – not drawn, quartered and executed simply because someone thinks you’re an outlaw.
- Jim Louderback
CEO - Revision3
UPDATE
We’ve received several requests for some technical data to illustrate the specifics of the attack. So we’ve provided a text file with some more “under the hood” data.
This file represents every packet we identified as being part of the DoS for a period of time less than .02 *seconds* on Monday morning. If you count, there’s a total of 96 packets. (We removed 12 legitimate packets from the trace). We used a combination of tcpdump and wireshark to gather this information. (this particular trace is from tcpdump)
View the text file: rev3packettrace.txt
