Denial-of-service attack on Amazon S3

You can see a non-technical summary of this article here.

Amazon S3 is a new service which uses Amazon’s world-wide network of computers to provide fast, secure, and essentially infinite data storage on the Internet, metered and paid for according to usage. It is beautifully implemented and it is the kind of elegant technology that makes you want to need it.

We are working on a new Cardbox feature that will use Amazon S3 and we’ll announce it as soon as it’s ready for people to test. But in the course of development we’ve come across a vulnerability. An attack aimed at this vulnerability makes Amazon S3 (and any data stored on it) completely unusable by the victim. Note: the vulnerability is not inherent to Amazon S3 itself and the attack would work against any similar service.

Interestingly, the attack only works if you are using security software to protect yourself from computer worms and viruses: it is your computer’s own immune response that does the damage.

The discovery

The first we knew of any potential problem was when we were benchmarking the S3 storage by uploading a number of databases to S3 and then downloading them again. For this experiment we chose the sample databases that we normally distribute on the Cardbox CDROM. All eight databases were uploaded easily, but when we tried downloading them we got an error message saying that the link had been disconnected halfway through a download.

Repeating the experiment with someone watching the screen, the disconnection happened again. When we tried to look at our Amazon S3 data we found them inaccessible, and soon we realised that it was impossible to connect to the Amazon S3 server (s3.amazonaws.com) at all. It seemed that the service had ceased to exist.

Norton Internet Security errorAt about the time that all this started, a message had popped up on the screen saying that Norton Internet Worm Protection had detected and blocked an “ICC Profile TagData Overflow” attack from s3.amazonaws.com. We couldn’t find any useful explanation of what such an attack might be. Clicking through the various Norton options eventually brought up a screen that showed that the IP address 207.171.168.42 had been blocked. When we unblocked it, we were once more able to connect to Amazon S3.

The simple vulnerability and how to protect against it

Norton Internet Worm Protection had evidently decided that the file we were trying to download (an ordinary Cardbox database file called Album.fil, which is distributed on the standard Cardbox CD) contained an ICC Profile TagData Overflow attack, whatever that may be. We had several possible courses of action:

  1. Tell Norton to ignore ICC Profile TagData Overflow attacks.
  2. Tell Norton not to block “attacking” computers automatically. This would still make Album.fil inaccessible, but at least the whole of Amazon S3 would not disappear every time Norton thought it had detected an attack.
  3. Encrypt the data being uploaded and downloaded so that Norton cannot see any meaningful patterns in the data.

Data encryption is not always appropriate, especially for backups; but nevertheless it is probably what we will do with Cardbox in the future, since it is clearly unacceptable for users to find all their data suddenly unusable, and it is unreasonable to expect them to fiddle with their anti-virus settings simply in order to be allowed to use their own data.

The denial-of-service attack

Amazon S3 provides Internet-based storage so fast and reliable that people will use it instead of disk drives of their own – either for primary data storage or for backups. Think of iTunes, for example. For about $3 a month I can store my entire iTunes music collection on S3 and not worry about having to burn handfuls of DVDs at regular intervals: I might even have a program (“aTunes”?) that does all the backing up for me automatically. Or I may just want to have a very big hard disk that isn’t physically part of my computer, doesn’t need backing up, and is available to me wherever I happen to be: something like JungleDisk.

Either way, I am not consciously accessing the Web. The fact that Amazon S3 uses the HTTP protocol is something I may never even discover unless I read the small print.

Now suppose that an attacker intentionally causes the situation that we described earlier – where Norton Internet Worm Protection causes Amazon S3 to disappear from the known universe. I won’t just lose access to a web site or two: I will lose everything. All my files will disappear, all my music, all my data.

How to launch the attack

Files exist that make Norton Internet Worm Protection think that an attack is under way. (The attack does not have to be ICC Profile TagData Overflow: any attack recognised by NIWP will do).

  1. Upload such a file to Amazon S3.
  2. Create a Web page that makes a user’s web browser download the file.
  3. Get the user to visit that page.
  4. The user’s web browser will download the file; NIWP will detect the download, treat it as an attack, and block the IP address that the attack came from. Since this is the IP address of Amazon S3, the block will prevent any communication with Amazon S3. Result: all the user’s files, music and data cease to exist, just as we wanted.

We have tried this and it works.

Step 1 was easy because we already had our (unjustly accused) Album.fil. As for the Amazon S3 account: any Amazon customer can sign up for one of those. We renamed Album.fil to have a name ending in “.jpg” before we uploaded it.

Step 2 was easy as well: we created a web page that included a reference to the uploaded “.jpg” file. The fact that the “.jpg” wasn’t really an image file didn’t matter: the web browser couldn’t tell it wasn’t one until after it had downloaded it, and the downloading was all we needed for the attack. We made the image reference very, very small (height and width both 1 pixel) so that there was nothing strange in the appearance of the page.

We haven’t attempted step 3. But there are some elegant ways of mounting the attack even without creating a page of one’s own. For example, many discussion forum systems (including Amazon’s) allow you to include images in your postings and replies. Including the bogus .jpg file in a message on such a forum will cause everyone who views the message to download the .jpg file, and will thus mean that everyone who views the message and has Norton Internet Worm Protection active will have locked himself out of Amazon S3.

Protecting against the denial-of-service attack

There are two things that the user can do:

  1. Don’t use Amazon S3 or any other online storage service.
  2. Don’t use Norton Internet Worm Protection.

Both are somewhat radical. On the whole, given the usefulness of online storage, I’d opt for not using NIWP. But many corporate users will not be allowed by their administrators to modify their “protection” settings.

There are characteristics of the HTTP traffic generated by Amazon S3 that distinguish it from ordinary web surfing. Most importantly, most of it is generated by programs or by specially programmed scripts. It will not be difficult for these to send an extra header in their HTTP requests that say “Please, Mr NIWP, don’t inspect the response to this request”. Ordinary web browsing won’t generate that header and so it will continue to be protected against genuine threats.

How easy would it be to persuade security software vendors to implement something like this? The online storage industry is young but it is growing fast, and some of the vendors who are moving into it are, like Amazon, not small companies. At some point the consensus will move from “S3 is not compatible with Norton” to “Norton is not compatible with S3″, and Symantec will have to do something long before that happens. It will be an interesting process to watch. For “S3″, read the name of any online data storage service. For “Symantec/Norton”, read the name of any security software vendor.

Granularity of trust

What has caused all the trouble in this case is a single unspoken assumption: that IP addresses can be used to determine trustworthiness and that if suspicious data have come from a given IP address then everything coming from that IP address is equally untrustworthy.

This idea (granularity of trust ≏ granularity of addressing) works for small sites that live on one computer with one IP address. It under-protects when accessing sites that have multiple IP addresses (some load-balancing schemes do this). It over-protects when accessing sites that are hosted on a shared computer, since if one site on that computer is blocked as being suspicious then all the others will be blocked as well. It over-protects catastrophically with an online storage service such as Amazon S3, when just one malicious file can cause access to be blocked to all the data that the user owns.

The concept of “IP address = unit of trust” is becoming obsolete. It may be possible to avoid abandoning it immediately, if storage vendors and security vendors can reach agreement; but this may be nothing more than a temporary reprieve.

About these ads

4 Responses to “Denial-of-service attack on Amazon S3”

  1. Markus Kuhn Says:

    I’m surprised that you describe this as a vulnerability of Amazon’s S3. This has really nothing at all to do with Amazon S3 or even with any particular class of web services. This denial-of-service attack is entirely created and caused by a badly engineered, oversensitive intrusion-detection-system with overzealous automated response (namely blocking an IP address forever) on your computer, in your particular example Norton Internet Worm Protection. Amazon’s S3 became merely by accident the service that you happend to be playing with when you first stumbled across how dangerous it can be to install too aggressive intrusion-detection systems on your PC. They far too easily become the problem rather than a solution and ultimately increase the attackable surface of your PC by adding lots of badly thought through functionality to your local network stack. I would never want to buy or install any product that destroys the binary transparency of my network connections.

  2. cardbox Says:

    Markus, you are quite right, this is essentially a problem caused by badly engineered and overzealous intrusion detection software. But remember that users know (because Symantec spend a lot of money telling them) that Norton Internet Worm Protection protects their systems and cannot possibly do any harm: indeed, that a Windows system without Norton is only half a system and only irresponsible people would use it. Given the certainty that Norton can never be at fault, the user will perceive this attack as a problem with the first web service that he sees being attacked.

    Anyone who has ever sold software for Windows will know how much time is wasted working round the interference caused by undocumented, badly thought through anti-virus software. Without any public acknowledgement of what the software is actually doing, how it does it, and whether any of it makes sense, there is no pressure on the vendors to produce anything better. Microsoft themselves are not innocent in this regard: they regularly release Windows "security updates" that stop perfectly normal software from working: see, for example, our forum postings from 27 July 2005, 14 October 2005, and 21 April 2006.

    For the moment, problems such as the one I have described in this posting will be perceived by users as being problems with the Amazon S3 service, or with whatever other web service they are accessing – or with programs built on those services – because they know that nothing else can be at fault. But what is interesting is that at last we have a chance of reaching a tipping point, when "Amazon S3 has problems on an ordinary Windows system [which happens to be running Norton]" suddenly flips over into "Norton causes problems for ordinary Windows systems [which happen to be accessing Amazon]". This could not happen as long as the victims of Norton's PR-heavy incompetence were small software vendors; but now that some big players will be suffering, and there is real money involved, the change has a chance of really taking place.

    At that point, users will stop seeing anti-virus software as a panacea and will start considering its quality and what it actually does; and the vendors will have to change their culture completely in order to accommodate this. For many of them, this switch from undocumented, feature-laden (broadly speaking) "American" to "European" attitudes to software quality will turn out to be impossible.

  3. D. Richter Says:

    What if the IP address is not that of Amazon & you were not downloading anything from Amazon S3? The IP address listed on my Norton is 70.183.191.43(http(80)) with no mention of Amazon S3? Is it still some type of worm? Thank you! Deborah

  4. cardbox Says:

    You’ll have to ask Symantec. They wrote Norton Anti-Virus so they will know what the messages mean.

Comments are closed.


Follow

Get every new post delivered to your Inbox.

%d bloggers like this: