You can see a non-technical summary of this article here.
Amazon S3 is a new service which uses Amazon’s world-wide network of computers to provide fast, secure, and essentially infinite data storage on the Internet, metered and paid for according to usage. It is beautifully implemented and it is the kind of elegant technology that makes you want to need it.
We are working on a new Cardbox feature that will use Amazon S3 and we’ll announce it as soon as it’s ready for people to test. But in the course of development we’ve come across a vulnerability. An attack aimed at this vulnerability makes Amazon S3 (and any data stored on it) completely unusable by the victim. Note: the vulnerability is not inherent to Amazon S3 itself and the attack would work against any similar service.
Interestingly, the attack only works if you are using security software to protect yourself from computer worms and viruses: it is your computer’s own immune response that does the damage.
The first we knew of any potential problem was when we were benchmarking the S3 storage by uploading a number of databases to S3 and then downloading them again. For this experiment we chose the sample databases that we normally distribute on the Cardbox CDROM. All eight databases were uploaded easily, but when we tried downloading them we got an error message saying that the link had been disconnected halfway through a download.
Repeating the experiment with someone watching the screen, the disconnection happened again. When we tried to look at our Amazon S3 data we found them inaccessible, and soon we realised that it was impossible to connect to the Amazon S3 server (s3.amazonaws.com) at all. It seemed that the service had ceased to exist.
At about the time that all this started, a message had popped up on the screen saying that Norton Internet Worm Protection had detected and blocked an “ICC Profile TagData Overflow” attack from s3.amazonaws.com. We couldn’t find any useful explanation of what such an attack might be. Clicking through the various Norton options eventually brought up a screen that showed that the IP address 126.96.36.199 had been blocked. When we unblocked it, we were once more able to connect to Amazon S3.
The simple vulnerability and how to protect against it
Norton Internet Worm Protection had evidently decided that the file we were trying to download (an ordinary Cardbox database file called Album.fil, which is distributed on the standard Cardbox CD) contained an ICC Profile TagData Overflow attack, whatever that may be. We had several possible courses of action:
- Tell Norton to ignore ICC Profile TagData Overflow attacks.
- Tell Norton not to block “attacking” computers automatically. This would still make Album.fil inaccessible, but at least the whole of Amazon S3 would not disappear every time Norton thought it had detected an attack.
- Encrypt the data being uploaded and downloaded so that Norton cannot see any meaningful patterns in the data.
Data encryption is not always appropriate, especially for backups; but nevertheless it is probably what we will do with Cardbox in the future, since it is clearly unacceptable for users to find all their data suddenly unusable, and it is unreasonable to expect them to fiddle with their anti-virus settings simply in order to be allowed to use their own data.
The denial-of-service attack
Amazon S3 provides Internet-based storage so fast and reliable that people will use it instead of disk drives of their own – either for primary data storage or for backups. Think of iTunes, for example. For about $3 a month I can store my entire iTunes music collection on S3 and not worry about having to burn handfuls of DVDs at regular intervals: I might even have a program (“aTunes”?) that does all the backing up for me automatically. Or I may just want to have a very big hard disk that isn’t physically part of my computer, doesn’t need backing up, and is available to me wherever I happen to be: something like JungleDisk.
Either way, I am not consciously accessing the Web. The fact that Amazon S3 uses the HTTP protocol is something I may never even discover unless I read the small print.
Now suppose that an attacker intentionally causes the situation that we described earlier – where Norton Internet Worm Protection causes Amazon S3 to disappear from the known universe. I won’t just lose access to a web site or two: I will lose everything. All my files will disappear, all my music, all my data.
How to launch the attack
Files exist that make Norton Internet Worm Protection think that an attack is under way. (The attack does not have to be ICC Profile TagData Overflow: any attack recognised by NIWP will do).
- Upload such a file to Amazon S3.
- Create a Web page that makes a user’s web browser download the file.
- Get the user to visit that page.
- The user’s web browser will download the file; NIWP will detect the download, treat it as an attack, and block the IP address that the attack came from. Since this is the IP address of Amazon S3, the block will prevent any communication with Amazon S3. Result: all the user’s files, music and data cease to exist, just as we wanted.
We have tried this and it works.
Step 1 was easy because we already had our (unjustly accused) Album.fil. As for the Amazon S3 account: any Amazon customer can sign up for one of those. We renamed Album.fil to have a name ending in “.jpg” before we uploaded it.
Step 2 was easy as well: we created a web page that included a reference to the uploaded “.jpg” file. The fact that the “.jpg” wasn’t really an image file didn’t matter: the web browser couldn’t tell it wasn’t one until after it had downloaded it, and the downloading was all we needed for the attack. We made the image reference very, very small (height and width both 1 pixel) so that there was nothing strange in the appearance of the page.
We haven’t attempted step 3. But there are some elegant ways of mounting the attack even without creating a page of one’s own. For example, many discussion forum systems (including Amazon’s) allow you to include images in your postings and replies. Including the bogus .jpg file in a message on such a forum will cause everyone who views the message to download the .jpg file, and will thus mean that everyone who views the message and has Norton Internet Worm Protection active will have locked himself out of Amazon S3.
Protecting against the denial-of-service attack
There are two things that the user can do:
- Don’t use Amazon S3 or any other online storage service.
- Don’t use Norton Internet Worm Protection.
Both are somewhat radical. On the whole, given the usefulness of online storage, I’d opt for not using NIWP. But many corporate users will not be allowed by their administrators to modify their “protection” settings.
There are characteristics of the HTTP traffic generated by Amazon S3 that distinguish it from ordinary web surfing. Most importantly, most of it is generated by programs or by specially programmed scripts. It will not be difficult for these to send an extra header in their HTTP requests that say “Please, Mr NIWP, don’t inspect the response to this request”. Ordinary web browsing won’t generate that header and so it will continue to be protected against genuine threats.
How easy would it be to persuade security software vendors to implement something like this? The online storage industry is young but it is growing fast, and some of the vendors who are moving into it are, like Amazon, not small companies. At some point the consensus will move from “S3 is not compatible with Norton” to “Norton is not compatible with S3”, and Symantec will have to do something long before that happens. It will be an interesting process to watch. For “S3”, read the name of any online data storage service. For “Symantec/Norton”, read the name of any security software vendor.
Granularity of trust
What has caused all the trouble in this case is a single unspoken assumption: that IP addresses can be used to determine trustworthiness and that if suspicious data have come from a given IP address then everything coming from that IP address is equally untrustworthy.
This idea (granularity of trust ≏ granularity of addressing) works for small sites that live on one computer with one IP address. It under-protects when accessing sites that have multiple IP addresses (some load-balancing schemes do this). It over-protects when accessing sites that are hosted on a shared computer, since if one site on that computer is blocked as being suspicious then all the others will be blocked as well. It over-protects catastrophically with an online storage service such as Amazon S3, when just one malicious file can cause access to be blocked to all the data that the user owns.
The concept of “IP address = unit of trust” is becoming obsolete. It may be possible to avoid abandoning it immediately, if storage vendors and security vendors can reach agreement; but this may be nothing more than a temporary reprieve.