$5,000 Security Breach

sergiotapia 12 years ago |

>'They' spin up spot instances which isn't subject to Billing Alerts. You'll need to cancel those spot instances, revoke your AWS credentials, and change your account password," he said.

This doesn't make sense at all. Amazon should let us if monthly bill > X send me a priority email and phone call. Why do they hide behind these dark patterns? I thought they were better than that.

theboss 12 years ago | |

The fact of the matter is amazon is a giant company and cannot thoroughly think through each piece of logic in their system.

The reward for focusing on this before-hand is much lower than just writing a check for $5k to this person and then fixing later (lot of $5k checks from amazon today. Wheres mine?)

bequanna 12 years ago |

>"'They' spin up spot instances which isn't subject to Billing Alerts.

Spot Instances aren't subject to AWS Billing Alerts? Is this common knowledge?

ceejayoz 12 years ago | |

I'm not sure it's true.

matthuggins 12 years ago |

Why the cliffhanger? Why not just finish your thought? As is, this story provides nothing other than showing the Amazon provided some helpful customer service.

Jare 12 years ago |

We had an almost identical event last Sunday in one of our dev accounts: multiple high end spot instances in multiple regions with a newly created security group pointing to a suspect IP.

We caught and corrected it quickly, but we still don't know how the keys leaked out - we have chalked it up to lower security practices since it's not a production account and is shared by more people (e.g. no 2-factor on it). We started to investigate, but then Heartbleed happened.

I wish there were more mechanism in AWS to prevent bills from mounting up, but the basic billing alarms worked in this case. I can't imagine how or why spot instances would be excluded from alerts, their cost certainly is included in the estimates that alerts are based on.

colinbartlett 12 years ago |

Another reminder to be extra careful about checking in AWS credentials into version control. You never know when you might open source that repository and someone can easily yank it from Github.

friendstock 12 years ago | |

Actually this is what happened to us... our AWS credentials had accidentally been put into Github for a hackathon project several months ago.

Coincidentally, the incident also occurred around the same time (April 1-2). We were hit with $13,000 worth of EC2 usage before we shut them down and changed our AWS key... We reported to Amazon, and they are working on a refund.

nilved 12 years ago |

The instance was spun up on April 2, but Heartbleed wasn't disclosed for almost a week later. I highly doubt anybody used the Heartbleed 0-day to access your account.

viseztrance 12 years ago | |

According to Cloudflare (http://blog.cloudflare.com/answering-the-critical-question-c...), exploiting heartbleed may actually be very difficult. So yeah, it's very unlikely for that to have happened.

dmix 12 years ago | | |

Well, getting an SSL private key is difficult as they don't often get into memory and are quite long (difficult to get from 64k at a time). Whereas AWS credential keys are something that get into your servers RAM much more frequently and are shorter strings. So it could easily be remote memory exploitation. But more likely social engineering or some other easy path in.

ceejayoz 12 years ago | | |

People accidentally send those keys off to Github all the time. I'd suspect that sort of thing.

mschuster91 12 years ago | | |

Heartbleed only exposed SSL memory (like incoming or outcoming connections), but not other memory (particularly not program memory), containing AWS keys.

abhimir 12 years ago |

A similar thing happened to me when I made public a previously private repo on Amazon and forgot to scrub it for AWS key. Luckily I got an email from Amazon within 24 hours of the instances being started, and as such my bill was just $360, which in any case they waived off. You might want to check your Github repos.

jes5199 12 years ago |

I have 2-factor auth enabled on my AWS login - but am I right in thinking that if someone has my API keys that they don't need the 2nd factor?

prattbhatt 12 years ago | |

Yes, someone with your access and secret keys can spin up instances, create buckets, and do everything else that the stolen keys are authorized for.

ceejayoz 12 years ago | | |

Which is why most things should be done with IAM keys specifically locked down to minimal privileges.

jes5199 12 years ago | | |

Apparently I have access keys that predate the release of IAM ! Fortunately there's a convenient "disable" link on security keys page.

prattbhatt 12 years ago | | |

Agreed

Havoc 12 years ago |

The fact that the consultant - unprompted - knows exactly what is going on suggests to me that Amazon needs to fix this. If one customer gets burned then the customer is an idiot. If lots of customers get burned then maybe its not the customers that are the problem.

sandy23 12 years ago |

Is there anything you have done that might have compromised your account. Moreover bill could as well been much larger than $5000 i guess.

josephagoss 12 years ago |

Is it too pedantic to want the tech support to just say they were probably mining cryptocurrency instead of bitcoin? It's most likely the intruder was mining either litecoin or whatever coin is most profitable for the month. I know it's all very much the same but they were almost certainly not mining bitcoin.

A $5,000 AWS instance would mine about $1 worth of bitcoin and would not be worth the time logging into someones account.