Ask HN: No clue how AWS account was compromised – how to move forward?

16 points by 71a54xd 2 years ago | 5 comments

I recently got an email from AWS about an account I created for consulting work nearly six months ago. Someone managed to get in and stand up a bunch of EBS clusters. Fortunately, this was caught by logs at the startup where I was consulting via their analytics. This was an account I'd only logged into a few times with it's own separate email.

I'm sort of afraid to keep doing anything important or sensitive on the macbook where I logged into this account. Fortunately amazon has been great with support and it looks like I'm off the hook for the fees. But I'm in desperate need of advice for how to prevent this (even with 2fa) in the future and how to safely move all of my bitwarden credentials to a new completely sterile machine?

I guess I'm just a bit flustered since I've never had this happen before and I consider myself a solid developer with good security practices.

leros 2 years ago |

The same thing happened to me a year ago. I had an AWS account I barely used except for an SQS queue. Somehow, someone got in, changed the password, and set up a machine learning pipeline. I couldn't turn it off since I was locked out of the account and I racked up a $20k bill.

I also have no idea how they got it. I had 2FA set up and only had one service key created that I used in a Heroku environment variable.

Here's what happened:

- I contacted customer service. It took them several days to get back to me. Initially they told me they couldn't help and I would be responsible for any charges per their ToS as it's my responsibility to secure the account.

- After some back and forth, they reset my account credentials (the email was changed from me@mycompany.com to uuid@random.ru so it was obviously an account takeover).

- They listed out a list of services that had been started after the compromise and told me it was my responsibility to disable them and then tell them I did so.

- I cleaned things up the best I could and then told the service agent. They said I missed a few things and gave me more clear directions.

- By this point I had a $70k bill. Things had been running for about a week.

- I asked about getting a refund and they said they could do that but only after I set my account up with a proper security setup, which involved creating a bunch of separate small user accounts with minimal permissions.

- I did that, they refunded the charges, and then I deleted my account.

Long story short, it took a while and they weren't initially too helpful but ended up being very nice and helpful in the end.

busyant 2 years ago | |

> By this point I had a $70k bill. Things had been running for about a week.

Jesus. This is terrifying.

aborsy 2 years ago | |

Imagine if they had not reimbursed it!

leros 2 years ago | | |

No kidding. They told me they wouldn't reimburse me if it happened again which is why I deleted my AWS account and moved to something else.

KomoD 2 years ago |

> how to safely move all of my bitwarden credentials to a new completely sterile machine?

Bitwarden is synced so just wipe the machine and log in again?