Tell HN: Rise in AWS accounts getting hacked and owner being stuck with the bill

112 points by Kpourdeilami 4 years ago | 105 comments

I have been seeing a lot of posts on Reddit and other forums of mostly students setting up an AWS account only for them to be hacked and account owner being stuck with a significant bill.

Most likely scenario is hackers are trying leaked username/password pairs from other breaches against AWS and gaining access to those accounts.

They then spin up EC2 instances in all sorts of regions on the compromised accounts

PSA set up MFA on your account if you haven't already.

Some examples:

https://www.reddit.com/r/aws/comments/rv3lm5/i_lost_55k_from...

https://www.reddit.com/r/aws/comments/rvbncu/account_hacked_...

https://www.reddit.com/r/aws/comments/qx8i02/got_hacked_and_...

https://www.reddit.com/r/aws/comments/rv4mnq/my_account_was_...

ujetin 4 years ago |

I deleted my AWS account yesterday. It is obviously catered towards large organisations - very complicated tools and pricing that I couldn't really fit into my use case. I tried to just shut down the services that were using money but wasn't even sure I had found them all so I just closed the whole account.

I don't even like the idea of any of this stuff. I want to run my own little raspberry pi server or whatever, it seems much more fun and startupish than aws, which appears to be all of the corporate stuff I left (AIMs etc). This is funny because I remember AWS being thought of as great for "just experimenting with stuff".

stackbutterflow 4 years ago | |

It says here that closing your account isn't enough:

> Closing your account might not automatically terminate all your active resources. You might continue to incur charges for some of your active resources even after you close your account.

https://aws.amazon.com/premiumsupport/knowledge-center/termi...

toomanydoubts 4 years ago | | |

Is that even legal? How is it the user fault that aws can't properly track and terminate the account resources? (Altough they seem very efficient in tracking usage when the goal is to charge credit cards).

Sebb767 4 years ago | |

AWS is great if you have a very large infrastructure budget. Playing around on a test AWS account where thousand dollars plus or minus is no big deal is a lot of fun. Playing around while trying to keep the bill under 10$ is just torture.

300bps 4 years ago | | |

AWS has a free tier that lasts for a year. I actually used it to get hands-on experience with enough services that I was able to get my AWS Certified Solutions Architect, Cloud Practitioner and Certified Developer certifications. I easily spent 60 hours creating and deleting services and it didn't cost me a cent.

I don't even know how you'd accidentally get to thousands of dollars accidentally unless you were doing something more complicated like setting up an autoscaling group.

Spin up a t3a.micro EC2 instance and run it as much as you like - it's free. If you go into it (as I did many years ago) thinking, "Let me just spin up a 16 core server with 128 GB of RAM" because that's what you'd set up in your on-prem data center then of course you're going to have a big bill.

But that's not what cloud is about. It's about resources popping in and out of existence for the minimal time and resources needed to accomplish a task.

thejosh 4 years ago | |

Yep, the only way to do this is using aws-nuke [0], and the billing is such a mess for small use cases.

https://github.com/rebuy-de/aws-nuke

Osiris 4 years ago | |

I'm really surprised that people immediately jump to cloud providers even for personal projects when you can lease VPS and server instances for much cheaper.

A buddy of mine runs a crypto currency validator in AWS and pays over $1600 a month.

You can rent a dedicated server with 32 cores, 256GB RAM and 2TB of NVMe storage for $400 a month. Tent two in different data centers for redundancy and you'll still paying half, not to mention no surprise monthly bills due to increased traffic or creating an expensive snapshot of your DB.

halilduygulu 4 years ago | | |

how is it still profitable after 1600 cost?

guywhocodes 4 years ago | |

I shutdown the credit card i was using. They're intentionally making it hard and I decided it was less work to update payment methods on everything else.

dzhiurgis 4 years ago | | |

Good point. I probably should replace all the credit cards I have with temporary ones...

oneplane 4 years ago | |

Depends on your needs, scale and experience. If the product doesn't suit you, it's worthless (or very expensive depending on perspective).

If you just want 'a server', then AWS isn't the place to go to.

To elaborate a bit more: I tend to draw the line on IaC. For example: if you're doing GitOps with Terraform and Atlantis, then go for AWS. If not, you're probably not at the point where it makes sense anyway.

karanbhangui 4 years ago | |

love digitalocean for this

stunt 4 years ago | |

In AWS, you can create everything under a Cloudformation stack, once you delete the stack, all its resources will be deleted automatically.

The problem however is all the beginner guides including those coming from AWS team themselves teach people to create resources via console without using a Cloudformation stack.

alanfranz 4 years ago | |

> It is obviously catered towards large organisations

And yet it seems that only the big cloud providers offer a reasonable authentication and authorization approach.

For many other providers, api keys lend full account access (no resource limitation, no read-only/subset of actions), which is unacceptable from a security pov.

nam17887 4 years ago | |

Have you tried the billing explorer? It shows a pretty detailed report of what you use.

andrewguenther 4 years ago |

MFA needs to be forced on by default for all new accounts created with the UI. It is utterly irresponsible for AWS to not do this. People always counter with "you should know better!" But AWS markets itself to college kids. People were all up in arms and Robinhood allowing 18 year olds to create accounts which could result in unbounded losses, but AWS somehow gets a pass?

jackson1442 4 years ago |

@aws, why not mandate MFA for a root user? in child org accounts where this is less feasible, you could allow access to the root user only from the parent account, no direct login at all.

staticassertion 4 years ago | |

One option is to set your root user's password to some random 64 character string and forget it. Any time you want root access (rare) you go through a reset flow, which means your root auth is tied to your email. Something like GMail has pretty strict controls so this is actually imo the safest option available.

jackson1442 4 years ago | | |

While this may be safest, it doesn’t make sense why Amazon doesn’t save themselves a couple (hundred) grand in refunds by locking down root accounts.

joombaga 4 years ago | | |

You can keep the password unset as well.

oneplane 4 years ago | |

You can create rootless organisation-managed child accounts. This is a solved problem.

andrewguenther 4 years ago | | |

Organization child accounts are not rootless, they're just seeded with a random password

smackeyacky 4 years ago | |

You just reminded me to set up MFA on my root account so thanks!

BackBlast 4 years ago |

I migrated off AWS in December and closed my account. With the outages, they are likely looking at revenue shortfalls. Outages don't scare me, but their billing practices are already uncomfortable. This may make them prone to be more stingy about bill forgiveness. Bill forgiveness is inherent in the unsafe billing model which makes even having the account open even more of a huge liability.

I'm done.

javagram 4 years ago |

Good reminder to completely close my AWS account. I have TOTP MFA on it but having a AWS account that had the same root login as my Amazon retail account was risky and a mistake from the beginning. Luckily I haven't used it for anything in years so it was as simple as following the "Close account" procedure.

I'll use Digital Ocean for anything small if I need to spin up a server in the future.

sgarg26 4 years ago |

This happened to me. I had a 9k bill and I got hacked and stuck with it on my personal account. I gave AWS $100k business and that did not matter through being a decision maker at a startup I work at. AWS does not care about your business unless you are going to IPO. True story. Feel free to message me at Sgargconsulting --> GMAIL

vmception 4 years ago |

This happened to me, the bill was so ridiculous that I wasnt even bothered by it. It got voided by aws support as predicted.

MFA does not prevent this. Its IAM keys.

technion 4 years ago | |

This is an underrated point. Every "best security practices" guide you read has you setup MFA for console access, then create IAM keys with no such protection.

The credentials I'm using with Terraform require MFA in order to call 'AssumeAdmin'. Everyone I've ever shown this configuration has complained about it being overkill and tried to argue Terraform should just have IAM keys sitting on disk, one desktop compromise away from taking everything. And since it's used to provision basically everything it's a highly privileged account.

notemaker 4 years ago | | |

aws-vault works too

heavyset_go 4 years ago | |

How big was the bill?

vmception 4 years ago | | |

Like $120,000 after paying like $.30 for cloudfront for 10 months

It was pretty obviously uncharacteristic

anothernewdude 4 years ago |

If only they had some kind of charge limiting on accounts like their customers have been asking for years now.

Aeolun 4 years ago |

I find it hard to believe AWS would try to suck blood from a stone. I’m willing to believe they get a 55k bill, but do they actually end up paying those?

anduru_h 4 years ago | |

I think you can get refunded for fraudulent charges, like if you get hacked. But if you leave an instance running or screw up auto-scaling somehow, that's normally when the charges may tend to stick around. I believe they have a one strike policy, though that could have changed.

NikolaeVarius 4 years ago | |

No.

quickthrower2 4 years ago |

Thanks! You've reminded me to close my hobby AWS account. I will do that now.

Edit: Done! Was quite easy. Luckily I had no services running or needed.

blibble 4 years ago |

their hardware MFA functionality is worse than useless

it only permits a single hardware token to be registered to an account

so good luck if you misplace or break your hardware token

halotrope 4 years ago | |

When setting up the token you can scan the QR with multiple devices. E.g YubiKey and Authenticator App. This at least allows for a backup in case one goes missing. I agree it is kind of incredible that multiple tokens are not supported.

Sebb767 4 years ago | | |

You actually can't mix hardware tokens and OTP apps. You're only option is to scan the code twice and skip hardware tokens entirely (which is quite reasonable, as the recovery for an app would be easier than for a failed/lost hardware token).

Note, though, that the new SSO login actually supports MFA in a normal way.

aborsy 4 years ago | | |

You create several users with admin privileges each with separate MFA.

nefitty 4 years ago | | |

Don't token limits reduce attack surface?

timf 4 years ago | |

I did not like that limitation either and get around it by using the Yubico Authenticator app with three separate hardware tokens configured with the same shared TOTP seed. There's also path to reset your root account access, btw.

jeppesen-io 4 years ago | |

The point is to block unauthorized access. It is not less than useless. It absolutely does that. You really think if your lost your token Amazon would lock you out forever? Wrong assumption

Besides, just use AUTHY if that's your concern

javagram 4 years ago | | |

FWIW, at a workplace I'm familiar with, the hardware MFA devices for a group 'disappeared' from the office during COVID and despite the company having an active relationship and contracts with AWS, it was taking over a year to get access to the root accounts in question reset. It is not an easy process.

That said, I have to imagine this is the wrong procedure and there's some way to duplicate hardware MFA devices to have redundancy for such a case...

staticassertion 4 years ago | |

That's bad and annoying but it is not at all worse than useless. A hardware token is a very worthwhile investment.

jeppesen-io 4 years ago |

MFA is good advice

Also, I'd create a IAM user and severely limit your usage of the root account, and over time stop using the root completely.

staticassertion 4 years ago |

Well, yeah, of course they're stuck with the bill. I feel like people think AWS is supposed to have infinite guard rails regardless of what the engineers using it do, like when people write code that infinite loops and it blows up their bill.

AWS gives money back in a lot of cases that I think they legitimately aren't responsible for.

I don't know that other cloud providers are going to do any better - an attacker who has your credentials and spins up 10's of thousands of dollars of infra will cost you thousands of dollars.

I'll certainly echo the advice for 2FA but, more importantly, use a strong, unique password.

Gigachad 4 years ago | |

I don't feel that some kind of quota management or predictable pricing is a significant ask. Most people or orgs do not need instant and infinite scaling. And would rather a 1 hour outage while they sort things out rather than a $100,000 bill for a minor bug.

caslon 4 years ago | | |

Don't use a "cloud" vendor then? Just use a VPS vendor.

cortesoft 4 years ago | |

Why should a customer be stuck with the bill in the case of fraud? If someone fraudulently buys something in a store with my stolen credit card, I am not liable to pay for those purchases. Why would it be different for AWS services?

46756e 4 years ago | | |

Right, but if someone buys say a shirt with your stolen card, it isn’t the store that picks up the bill. It’s the credit card company.

technion 4 years ago | |

I'm surprised MFA isn't mandated by default. It is in Azure:

https://docs.microsoft.com/en-us/azure/active-directory/fund...

zinekeller 4 years ago | | |

That link looks like MFA can be disabled but this chart shows otherwise (baseline versus opt-in enhanced): https://docs.microsoft.com/en-us/azure/active-directory/fund...

Olreich 4 years ago | |

Yep, buying goodwill is very effective.

smackeyacky 4 years ago |

I know it's easy to get lazy about checking your AWS billing dashboard but I do it once a week - you can set up alerts and whatnot but I find it easier just to go look at the current usage to make sure nothing has gone awry.

danial 4 years ago | |

I agree that you have to make it a daily or weekly habit to check because 30 days is too long a time to incur costs you're not aware of. That's why I built Billgist.com, same idea but it sends a daily email with usage amount right in the subject, like: $16.56 XYZAccount – Daily AWS billing alert.

digitalsushi 4 years ago | |

I think the point is that someone could bankrupt the average hobbyist in an hour or two, if they were expected to pay it.

danial 4 years ago | | |

I think that rate of usage would trigger EC2 abuse alerts. Usually it's something that looks plausible but accumulated over 30 days.

spekcular 4 years ago |

The fact that AWS has no way to limit billing seems insane to me. Your only recourse for an accidental (or malicious) overcharge is beg customer support. It's an incredible liability.

aborsy 4 years ago |

Services like Lightsail have caps and flat fees. AWS needs more of this.

halilduygulu 4 years ago |

please, everyone enable MFA and billing alerts in your accounts. do not commit access&secret keys to github.

edoceo 4 years ago |

Can you connect CloudWatch to your billing so you'd get an alert on some kind of spikes?