Google Cloud: $72,000 bill overnight(theregister.com) |
Google Cloud: $72,000 bill overnight(theregister.com) |
Although tangential to the billing issue, this is reckless. If you’re building a crawler of any kind, please, please, please prioritize ensuring this doesn’t happen so I don’t have to wake up at 3 AM.
I run the infrastructure for a moderate-sized site with probably about a hundred million pages or so. We can handle the HN hug-of-death just fine. But poorly-made crawlers that recurse like this? They’re increasingly problematic.
If your solution to fixing your crawler is “throw more concurrency at it and ignore the recursion,” and suddenly your requests start timing out, that’s a pretty damn strong hint that you’re ruining someone’s day.
From my perspective, this will look like an attack. I’ll see thousands of IP addresses repeatedly requesting the same pages, usually with generic user agent headers. Which ones are actual attacks, and which are just poorly-made crawlers? Well, if you’ve got a generic user agent string that doesn’t link to a contact page, and you’re circumventing rate limiting by changing your IP address, and you had the bright idea to let your test code run overnight, I’m going to treat it as an attack. At 3 AM, I’m not inclined to differentiate between negligence and malice.
This is happening more and more often, and I partially blame it on the ease of “accidentally” obtaining a ridiculous quantity of cloud resources. People deploy shoddy test code and go to bed. They turn it off in the morning when they see the bill.
It’s become so prevalent that our company has come up with an internal term for these crawlers that spin up a new thread/container for every page: snowballing crawlers.
Save a sysadmin: don’t snowball.
Oh, and include a useful user agent header so we can contact you instead of your cloud provider.
Puppeteer etc. are nice and all but if you can get away with raw HTTP requests grabbing and parsing the HTML without pulling down stylesheets, JS, etc. do it. It is WAY more efficient than requesting the full overhead for the user experience from these folks and threading out 5-10 workers to gracefully crawl a site this way doesn't typically cause things to melt down on your target's end.
You may be saying "well I need a browser-stack or evaluated JS to do my work" and you may be right... but honestly though 90% of this stuff is reverse-engineer-able with Charles Proxy and some basic webdev experience. Heck - I've even sandboxed JS from a target's site to generate tokens/etc to cut down on repeat requests. Even CAPTCHA stuff can easily be done without having to pull down full UIX overhead these days.
---
"Save a sysadmin: don’t snowball."
Implement thread limits, rate limiting, throttling, intelligent caching, and try to fit within your target's hosting capabilities without being disrespectful. Often I will "smear" large jobs over weeks worth of time so that it's only a trickle of traffic here and there (and to also fly under the radar... sorry).
Also - on the custom UAS: Unless you're trying to make it easy to get blocked/identified then don't take this advice. Let's face it - this is a gray area for most. The best way is to not "snowball" and to make your scrapers indistinguishable from a reasonable stream of real users from real networks. I would never expect a sysadmin to contact me because frankly they aren't paid to.
---
One last thought - the people who are out there writing these bots/crawlers/etc. are often the lowest common denominator. They're the type that will get something "working" and hurry onto the next job because the nature of the work tends to be a ton of low-paid contract stuff. Also, at almost every place I've worked at in ecommerce that has scraping involved it's the bottom-rung dev talent that's assigned to the work.
Sucks, but near-100% I attribute your "snowball" situation to that.
I can’t speak for other sites, but we’re pretty good at picking up on crawlers that don’t have a unique UA. The problem is that we’re going to have a hard time differentiating your well-behaved crawler from more malicious crawlers, and you’re going to get caught in the crossfire.
> if you can get away with raw HTTP requests grabbing and parsing the HTML without pulling down stylesheets, JS, etc. do it.
If you combine that with the lack of an identifying UA, there’s unfortunately a good chance you’ll get caught in the crossfire during an actual attack. That being said, it’s good advice otherwise. If you’re trying not to be identified as a crawler, it’s really going to stand out, though.
> I would never expect a sysadmin to contact me because frankly they aren't paid to.
I am. Furthermore, as long as you’re being transparent about your activity (see: UA), I don’t mind working with you instead of your provider. I understand that writing good crawlers is a learning experience; mistakes do happen. When I send abuse reports, usually people just get a slap on the wrist, but not everyone is that lucky.
But, if your UA has contact info, I can:
1. Easily rate limit or block you until the issue is resolved
2. Contact you directly, explaining exactly what’s wrong
3. Easily unblock you once it’s fixed
Sure, I’m not going to be happy about it, but I’m going to be a lot happier than if you try to blend in—a situation in which I’m not going to have any sympathy.
Unfortunately, most sites don’t respond that way and would rather just block anything remotely suspicious. But since you can always change your IP address, maybe try with an identifiable UA first—please? :)
Edit: Also, a few recommendations to add:
1. Be prepared to handle obscure HTTP status codes. 503 indicates you need to back off. Frequent 500, 502, or 504 means the same thing. 429 and 420 mean you’re being rate limited; slow down. 410 means you should stop requesting the given URL. 400 or 405 means you probably have a bug. Any unrecognized 4XX or 5XX error should be flagged and examined so you can handle it better in the future.
2. You can send an X-Abuse-Info header and a generic UA if you want capable sysadmins to be able to identify you but want to avoid being blocked by inexperienced webmasters.
3. Don’t ignore abuse reports.
4. Try to be consistent and ramp up slowly. It’s harder to cope with unnaturally-abrupt increases in traffic.
Typically, one doesn't care whether the same page has been visited before. What one does care about is avoiding storing duplicate data.
If it walks like an attack and it quacks like an attack...
I am lacking in sympathy for the perp here, as being careless like this has probably caused problems and possibly cost significant money for a lot of people.
However, this is also a compelling demonstration of why cloud services should be required to provide a hard price cap option for safety reasons. I've heard all the self-serving arguments they make about how turning things off surprisingly might be unwanted behaviour and so on. If that's the case, the admin won't set a cap. But there are exactly zero circumstances under which someone who intended to cap their usage at a level that would cost single digits of dollars or remain within a free plan intends or wants to run something that costs four orders of magnitude more than that, and IMNSHO such predatory pricing models should be illegal (assuming that the charges aren't already considered unenforceable by courts under such circumstances; I haven't checked).
We do have various ways to combat these issues; like any website of sufficient size, we have pretty complex methods of detecting problematic traffic and assessing the risk of any given request or session. However, no solution is perfect, and with the number of broken crawlers we see, some will inevitably cause problems.
To be clear, we can adjust our code and block them—that’s not an issue. The issue is that I have to wake up at 3 AM to do it, and even if it’s blocked, dealing with that traffic can be expensive. This guy got his $72k bill forgiven, but don’t expect the websites on the other end to be so lucky. (Yes, yes, ingress bandwidth is often free, but it’s never that simple. Scaling up? Bezos takes a cut. More database traffic? Pay the Bezos tax. Replication of enormous logs to other providers? Bezos hungry!)
Negligence is negligence. If you get in a car and drive recklessly without proper training, even if you didn’t intend to hurt anyone, you’re not going to get a lot of sympathy when you mow down a pedestrian. Likewise, I have little sympathy for people who face enormous bills for abusing powerful tools.
That’s not to say cloud providers don’t have billing problems. The delays are unacceptable, and the budgeting tools are often unintuitive or, as was likely the case here, outright inadequate. But in no universe was deploying code that spun up a container for every URL encountered a good idea.
Should such a mistake result in a $72k bill? Eh, probably not. I doubt this person will make the same mistake again, even with the bill forgiven. Or maybe they’ll just blame Google and attempt the same thing on AWS.
I noticed Cloudflare is doing the same but 1 level deeper with XDP drop: https://blog.cloudflare.com/how-to-drop-10-million-packets/
This isn't a mistake, the design is their business model. While we don't have a specific formal definition and name for it in the category of dark patterns, I'd like to name it "scumbag billing," where we got scumbagged.
It is — kid you not — recommended to terminate your individual services to avoid additional billing.
Technical limitation of a trillion dollar company? I say scum bag billing
Instead I use DigitalOcean where you have droplet limits that you can set, and the ability the pre-pay if you pay by PayPal, and never enter my bank card.
If anyone from DO (or another provider) is reading this, any chance of pre-payment from bank cards? After reading enough of these articles, this could really swing a cloud provider choice for a small company. (Pre-paid gift vouchers would be cool as well, give someone $10 to spend for Christmas).
It's too bad that DO only allows pre-payment via PayPal. (I recently lost access to my PayPal account due to them having a wrong phone number - and I can't login to correct it, nor have any contact for support.) I'd love to remove PayPal as a dependency, and pay DO directly from a bank - but not give them permanent permission to withdraw from it.
It's a clear dark pattern that GCP/AWS does not provide a simple way to prevent such unexpectedly huge bills.
I can attach a gift card to paypal and use it.
I tested the server for about 5 minutes and was charged a couple hundred dollars for "spinning up " the instance. Something the AWS sales guy assured me on the phone would not happen.
I still dont know why I didnt appeal I guess I know better than to try.
I've been pretty happy with Cloudflare, but at some point I added my credit card (silly me) and now I live scared of a DDOS costing me a lot of money.
If you're talking about something pausing service, maybe call it a billing cap for clarity.
In other words, the only way to access the "free trial" is to give a blanket promise to pay unlimited amount of money if something goes wrong.
There is no way I would agree to that, so I just closed the browser tab and forgot about the whole thing. That is, until this debate reminded me of it.
For an online service, implementing the cap should be quite simple, so if it is not available, I am going to assume this is intentional.
How many times do they have to do that? Because if it is a high number, they would be operating at a loss.
What other kinds of businesses or services let you run up a bill of tens of thousands of dollars and then say, "Ok, you made a mistake, you can take it back"?
Any educated guesses on what this compute might actually cost Google?
I assume they're able to do this because the fixed costs have mostly been paid for already and the marginal cost of the electricity, system wear, and bandwidth are negligible, but I'm not sure.
I'm not using the service anymore,
However, setting usage limits would be a solution for both companies and hobbyists. AWS could then calculate the maximum spending per month that is possible with the current settings. I bet they would never build such a calculator and the necessary usage limits because it makes it easier for customers to optimize costs.
But to be honest, it’s somewhat surprising that companies are willing to take on the risk of unbounded financial liability if someone makes a mistake.
Most AWS users would rather lose money than data and service for their customers, and bills are easier to negotiate than trying to recover your infrastructure.
The in-between approach is to create rate limits (either per sliding scale or total), which exists for some products but is probably too complicated to implement for everything.
Shutting off services can mean destroying the customer's data with no way for them to recover it. That could be from terminated ephemeral disks, or a terminated database, or cutting off a critical upload stream into their instances.
Its a lot easier to reduce/forgive a bill when a customer makes a mistake than to recover their lost data (or loss to their business).
I imagine they would have been more forceful if it was a larger bill.
Never used one for this purpose but since billing happens after the fact (and monthly), AWS won't be aware of the limit until after the monthly billing occurs. They’ll just tell you the card failed and you have a billing liability to take care of (and give you some time to fix it) while still letting you rack up additional debt with services running after the billing fails; they definitely won't cut you off when you've reached the level that would meet the limit on your card (and couldn't even in theory without realtime notification of other charges against the card, even if they were inclined to.)
What if the new services that are being suspended are writes to queuing systems that are used for order fulfillment or other business processes, should we drop these orders on the floor?
It's much easier to handle it post facto, and write off the expense on the cloud provider side, which doesn't cost them that much anyway. There are some guard rails that prevent people from doing catastrophic things that they can't write off (eg. taking all of the compute capacity of a region for hours on end, preventing other customers from actually using it) using limits that require manual intervention to be raised.
The rest of this is arguing against something I'm not saying, which is fine, but thinking about a solution is not condoning the problem.
Indeed, you can, and there are situations in which it makes sense. However, it doesn’t really help when it comes to detecting abuse of this sort. For one, CGNAT causes problems. There’s also the issue of people linking to articles from sites like HN and Wayback Machine. Those two alone make it nearly impossible to automatically rate limit based on an ID in the URL.
CGNAT is a big issue that Western companies tend to neglect. However, it’s increasingly common in places like India, and it’s even seen at times in the US, especially in rural areas.
And, of course, public VPNs are growing in popularity.
Unfortunately, all of these factors mean that performing any sort of risk analysis or rate limiting on IP address alone tends to be ineffective or outright harmful for moderately large sites. You can do some fairly basic categorization (this is from a residential ISP, this is from a datacenter), but beyond that, it’s not particularly useful.
Hypothetically, let’s say:
1. We tag every URL with an IP address association in some way.
2. Someone posts a link on HN.
3. We see lots of requests with IP address tags that don’t match the actual requesting IP address, so we block or rate limit them.
4. We’ve just blocked traffic from HN.
Another hypothetical:
1. We design, calibrate, and test a rate limiting system in the US.
2. Some large percentage of real-world traffic comes from India and is behind CGNAT.
3. We’ve just rate-limited most of India.
4. So we exclude India.
5. But now we’ve rate-limited Nigeria, and malicious traffic from India isn’t blocked.
What we actually end up doing is similar but mostly relies cookies instead, and it’s only a single risk factor. It’s not perfect, and it has some caveats that the URL solution avoids, but it has far fewer false positives.
I'm not sure that (4) matters for larger sites? Their rate limits are usually a drop in the bucket compared to the background traffic.
Generally, though, unless you screw up badly, submit forms, or blend in with a more problematic crawler, nobody’s going to care (or even notice).
You can cancel your credit card, but I imagine big Amazon and Co shops/sells their collections to the best on paper deal debt collectors (aka, truly the scummiest and worst ones).
If you have ever been pursued for unpaid debt like this, despite consumer legal protections, it is years of Hell, legal threats/letters, calls, and other gray intimidation. All while wondering if your credit score will just be nuked over night.
Source: Victim of identity fraud
Getting source proxy lists of high-reputation networks is just $$ and a simple API integration game anymore.
As for stuff like Luminati, if you’re being sufficiently sneaky, chances are you’re not going to snowball in the first place. I’m not sure why anyone would bother paying for Luminati to crawl sites like the one for which I work, but I have seen people use it to scam.
We can’t really be bothered to waste resources blocking well-behaved crawlers. Just keep it at a reasonable pace, respect errors (especially 429, but also 410 and 503), and ensure we have a way to contact you if things go wrong.
Frankly just any errors - if I see more than say 5-10 jobs fail within a 2-3 minute time period things are designed to wait X time, try again... and stop if they're still encountering errors and ping me to come in and investigate.
Faulty retry logic is just as dangerous as the forked/distributed run-off situation.
There is even a default 'task limit' they enforce which we had to increase by sending an email request.
Typically though, as in the article above, it's the database service scaling that causes big shocks.
Rule of thumb would be to always ask if there is an upper bound to every cloud service one uses.
But to answer your question (about AWS EC2 vs a DO droplet?), about other costs, you still have data transfer costs, which is currently:
AWS (for US East: Ohio):
Inbound: - first GB free - then $0.09/GB after (until 10TB, then you go to the next tier, paying a little less per GB.
Outbound - Well, I couldn't figure it out. The page was too complicated for me! (I think it's $0.01/GB? From this text: "Data transferred “in” to and “out” from public or Elastic IPv4 address is charged at $0.01/GB in each direction")
Source: https://aws.amazon.com/ec2/pricing/on-demand/#Data_Transfer
DO: Inbound - free
Outbound Free tier: depends on which droplet and how long you keep the droplet powered on for, but for the cheapest $5/month powered on all month, you get 1TB free. After free tier: $0.01/GB
Source for Inbound: https://www.digitalocean.com/docs/billing/bandwidth/ Source for outbound calculator: https://www.digitalocean.com/pricing/bandwidth/
Anyone who understands it better than me (especially the AWS pricing), please feel free to comment, I'd genuinely be interested to understand it better; with the way it's documented, I don't really understand it very well.
The way AWS complicates their pricing to the point where it's hard to tell what you're on the hook for just comes across as so... shady to me. I understand what they offer, and which problems they solve, I just don't personally like doing business with entities like AWS.
No, I'm not building anything that really needs the scale of AWS, and yeah I guess that invalidates my opinion of it to a certain extent. I'm just a stranger throwing their voice into the void for fun and to learn new things :P
Compare this to Hetzner where it costs EUR 1/TB and many server types include unlimited traffic at 1 Gbit/s.
I think the $0.01/GB you cited is region internal traffic which was sent through public IPs instead of private IPs.
it's true that outbound traffic does cost (and in theory if you do a lot of traffic), but the budget instances don't have that much bandwidth to start with. Also if you have a huge amount of traffic depending on what you do you might be better off with S3+cloudfront