Reducing BigQuery Costs(blog.peerdb.io) |
Reducing BigQuery Costs(blog.peerdb.io) |
Has anybody else had this experience? Or are we just doing it wrong?
This is not intended to be a rant, just curious.
The best way I've found to approach it is to treat GCP as something that has to be evaluated at an individual service level. It's great if you're on one of their expected workflows/golden paths, and you can get lucky with a good fit if you aren't, but they seem to have a lot of unspoken assumptions and limits baked in that might or might not align with your use case.
Disclaimer: My use cases are pretty unusual from talking to our account rep, so this might be over-fitting to weird data.
But really soon I noticed the slow startup … simple queries took too long (1.2 sec vs milliseconds in a traditional database)
Then I learned a lot about BigQuery views. That helped a little.
At some point I simply wanted to export data. New Google tools needed to be learned: Cloud Storage, Data Flow.
After 18 months of using BigQuery on roughly 850 million rows, I switched back to a traditional database.
I’ve never used Pub/Sub or Cloud Run, but have been quite happy with BigQuery and GKE.
We found that all of these had significant caveats that required careful planning. We had a few instances of runaway AWS costs due to basically not knowing enough about AWS and we had to be careful to only use the "good" AWS products, Digital Ocean never had runaway costs but they did keep turning off production services because our use-case was not one they were familiar with (dev machines, off-site backups). Bare metal was a minefield, we found we couldn't reliably run Prometheus because it ate SSDs. As for GCP, it did require understanding the pricing and it was possible to shoot yourself in the foot with things, but no more than anything else.
There are going to be gotchas everywhere. Overall we had a great experience with GCP, to the point that the company has remained on GCP post-acquisition by another company who were mostly on Azure.
But I do agree, there are some gotchas. PubSub examples: Duplicated messages, shitty DLQ implementation (in my opinion), some developers had improper error handling which lead to to infinite resends of messages (because they nacked message on error), etc..
However, I think the scaling and setup weighs up for all of that. You just need to specify a topic and subscription, and then you don't really have to care about resources or scaling at all, and that is SUPER nice. Also, PubSub is stupidly cheap in comparison to any other similar product, at least that I know of.
BigQuery has two completely separate pricing models: on-demand, and slot-based. The two models bill on completely separate things:
* On-demand pricing bills per GB of data processed by your query.
* Slot-based (or editions) pricing allocates your project a number of CPU/memory slots, and you pay per CPU-second.
You can find the costs each query would have by looking at the total_bytes_billed and total_slot_ms columns in the INFORMATION_SCHEMA.JOBS_BY_ORGANIZATION table, and multiplying those values by the slot-ms cost (total_slot_ms * 0.01111) and the bytes-billed cost (total_bytes_billed * 0.0000059). Then you can go through your queries and allocate them to either on-demand or slot-based pricing, depending on which is cheaper.
Usually slot-based is cheaper, but queries that do a lot of internal joins can have really huge CPU costs but cost very little in on-demand if they're not reading a lot of bytes.
Somewhat annoyingly, these billing models are configured at a per-project level; you can't switch between the two of them in a single project. Fortunately, you can query tables from other projects easily.
Rule #2. BQ is amazing for being able to churn through and analyze massive amounts of data, and can very well be the best option in some use cases.
Rule #3. Letting "just anyone" run queries is as dangerous as casually handing a credit card to your drug-addicted cousin. Just wait until you get the bill!
Rule #4: Partition and cluster your data wisely. You don't have indexes.
Rule #5: Duplicate data. Throw all of the normal forms out the window. Storage is cheap, computation is expensive.
Rule #6: BQ is not meant to be used like MySQL. It's "spin up" time is too slow, but you would be hard-pressed to beat its performance on truly large data sets.
My perspective: One of our customers has a database growing by 17 gigs a day. One of them. There's several on the same scale. Yes, it's necessary. Another instance: One of our customers spent $8k in one month because limits were not properly placed on the account and we didn't catch it until the bill came. We monitor better now. A different instance: We had a dev trying to optimize a query, and they spent $250 in queries to get the cost down from $50/query to $15/query. Most of the time, though, our queries are only pennies.
Now that I've written all of this out, I feel like I need to record a video about it. There's not a lot of BQ info aside from the marketing fluff put out by "teh Google".
Ditch the cloud. Go buy a bunch of fast computers. You’d be surprised at how fast modern computers are.
Re-learn the ancient art of running your own systems…. careful, all the naysayers will tell you that no company can run their own systems, to try would be crazy!
It has its own gotchas and quirks, but cost-wise it's not even comparable.
Thank you! Glad it helped you.
ClickHouse is in particular well-optimized to take on real-time analytical workloads from BigQuery - both functionally and from a cost perspective.
For anyone wanting to try it yourself, there guide to follow when syncing or migrating workloads from BigQuery to ClickHouse, with some benchmarks and back-of-the-envelope cost comparisons: https://clickhouse.com/blog/clickhouse-bigquery-migrating-da...
There is a place for BQ but it is good to set expectations correctly and also look at the constraints. They are sometimes not obvious. The docs do helpfully outline the limitations, for example:
- Materialized views: https://cloud.google.com/bigquery/docs/materialized-views-in... - Indexes: https://cloud.google.com/bigquery/docs/search-intro#limitati...
1. Google Big Query is super powerful, but it's straightforward to shoot yourself in the foot and end up with a huge bill. I would agree that Big Query shines for advanced users. For regular, it's better to use tech that can't be misused by business analytics that just try to get their job done with BI and SQL.
2. I wish a lot of tooling provided better defaults and suggestions to use it efficiently.
Who would have thought?!
A bizarre conversation I witnessed between the BigQuery team and my company at the time (a major customer):
Company: "We need to be able to see our consumption and a breakdown of our bill"
BQ team: "Oh, yeah. We can see how that would be useful. We should probably build that..."
Like, this was a GA product without any thought given to self-serve billing visibility.
I realize billing is usually the last thing bolted on, but I'd expect some basics to be in place before the product ships.
1. for some it will make sense to move to pricing based on CPU time per query vs billing on scanned TB of data. This can be done through the commits in UI.
2. there is option to have storage billed on logical or physical bytes. If you have data with a lot of duplication (enums, customer ids etc) then physical billing can be a lot better option. Last I looked this was only available through CLI setting in dataset and you may need to ask Google to include you in their beta. We lowered our billing with 30% for storage.
I try to keep an eye on GCP release notes to find things like the physical vs logical billing.
Used this to power a +$30M revenue affiliate platform tracking.
My record is $20k and it raised some eyebrows. But it was not really a mistake, just a sub-optimal backfill.
The data was filling a need for making appropriate business decisions, and compared to all the money lost by business developers making investments on a hunch, this was a very small bump in the road.
When you put a query in the BigQuery console, it'll tell you "This query will process ??? MB when run" at the top right.
So if you code all your queries interactively in production (which is what everyone else is doing anyway) it's not too hard to keep an eye on.
Note that this is not the default! :-)
It was originally estimated like 10k or something per test which was approved at the time (had like 3 level of management all down my neck for getting it out, hence using lambda originally).
We did deliver, just needed one more sprint to rewrite it as a distributed system on servers. ;) Moved to like 20 machines w/ 128gb of ram that we could spin up as needed (testing millions of events a second, system in NodeJS!)