The data rules worth $40k a day(tinybird.co) |
The data rules worth $40k a day(tinybird.co) |
Can’t think of a worse advertisement for their product.
After reading it, i was curious what the writer’s relationship was with TinyBird. Was he/she a recent user?
Then i put my palm over my face..
"If my product had 10,000 daily active users generating 100 events per day, I’d easily hit that size in less than a year ... If I built an application to visualize this result and refreshed it every second, I would spend $40,340.16/day on Tinybird’s Pro-plan"
There is no plausible business sense in refreshing the display of a year's worth of data every second, and even scaling back to a likely still-unreasonably-frequent refresh rate of once per hour you're down to just $11/day.
If your job is to slice&dice data all day, and can't be bothered to learn SQL, I don't know what to say.
Also it’s not unreasonable to see people spending $10K+ a day in Snowflake because bad practices just like this.
Learn SQL ffs.
oh my god
i'm currently $13k/mo in dynamodb costs because of this whereas the same requirements sql database costs $2k/mo
https://www.postgresql.org/docs/current/rules-materializedvi...
> That work towards incrementally updated views is happening and progressing. For now, it's a separate extension, though: https://github.com/sraoss/pg_ivm.
in my experience materialized views are critical for most large databases.
Materialized views wouldn't be a silver bullet, but it would certainly help by allowing us to "cache" all of the joins and pre-processing into indexable views.
I worked on a system to capture the production test data in a semiconductor company. We had trillions of rows and terabytes of data. While we were figuring stuff out, I'm sure I ran queries that scanned the entire dataset accidentally. I imagine one of those queries would have cost at least 1k to run. Our entire setup cost less than 10k a month to run on AWS regardless of how many queries we ran. I can't imagine spending 40k on 14gb of data regardless of what you were doing.
What? Even if I used SQLite on my laptop and queried this thing every second, I'd still use <$3 a day. Also, this platform has no concept of caching? Don't understand this post at all, total clickbait based on an inefficiency in your platform you really shouldn't be advertising.
So this product is not only expensive, but I have to think of the execution plan myself? Or I am wrong and modern dbs don't do that?
Query 1 -> join Table A with Table B, both have 1M records
Query 2 -> Filter Table A to 10k records, then join Table A with Table B (1M records)
I would expect Query 2 to execute faster - I don't think the exec plan would've optimized Query 1 equivalently.
I try to never underestimate the potential for someone to do something really stupid and I'm sure there are some egregious examples out there where a DB was set up and run such that outrageous charges resulted; but has anyone seen a situation in real life anywhere close to this kind of example?
Even if there was a instance where a poorly designed and implemented data set caused a $40K charge for a single day; I wonder how long it would take for the bean counters to notice and take action?
Learn from history, specifically, SQL. Or people who think they are too important to learn SQL aren't that important after all
remember, google only needs to be "kinda" CORRECT, unlike a relational db. try 'plaining that to management.
https://www.protechtraining.com/blog/post/the-value-of-value...
the stars you see died eons ago. special relativity is your friend. we witness "immutability" casually.
nothing in the relational model that i am aware of says that a materialized view cannot be a simple immutable table of "facts". same input same output. unambiguous.
simply use the dynamic queries to summarize the "facts" in, typically, a small mat view tables and your gui will never lie.
Personally I view a true document (not a table row turned into JSON) as being the deeply-nested kind, and ideally generated from the relational data itself, to allow different "dimensionalities" to be represented without needing pivots/windows/analytical queries, and that's very seldom what I see it being used for in practice. Again, most people just have a RDBMS row but stored in JSON.
example: in the "netflix" example, your movies, your actors, your users, your likes, etc are all relational, and then you build a document collection that is good for searching movies, a collection that is good for displaying user data/history/settings, a collection for displaying actors' filmography, etc, but all are generated from the same actual, consistent relational data.
I built a new general-purpose data management system that uses key-value stores that I invented to attach meta-data tags to objects. These key-value stores can also be used to create relational tables.
Because each table is basically a columnar store, I can map multiple values to each row key to create a 3D table. It seems ideal for importing Json data where any item in a document can be an array of values. I am trying to figure out how useful this system might be to the average DBA or NoSql user.
See a quick demo at https://www.youtube.com/watch?v=1b5--ibFhWo
The idea is you always have a relational "source of truth" and optimize that for OLTP, but also get the scalability benefits of documents/microservices/etc by having data already pre-coalesced/pre-digested into your correct format(s), so you're not doing complex analytical/window/aggregation queries on the RDBMS for every request. You run the analytical queries once, convert the result to json, and store that in the NoSQL.
Of course you still potentially have some "sync time" between the OLTP and the final commit to all the various nosql collections... unless you hold OLTP locks until everything is synced, which would be excessive. But this goes back to CAP and there's no magic wand for that - you can either put everything inside the RDBMS and take the performance hit, or you can have external nosql read replicas and accept the inconsistency due to the sync time, or you can hold locks until both systems are consistent at the cost of "availability" (updatability).
What are the advantages/disadvantages? Or what am I misunderstanding?
BigQuery in these cases has been very useful. Everything I throw at it returns in seconds. I can also do more text comparison queries that on a standard RDBMS would have required a full text index (which would probably be huge). But with BigQuery even those queries take seconds and I don't have to take production resources to view it.
If you need fulltext search, I've enjoyed using Sphinx and Lucene in the past. Is the column you want to do fulltext on 1 TB? 1 TB of RAM in 128 gig DIMMs is <$10k these days, so might as well get at least that much if you're running anything like the bills these guys are talking about.
If your data scientists can't do data science, then that's pretty bad