118M Queries per Second on Neki(planetscale.com) |
118M Queries per Second on Neki(planetscale.com) |
That sounds unlikely unless those machines had access to crazy disk I/O. A local RAID 5/10 with 8 drives would still take 30-50 minutes just to read that much data. Even with a mid-range SAN you still would have spent 15-25 minutes just reading data. This assumes 7.2K SAS/SATA since SSD/NVMe were not ubiquitous in 2015, but even with 2015-era SSDs you're still looking at half that much time spent reading.
It’s cool to see the flip side of that argument. Planet scale is answering the question, “what does it look like when you DO fan out your workload to >100 nodes?”
There’s a place for both technologies. Very cool stuff.
Do you have the timestamp where they are talking about this? The claim doesn't pass the smell test for me. If you're talking about latency, then perhaps. On throughput, I don't understand how a single-node system could deliver higher throughput than a three-node system
RonDB now has support for infiniband, so ought to blow through the 1B ops/sec. For reference, that is 1 GHz of transactions/sec.
[ref] https://www.slideshare.net/frazerClement/200-million-qps-on-...
Neki 1 primary + 2 replicas: ~$5.0M/month (just the AWS bill)
Google Spanner w/ 3 replicas built in: ~$3.85M/month
Clickhouse is also on the right track of building some amazing opensource integrations with postgres, they have superior*[1] managed postgres looks like from their recent blog. I hope they do some OSS sharded postgres solution.
[1] - https://clickhouse.com/blog/benchmarking-nvme-managed-postgr...
Maybe I am wrong, but I am yet to read stories on operating tens of TB scale workloads on citus.
> 512 shards, each with one Postgres primary each on an r8g.16xlarge
> 480 Neki routers, each on its own 8xlarge instance
> We sustained 118,538,803 QPS for 16 minutes across 512 shards and 1.22 PiB of data. Our largest recording was 118,747,267.
Component Detail Monthly Hourly 16-min burst
---------------------------------------------------------------------------------------------
Shard compute 512x r8g.16xlarge $1.41M $1,930 $515
Router compute 480x r8g.8xlarge* $661K $905 $241
Storage (gp3 floor) 1.22 PiB @ $0.08/GB-mo $102K $140 $37
Storage (io2 floor) 1.22 PiB @ $0.125/GB-mo $160K $219 $58
IOPS (io2, light) 5K IOPS/shard, tiered rate $166K $228 $61
IOPS (io2, medium) 20K IOPS/shard, tiered rate $666K $912 $243
IOPS (io2, worst-case) 231,517 IOPS/shard (0% cache) $4.56M $6,251 $1,667
---------------------------------------------------------------------------------------------
Total (gp3 floor) $2.17M $2,975 $793
Total (io2 floor) $2.23M $3,054 $814
Total (io2 + light IOPS) $2.40M $3,282 $875
Total (io2 + medium IOPS) $2.90M $3,966 $1,058
Total (io2 + worst-case IOPS) $6.79M $9,305 $2,481Multiple live shards: yes
Query routing across shards: yes
Online shard splitting: yes
Zero-downtime resharding: yes
Multiple independent shard groups: yes
Data topology management: yes
HA / automated failover: yes
Multi-AZ: yes
Connection pooling: yes
Online schema changes: yes
Workflow-driven migrations/cutovers: yes
Zero-downtime imports: yes
CDC / logical replication: yes
Online Postgres version upgrade workflows: yes
Cross-shard transactions: coming
Multigres today:
Multiple live shards: no
Query routing across shards: no
Online shard splitting: no
Resharding: no
Multiple shard groups: no
HA / failover: yes
Multi-AZ: yes
Connection pooling: yes
Logical replication/import work: in progress
Distributed migration/resharding workflows: no
How it is an it's an alternative? Do you just say things without validating?
I'm sure it's a great product (it seems like planetscale do good engineering and the folks I know who use them seem fine with it) but I don't do vendor lock-in as a service personally, I'll use whatever employer uses because that's the deal but for personal stuff, well this isn't designed for that really, wrong order of magnitude on scaling.
512 shards, each with one Postgres primary each on an r8g.16xlarge
480 Neki routers, each on its own 8xlarge instance
That's ~250K queries/sec per router which seems lowish for this type of workload? The routers won't be doing very much (parse query, route it to proper shard?).
neki is still in platform preview, this experimentation allowed us to collect profiles at such scale and ship some nice optimizations, more are coming
I mean... What's the point of this "benchmark"?
No HA either, or backups I assume.(single AZ?)
But think the point of it was not to benchmark PostgreSQL but to demonstrate the linear scalability of the shared-nothing/stateless routing architecture. I think this was already known from Vitess though, which shares the same architecture. Also, as part of a new product release benchmarking shit like this internally is cool/interesting, why not blog about it.
The more cynical side of me says it was a money burning exercise to give the marketing team a number to tweet about. :)
In practice even NVMe SSDs at the time were maybe 10x faster for sequential R/W and maybe only 200-500x for random I/O. Most databases in 2015 were limited by serial dependency chains like walking a B-tree to figure out what nodes to read next and couldn't max out what an NVMe was capable of. Not to mention we were stuck with libaio in 2015 that made O_DIRECT ops synchronous.
So the OPs claim of reading multiple terabytes of data in seconds on AWS in 2015 literally wasn't technically possible.
SSDs simply were not as ubiquitous in 2015 as they are today, and I didn't know anyone using them outside of mission critical systems were the cost made sense.
The biggest problem of Citus was migration effort, transition from single node to multi-node was not trivial. Here I’m not talking about single table use-cases, more classic relational, multi-tenant apps with 100s to 1000s of tables. This is partly expected with most sharding technologies, though.
Sharing some insights based on my multiple years of experience working with Citus!
Here are few customer use-cases I could found:
https://docs.citusdata.com/en/v10.0/get_started/what_is_citu...
https://info.citusdata.com/rs/235-CNE-301/images/Citus_Data_...?
Schema changes need locking... everywhere. Citus is no different. And they need proper design everywhere. If you mean that it requires distributed transactions, well, yes, again: expected and solved. Not even all sharding solutions support this.
> coordinator node
Not sure what the problem is here. If what you mean is that a single coordinator, even with an HA replica, can saturate, that's true, but you can add multiple "query routers" (that's our name in StackGres, see [1]).
> Maybe I am wrong, but I am yet to read stories on operating tens of TB scale workloads on citus.
For example, we have a customer that ingests some 30TB/day, and it's ramping up towards 200TB/day of ingestion. On 24 worker nodes.
[1]: https://stackgres.io/doc/latest/administration/sharded-clust...
However, in general i would not trust a product that is not open source (foss).
I do like what grafana has been doing for example, with licensing options at the two extremes: gpl/lgpl v3 on one side and proprietary hosted saas version on the other.
We run our own grafana/lgtm stack at work, we’re outgrowing it and we’re evaluating the saas offerings.
But then again, not criticising your work and product, you know better than me about it and about your business, just writing down my proverbial two cents.
I get that PlanetScale has the currently superior product, and I listed the closest thing to it. You're right it's not ready, but in the case that the parent had mentioned of closed-source making it a deal breaker, it's the best alternative.
Neki looks like a good product, and I trust that something coming from the folks behind Vitess is going to be a damn fine offering, but some of us don't love tying our workloads to fully proprietary products. I wouldn't even want to host something like this myself, but knowing there's a path out of the proprietary hosting (which we've seen time and time again is something we should plan for) is critical for a lot of us when choosing solutions as important as our data layer.
If I needed a truly sharded PG at the moment, Neki is probably the best option I could find.
As for the "alternative" argument specifically, it's not on par at the moment, you're right, but that's their goal and they're attempting to fill the same void. In the case of someone who has a requirement their solution be open source, then Neki is flat out of the running too.
I'm rooting for a good Postgres sharding solution to come out. Your team appears to be doing some good work. I would also like to see where both Neki and Multigres are in five years.
Anyone can start a project and say it's an alternative until it achieves the same results is not an alternative. They have raised 10x more venture funding than my company, and we have out executed them with hard work. I will absolutely defend that work.
Supabase hire companies to write fake AI slop to slander competitors. https://dev.to/sivarampg/how-a-neutral-supabase-vs-convex-co... They also claim ownership over projects they dont contribute to https://x.com/jrdntgn/status/2022003873171620160 https://news.ycombinator.com/item?id=45321794 (they have no vitess maintainers on staff now and they did not then either)
This is who you are carrying water for.
Not sure how I wrote your product off. I spoke about how it wasn't open, which it isn't.
> Anyone can start a project and say it's an alternative until it achieves the same results is not an alternative. They have raised 10x more venture funding than my company, and we have out executed them with hard work. I will absolutely defend that work.
Good on your team. I believe the animosity here is mostly in our differing opinions of the word "alternative".
> This is who you are carrying water for.
I am not "carrying water". I listed another company. I've never used a Supabase product.
All this because I mentioned something that was open and in the same lane as your new product.
How's this: I won't refer to it as an alternative because it isn't up to at feature parity yet. That's fair. I will however say that you've taken multiple angles unrelated to the original core of my first reply's parent that was that Neki is not open source. That is true and many of us will see it as a downside when chosing a product. That isn't bad. That's just a thing that is.