Even the most fundamental information like available configuration options, command-line arguments, deployment information and so on is missing.
Looking at the code it appears that you need Consul, Zookeeper and Redis to make this fly and the docs don't mention this anywhere. They (barely) explain how to run the SQL proxy on a local machine but thats it.
I wonder if the testimonials on your website are just pulled from thin air, I don't think any sane person would even experiment with this anywhere near production environments.
We've been in pretty heavy development and have been heads down on getting ReadySet into your hands as quickly as we could. We'll be doing a major documentation pass soon which will have more info about clustering, etc.
There's also a bit more detail in our development guide - see https://github.com/readysettech/readyset/blob/main/developme...
> need Consul, Zookeeper and Redis to make this fly
A hard dependency on 3 key value stores?
That part of the blog post refers to ReadySet having a SQL interface, rather than a key-value one.
> It can serve millions of reads per second on a single node ...
I'm not a network guy but that seems just astonishing - what is a 'node' here?
> ReadySet incrementally maintains result sets of SQL queries based on writes to the primary database.
So basically you've solved the general materialised view incremental update problem? That's an unsolved problem in general, surely?
Edit: not dissing but trying to see where the limits are.
It's not. Materialize (my employer) incrementally maintains views too, using tech (Differential Dataflow) that has existed for almost 10 years: https://cs.stanford.edu/~matei/courses/2015/6.S897/readings/... .
ReadySet is based on Noria (Jon Gjengset's Ph.D thesis, explained for non-experts here: https://jon.thesquareplanet.com/noria-in-simpler-terms.pdf).
Taking a research project and making it into a production-ready product is hard work -- congrats to the ReadySet team on the launch, and best of luck!
CREATE MATERIALIZED VIEW rental_by_category
AS
SELECT c.name AS category,
sum(p.amount) AS total_sales
FROM (((((payment p
JOIN rental r ON ((p.rental_id = r.rental_id)))
JOIN inventory i ON ((r.inventory_id = i.inventory_id)))
JOIN film f ON ((i.film_id <> f.film_id))) -- tweak
JOIN film_category fc ON ((f.film_id = fc.film_id)))
JOIN category c ON ((fc.category_id < c.category_id))) -- tweak
GROUP BY c.name
HAVING sum(p.amount) NOT IN (196, 203, 791) -- tweak
ORDER BY sum(p.amount) DESC
There was a cool article about intercepting the Postgres connection with Elixir not long ago: https://docs.statetrace.com/blog/build-a-postgres-proxy/
If you want to dig-in more, hop into our community slack: https://readysetcommunity.slack.com/
[1]: https://www.postgresql.org/docs/current/ddl-rowsecurity.html
I've played with Safepeak (1) which runs on Windows Server. It was sold later to an Israeli company (2), which have since gone out of business and assets ended up with another company and now sold as ScaleArc (3)
The original SafePeak is available free but no maintenance or anything, so not really production ready. It works, as tested in a test environment but eight years without support or updates...
(1) http://www.safepeak.org/ (2) https://en.wikipedia.org/wiki/SafePeak (3) https://www.devgraph.com/scalearc/
Great that someone is productionalizing this!
Btw, is Jon involved in ReadySet?
[1] Partial State in Dataflow-Based Materialized Views - https://github.com/mit-pdos/noria
https://twitter.com/jonhoo/status/1511401461669720068 https://readyset.io/blog/introducing-readyset
It has a global edge network, so no infrastructure to deploy and AI managed cache and auto invalidation, so no cache configuration needed.
I don't see the point in using an extra app - you can do this natively in Postgres. Materialized views. https://www.postgresql.org/docs/current/rules-materializedvi...
And finally, ReadySet's (Noria's) big innovation is that the result set can be only partial, storing only the elements of the result set (and underlying data flow graph) that are frequently accessed, instead of the whole result set like a materialized view would.
Is this really the case that queries can't be cached on traditional databases?
For example, MySQL deprecated their query cache, but previously it would only cache until there were any writes to the tables that the queries were referencing https://dev.mysql.com/doc/refman/5.7/en/query-cache-configur...
Even CochroachDB doesn't do query cache, only query planning is cached. [0] [1]
[0] https://www.cockroachlabs.com/blog/memory-usage-cockroachdb/
[1] https://www.cockroachlabs.com/blog/query-plan-caching-in-coc...
You can read more about how it works here: https://docs.readyset.io/concepts/overview
I'm curious what might be pathological cases, patterns of query watching and updates that give the cache a lot of work to do to keep up
Do you have a sense for when people can try it? Most of our app is reads and we're using Rails + Redis and it's fine and sometimes a pain. Would love to try it.
> ReadySet is licensed under the BSL 1.1 license, converting to the open-source Apache 2.0 license after 4 years. The ReadySet team is hard at work getting the codebase ready to be hosted on Github.
The same query could produce wildly different result based on the user issuing them, and the caching somehow has to take that into account. Is that something you address?
https://people.csail.mit.edu/malte/pub/papers/2019-hotos-mul...
I will answer that for sure later today when I’m back at my desk.
If you changed that back to an equals sign, yes, we could incrementally maintain your query.
I originally had added the HAVING clause to nastify it further, because this would cause values to appear and disappear, so to handle that you (probably?) have to materialise the entire result of the GROUP BY before applying a HAVING. Which is doable could cause some overhead.
I guess I could see it work for INSERT-only tables, with a lot of headache, but throw in UPDATE and DELETE and it could become awful.
Also we have to agree on what 'incrementally' means :)