Thanks for your interesting POV on the Redis/Kafka intersection on this. What could be interesting is that while Redis Streams are certainly totally inspired from Kafka streams, it's just a conceptual thing, so the two things can act in very different ways in practice.

For instance just imagine that I never touched a Kafka system in my life, never used it, don't even know the API, I only read all the documentation they have on the site about the design, to get the higher level picture and combine this with my own ideas about fixing the fact Redis was lacking a "log" data structure.

Pub/Sub + other data structures were not able to provide time series and streaming, but yet Redis streams remain an ADT (Abstract Data Structure), while Kafka is a tool to solve a very specific business case. So the applications have some intersection, but are also very very apart.

For instance you can create infinite small keys having streams, so Redis is good for many IoT usages where you receive data from many small devices. Redis Streams also stress on range queries, so that you can combine the time series with millisecond-range queries.

However, yes, the fact that I added also consumer groups is a way to put this "80% streaming" into a more usable streaming systems more similar to Kafka, for the use cases where:

1) The memory limits.

2) The speed.

3) The consistency guarantees of Redis make sense.

However at the same time, it was a great challenge and pleasure to do what I always try to do, that is to create an API for developers thinking like I'm designing an iPhone, and not some terrible engineering thing which does what it should but is terrible to use (I'm not referring to Kafka that I do not know). So I really hope that what you say "easy to setup and use" will be what developers will feel :-)

Thank you, I believe it's my fault that many potential users remained wondering. Sometimes I forget that the world is not inside Twitter... and I should instead blog more and tweet less, both for myself, because writing a blog post gives me much more sense to accomplish something, and for people interested in Redis that will find information more readily, and also searching via Google and so forth.

There are certainly "client oriented" (as opposed to webhooks) push APIs out there. However, such APIs that let you specify a starting position to read from are rare.

Sometimes APIs will give you tokens to use for resumption (e.g. SSE event IDs, or any long-polling API), but typically these are for a time-limited session rather than a stateless query against any point in a long-lived log.

Years ago, services like Friendfeed, Livefyre, and Convore had stateless long-polling APIs that returned a log of data, I believe. These kinds of APIs seem to have fallen out of fashion, though. There are still stateless long-polling APIs, but most of the ones I'm aware of don't return logs of data. For example, Dropbox and Box will let you query for a change notification against a starting position, but then you have to fetch the actual data separately.

That said, just because streaming APIs that let you set a starting position are rare doesn't mean they're impossible to make. My company (https://fanout.io) has built tools to help with this.

Edit: since you asked for a real example, Superfeedr is one such API: https://documentation.superfeedr.com/subscribers.html#stream...

Did you have issues with Kafka? Curious to hear because I’m about to try and start using it for something at work.

Redis Pubsub is fire-and-forget, so if you aren't listening when a message is fired, you'll never receive it. Redis Streams store messages, so you can connect and read all the messages since you last checked. It's a similar model to Kafka.

This article from Brandur does a great job of explaining a use case for Redis Streams [0]. As part of that he begins by explaining what they are, how they work, and a bit about how they differ from Kafka.

[0]: https://brandur.org/redis-streams

Redis streams will allow for direct modeling of time series data. For example,

     redis-cli>XADD AAPL 1516899637000.0 open 221.25 high 222.1 low 220.90 close 221.50 volume 1121223234234

(All data is imaginary.)

Found the rationale for this: https://github.com/redis/redis-rcp/blob/master/RCP11.md#rati...

Hi, since you're here just a quick question not really worth a Github issue - what is the reason for the 1-second resolution in the DELAY parameter in Disque and will that ever get more fine? We currently use Bull as a job queue on Redis and the delay is a key and visible part of our application, so that alone kind of eliminates Disque as a consideration for us.

Love that you are using sem version properly. Too many folks couple marketing reasons into their versioning schemes as opposed to simply, “did I break backwards compatibility?”

That sounds like a good plan, thank you!

What I loved about Disque is that it was the magical distributed systems thing that everybody really want for messages safety, but done in a way totally transparent for the user, and it actually worked well enough that there are people keeping using the RC1 for ages... That's why I want to resurrect such a project. The fact that it is multi master, auto federated and auto balancing, and all the auto stuff happen without any user stress, was kinda a good thing. As a Redis module everything will be the same, but, without all the code duplication, because otherwise Disque is like a 5000 lines of code project or alike, if not less.

Chapeau bas. But apart from my rather obnoxious joking - how the whole Stream and GIL interoperate. I mean, this is technically a complex problem.

great response

lol

It was meant as a joke.

Alas, I have been more than once bitten by poorly implemented GIL and in my personal experience any GIL is problematic in a high-throughput environment. It simplifies the problem of implementation of environment or API for the price of introducing global lock into the system, thus eliminating any possibility of lockless design. And we know it's doable.

I spent years trying to work around GIL in production environments and we have the tools about now to do it reliably. We have on_commit in Django, we have Celery with proper support for mechanisms like chords and a myriad of others.

I do get that Redis has a different approach to many things. But here's the deal - every time I had to rely on Redis as a critical component, there was a problem. Either there was that "one gotcha" in the config or I didn't understood something or else.

A while ago I was a part of discussion with the author of Redis on using Redis as cache. And quite some people, not only me, noted they had bad experience and their benchmarks didn't show Redis is better than Memcached. And, I think that's ok. Unless you pose your product as a competitor to memcached. Would I use memcached-db? No, it's a bad idea.

My problem is that Redis originally was a key value store with datastructures. And that's a great idea and not that many services let you model datastructures in a distributed way. I actually implemented a distributed system of progress control for a very critical piece of architecture at my current workplace. I'm a Redis user since 2014 in regard to commercial products.

My point of view is probably largely skewed but I don't understand and I never seen that explained why Redis tries to do everything. We have battle tested Kafka for streams, ElasticSearch and Solr for searching and Memcached for caching. Can't we get a super reliable data store that supports data structures? A data store that is _able_ to utilize the fact we have more than one core?

I know my view is unpopular, but I don't see addition of GIL as a genius idea. I simply don't see a coherent direction for Redis development.

BTW: is there a benchmark comparing latest Redis without the GIL and the latest with GIL?

Had to re-read a few times, but the comparison just seems like a poorly worded attempt at a joke/insult.

And an extra note when someone says "it's hard to implement ... without locks and so". Look at Firefox. I dumped it's bushy tail around 7 years ago when switching to the apple-scented world for good.

I came crawling back. Why? Because people at Mozilla spent their time working on their product, finally coming to agreement that the fact your product is OS doesn't mean you can cut your standards in half focusing on features you like instead the ones that actually make it better.

Here's the link in case someone missed it https://hacks.mozilla.org/2017/11/entering-the-quantum-era-h.... I'm not a Mozilla fanboy, to be frank for a while foxy browser scored lower on my "software like scale" than Redis.

And they won me over. In 5 minutes. And it's been proven time and time again that performance is a feature.

p.s. Now that I write it I realize that for certain reasons I should just try to write the damn thing and instead of writing pointy comments let antirez rip my software apart :)