RabbitMQ 4.0(github.com) |
RabbitMQ 4.0(github.com) |
Too many people end up with their own half rolled pubsub via things like grpc, and they'd be far better off using this, particularly in the early stages of development.
It's an infinitely more friendly version of Kafka, pub/sub, etc that is extremely lightweight. Yet every environment trends towards Kafka because it was "chosen" by the big corps.
It's nothing to do with these apps being superior in any way - often it is the opposite - like reasonable faults and glitches can add more hours to bill the customers for "fixing".
"Key value", "pub-sub", "microservices can live anywhere" And?
Why do I care? What does "microservices can live anywhere" even mean and what does it have to do with NATS?
I find RabbitMQ sometimes inscrutable but I think even their website is better
The new Khepri metadata store in RabbitMQ 4.0 allows for even better scalability.
Just wondering what the biggest selling points are for somebody making a decision?
We needed features like job priorities, and complex routing logic and on top of that our software needs to send millions of messages a minute. What we miss is a nice UI to see the state of jobs, and monitor for problems more easily like: https://taskforce.sh/
> RabbitMQ is a reliable and mature messaging and streaming broker, which is easy to deploy on cloud environments, on-premises, and on your local machine.
What does that mean? "Messaging and streaming broker"? I understand the need for worker queues to process videos, images, emails and such but I can't easily tell if that's what this is.
Also, what are the benefits of this over just processing incomplete records straight out of my database? i.e. using MySQL as a queue.
Mainly throughput and latency. I haven’t used MySQL recently so some of this may apply more to Postgres.
Postgres has LISTEN/NOTIFY which helps with latency. I don’t think MySQL has LISTEN/NOTIFY, which means you’d have to resort to polling.
You have to use the `SELECT … FOR UPDATE SKIP LOCKED LIMIT 1` features to grab a message from the queue table, so multiple consumers don’t pull the same message.
The biggest issue, if you’re trying to achieve decent throughput, is dealing with bloat (empty pages still sitting on the disk that haven’t been cleaned up yet). You can run vacuum but an online vacuum will only mark the pages as available for reuse (doesn’t free up the disk space). And if you run a full vacuum (which will free the disk space) it locks the entire database while it runs the vacuum. This can compound if you’re using indexes.
One way of dealing with this is setting up partitioning by message timestamp, so that as old messages roll out, you can just drop those partitions and not have to deal with vacuum.
It can work if your queue needs are low throughput or can tolerate higher latency, but there are some things to manage, and realistically setting up a Redis instance is probably less complex than trying to work around the database-specific quirks, unless you’re already very familiar with your database’s inner workings.
If you are looking at RabbitMQ with "Maybe this is too much". Beanstalkd likely has features you need with almost none of the setup. Just don't web expose it ;)
I only used it on one project years ago and it was a pleasure, dead easy to get up and running and rock solid.
You have to read a lot of docs or you WILL hold RabbitMQ wrong.
The ideal RMQ cluster has 3 servers and is dedicated to just a few apps.
The biggest issue back then was finding a quality client implementation for the language you were using. Not sure what the status of that is these days.
Its unfortunate your team ran into performance issues, as Erlang can be inefficient in some situations. Were you using static routes on the DNS load balanced parallel consumers, or relying on the disk caching mechanisms?
I haven't touched it in years so I can't expand, but when I did, I had to write so many wrappers and add extra logic to use it properly.
Does RabbitMQ have equivalent features?
Reminds me, I'll have to check if they have a working donation link someplace. =3
BEAM languages (including Elixir and Gleam) share the benefits Erlang enjoys by also being part of the ecosystem.
I loved it and the only issues we had were due to our fuckups.
Perhaps people are still thinking in single point of ingress design paradigms. Admittedly RabbitMQ can be a pain to administer for Juniors, but then again so are the other options. =3
I can’t speak to the new version, but it comes with support for even more messaging patterns out of the box.
Earlier this year I tried setting it up as a websockets message broker in a Spring Boot app but failed miserably. I ended up using Spring’s simple broker.
Edit: FWIW, NATS also supports MQTT: https://docs.nats.io/running-a-nats-service/configuration/mq...
They didn't lose my respect for the removal of the argument, however, they lost my respect for whatabouting the conversation calling SemVer a "no true scotsman" fallacy, then trying to claim that removing a redundant argument is not a breaking change, and other reality-warping nonsense, before blocking myself and other complainants from their issues - and even deleting some of their own comments to mop up some of their own terrible reasoning.
I'm sure there is no love lost on their side, either. Personal rant over.
Well, at least AMQP 1.0 is now supported so I expect that for most things you are able to use any client now.
https://docs.celeryq.dev/en/stable/getting-started/backends-...
https://github.com/rabbitmq/rabbitmq-server/blob/main/releas...
I would love to be able to use a distributed log like Kafka/Redpanda since it's HA out of the box, but it simply does not fit that use-case.
RMQ immediately slows down (due to mnesia causing delays) and processes start dropping messages despite having system resources to grow.
I decided to add expiry policies to each queue so that the system cleans itself from stale messages and that fixed all the message dropping issues.
4.0 Changelogs state that they are switching to a new k/v storage (switching from experimental to default)
Mnesia stores vhosts, users, permissions, queue definitions and more. This is being transitioned to Khepri, which improves a lot of things (maybe most importantly netsplits) but not directly message speeds.
Edit: spelling
https://docs.nats.io/nats-concepts/subject_mapping#for-traff...
(for more detail check out this and the other YouTube videos on the Synadia channel https://youtu.be/WH55czo1BNk)
Yep, similar symptoms. (OpenStack's services are also written in Python, or at least were back then, so probably similar to Celery.) We had regular problems with RMQ restarting. (Unfortunately I can't recall if it was for OOM or just some BEAM timeout.)
A few hundred messages in a few dozen queues seem ... inconsequential. I mean whatever on-disk / in-memory data structure mnesia has should be able to handle ~100K stale messages ... but, well, of course there's a reason they switched to a new storage component :)
The parent's comment is right - that is a lot of money for not a lot of value, particularly when you are early stage.
The trick is in finding balance between paying ridiculous fees (relative to your revenue/customer base) to make things more simple vs. find another way and spend your time instead.
A prime example are identify provider services, such as Auth0. The free tier is good enough for development, but as soon as you expect to onboard customers the free tier starts to feel deliberately gimped. Are you willing to spend $20 a month just to use a custom login domain? For the zero customers you have? $20 a month might feel like "nothing", but it's $20 a month forever and it's $20 a month that could be allocated to other things, such as compute or your accounting software.
It's not always that clear cut, however.
If you are an actual early stage venture I don't believe those costs meet a high enough threshold to matter.
When you think about it, ideally most managed services have found some natural price for their services that helps make the above logic work. And if it does not, it might be actually overpriced. I suspect a lot of folks that say roll your own are undervaluing their time.
RabbitMQ is a "classical" message broker. It routes messages between queues. Messages are treated like little letters that fly everywhere. They're filed in different places, and consumers come by and pick them up.
Core NATS isn't really a message broker, but more of a network transport. There are no queues as such, but rather topologies of routes where messages are matched from producers and consumers through a "subject". You don't "create a queue"; you announce interest in a subject (which is a kind of path that can contain wildcards, e.g. "ORDERS.us.nike"), and NATS routes stuff according to the interests. So there's nothing on disk; and if a consumer isn't there to receive a message, the message is gone. Thus you can send messages back and forth, both point-to-point or one-to-many. NATS itself isn't reliable, but you can build reliable systems on NATS.
A common example of the lightweight, ephemeral nature of NATS is the request-reply pattern. You send out a message and you tag it with a unique reply address, the "inbox subject". The subject is just a random string (it may be called "INBOX.8pi87kjwi"). The recipient replies by sending its reply to that inbox. The inbox isn't something that exists; it's just a subject temporarily being routed on. So the sender sends a message and waits for the reply. NATS encourages you to use these ephemeral subjects as much as possible, and there can be millions of them. You can do RPC between apps, and that's a popular use of NATS.
JetStream is a subsystem built on core NATS, and is what you get when the designer of NATS thinks he can outsmart the designers of Kafka. JetStream is basically a database. Each stream is a persistent sequential array of messages, similar to Kafka topics or a RabbitMQ queue. A stream can be replicated as well as mirrored; one stream can route into another, so you can have networks of streams feeding into bigger rivers. Unlike core NATS, but similar to RabbitMQ, streams and their consumers have to be created and destroyed, as they are persistent, replicated objects that survive restarts.
Similar to Kafka, streams are just indexed arrays; you can use it for ephemeral events, or you can store long histories of stuff. Consumers can go back in time and "seek" through the stream. Streams are indexed by subject, so you can mix lots of types of data in a single stream (as opposed to multiple streams) and simply filter by subject; NATS is very efficient at using the index to filter. Like RabbitMQ but unlike Kafka, streams don't need to be consumed in order; you can nack (!) messages, or set an ack timeout, causing redelivery if acks aren't sent in time. In other words, JetStream can work like Kafka (where you always read by position) or like RabbitMQ (where messages are skipped once acked, but retried once nacked). JetStream has deduplication and idempotency, which allows you to build "exactly once" delivery, which is awesome.
Similar to how someone built a database on top Kafka (KSQL), the NATS team has built a key-value store on JetStream, as well as a blob store. They work the same way, through message ID deduplication. A stream is basically a bunch of database rows, with the message ID acting as primary key. So the stream acts as a primitive to build new, novel things on top of.
I think it's fair to say that RabbitMQ gives you opinionated tools to do certain things, whereas NATS and JetStream are a hybrid "multi model" system that can be used for more purposes. For example, you can embed NATS in your app and use it as a really lightweight RPC mechanism. You can use JetStream as a classic "work queue" where each worker gets a single copy of each message and has to ack/nack so the queue moves forward. You can use JetStream as a log of all actions taken in a system, with retention going back years. (NATS/JS is actually awesome for logging.) And so on.
We use NATS for different use cases at my company. In one use case, clients connect to an API to follow live, low-latency change events. For each such client connection, we register a NATS subject; then tons of processes will see this subject (and its settings, such as filters) and will all start send changes to that one subject. There's no single "controller"; it's all based on point-to-point and one-to-many communication.
(Full disclosure: I'm not familiar with newer RabbitMQ versions or the streaming stuff they've added, so it's possible that RabbitMQ has caught up here in some ways.)
>We use NATS for different use cases at my company. In one use case, clients connect to an API to follow live, low-latency change events.
Do you require any form of guaranteed delivery for these events? And what do you consider low latency?
I must say that rmq in k8s, is possible but hard to admin. It’s not a “toy”. But has great documentation. Will take several iterations to key in the right configuration for the use case.
NATS and Kafka can handle higher volumes on the same resources but IMO the use cases are different or you have to write lots of code app side to implement what rmq does with these tools.
NATS, especially in its most elementary form, is braindead simple, stateless, and ootb functional. If I only want a message broker/pubsub, I pick that. If I know on day 1 that I need queues or persistence, I would probably pick Rabbit over NATS’ offering (Jetstream).
NATS, especially with JetStream now, is a Swiss Army knife of messaging. It can do RPC, Kafka-type batch streaming, low-latency realtime notifications, large-scale network transfer, offline sync, work queues, mirroring, weighted routing, partitioning... it's incredibly flexible. The simple but powerful primitive you have around subject routing/mapping/transforms, stream placement, replication, offline lead nodes etc. are just fantastic. It's super reliable. It scales from tiny apps to huge ones. The CLI and API tooling around streams and consumers is also fantastic.
Everything just feels well-designed and sensible. The new key/value store and blob store (both built on top of JetStream as client-side abstractions) are also super neat, and we've started using this functionality.
$20 here, $10 there, eventually ends up as $600 monthly or more, and no customer anywhere in sight. That may, or may not be sustainable or make sense.
Flush with cash? Knock yourself out. Bootstrapping? You can spend that money a lot more effectively than just loading up on a bunch of overpriced SaaS products to make life easier. You have to earn the easy route by growing your revenue.
If its a mission critical queue for the NYSE that has huge costs for downtime? Sure makes sense to be more in control as long as your control has a measured impact of less downtime.
Keep in mind that NATS does not yet support value operations other than setting the whole value. Optimistic locking is supported, but NATS does not have inc/decrement, append, set members, etc. I believe such support is on the horizon, however.