RabbitMQ 3.11 Feature Preview: Super Streams(blog.rabbitmq.com) |
RabbitMQ 3.11 Feature Preview: Super Streams(blog.rabbitmq.com) |
I've worked in 3 places where RabbitMQ has been a fundamental cornerstone of the architecture, and while it does require a little tuning around performance occasionally (generally because it's being used inappropriately or without full consideration of its limitations/best practices), it's rock solid, easy to debug/inspect, has an active and supportive community, and is generally just all around pleasant to work with and maintain.
Kudos to RabbitMQ and its developers!
As an aside, the addition of recent features such as super streams, streams and quorum queues make it a compelling all-in-one tool for solving a bunch of architectural based concerns/requirements in application and infrastructure development. I've often thought about why it's not more utilised in the ops side of the world for metrics gathering and other usecases. I have also wondered how useful it'd be for log ingestion with lazy queues etc.
Does anyone out there have examples of unusual use cases for RabbitMQ where it's outshone some alternative product? Would love to hear about them!
Honestly though my only experience with RabbitMQ has been as a backend for Celery (background task processor for Python) and I think my real issues are mostly to do with how Celery uses RabbitMQ in a very poor default setup.
Message confirmations are off by default and turning them on caused our queues to grind to a halt. Queues being single threaded and clusters don't much help with that without using some kind of sharding plugin. It seemed like getting to a good spot required a lot of arcane knowledge that wasn't so easy to find.
"Configuring RabbitMQ to be a performant message queue for background task systems" would be an excellent blog post I would share widely!
This also makes online upgrades extremely hard, the only acceptable way is to stand up a new cluster, switch producers and consumers and then shovel the data from the old cluster (which is also not too reliable).
They came up with quorum queues which have less features and require to keep all messages in memory. I don't like having servers with 90% unused memory for that one event where I actually need to queue a lot of messages because a consumer is broken.
I would never pump logs through RabbitMQ, if you get into a situation where you accumulate a large amount of data in a queue you will face trouble sooner or later. Most likely RabbitMQ will run out of memory, will "flow-control" producers and you'll have an outage you can't recover from.
I don't think this is true
> Quorum queues store their message content on disk (per Raft requirements) and only keep a small metadata record of each message in memory. This is a change from prior versions of quorum queues where there was an option to keep the message bodies in memory as well. This never proved to be beneficial especially when the queue length was large.
https://www.rabbitmq.com/quorum-queues.html#:~:text=Quorum%2....
At low volumes yeah sure, it just ticks along, and does what you want.
I only used it at one company, but it was the most unreliable and hard to diagnose piece of our infrastructure. We didn't even have high throughput and I wouldn't use it again without a really good reason.
At least you can tell Kafka to block producers if replicas aren't clean.
They go into queues based on attributes of the messages ("routing keys") matched against rules you set up ("bindings").
Other things pick up messages off of those queues and process them.
When they're done, they acknowledge the message and it's removed from the queue. If they crash, the message doesn't get acknowledged and it goes back to the queue.
What I Wish Someone Would Have Told Me About Using Rabbitmq Before It Was Too Late
https://ryanrodemoyer.github.io/what-i-wish-someone-would-ha...
It’s a good feature, there’s nothing wrong with that, but there’s nothing wrong with saying “we’ve brought this feature to RabbitMQ too” rather than trying to pretend it’s totally-not-the-same.
> Let’s talk about the elephant in the room: how does it compare to Kafka? We can compare a super stream to a Kafka topic and a stream to a partition of a Kafka topic. A RabbitMQ stream is a first-class, individually-named object though, whereas a Kafka partition is a subordinate of a Kafka topic. This explanation leaves a lot of details out, there is no real 1-to-1 mapping, but it is accurate enough for our point in this post.
I think that's fine. This is a post about a new feature in RabbitMQ, from the maintainers of RabbitMQ. It's not intended as a RabbitMQ-Kafka comparison post.
They acknowledge that it's equivalent to Kafka, and then chose not to spend time digging into the details of how it's different - I'm willing to take their word for it that there are all sorts of interesting technical differences here, and I'm fine with them not addressing those in the post where they announce their new feature to the world.
I don't see this as them pretending that it's totally-not-the-same.
What's the general practice for multi-tenancy? Say we have millions of clients with thousand being added daily and for various reasons, at least for _some_ message types, we'd like to keep them separate (for example, maybe strict ordering is super important, so we can't throw messages away, but we don't want a poison message to impact all customers).
I should find time to run some tests with quorum queues, this now actually looks usable. But in the end we will have to see how stable it is running production workloads.
other log-based messaging technologies like kinesis, kafka, etc do not care if a consumer goes down & are thus much safer.
However, adding another Kafka or Pulsar node would have been much easier.