I don’t want to be dismissive, but I often see these big numbers being posted, like “14M messages” or “thousands of messages” and then adding “per year” or something, which brings it down to toy level load.

Even the first “serious” example is about “thousands of messages” per minute. Say 5K a minute. That’s 83 per second, say 100. That seems .. not that interesting?

Am I being too dismissive? I think I am. I am not seeing something right. Can anybody say something to widen my perspective?

Looking at the volume though, kafka is overkill. They most likely could have just used the database and reaped the benefits of doing everything in a single transaction, with easier row level locking. The post acknowledges this.

I do think it highlights the need for a small scale kafka, though. It's conceptually great to have everything work off of logs, but kafka does add a non trivial operational burden.

Admittedly that database had some complicated queries with lots of business logic to get a useful view on the data. But at first I hoped Kafka would somehow make this easier, but of course our particular usecase with a low event volume (hundreds per day), high latency tolerance (next day reporting was considered good enough), highly complex business logic (various computations that required knowledge of what was done previously) all made Kafka just about the least suitable tool for the job.

Of course the contract was already signed (I was naturally never consulted up front), so this resulted in lots of solution looking for a problem. No suitable problem was found so I ended up leaving enterprise world for a scale-up and the CIO is still doing whatever he wants for god knows why

So in our case it's clearly not an anti-pattern, but the right tool for the job.

That being said, if you are not yet in a situation as complex as the one your tool is designed to deal with, there is a very good chance you will waste some time starting to use such a tool "early." You might get that time back later when you scale, you might have the right people to set up the complex tool the right way for your simple situation, but you are taking a bit of a risk. As long as you go into the situation with your eyes open I think most people end up ok. The horror stories almost always come from people who are working to fulfill needs they do not have and don't understand why their work isn't giving good ROI.

    Immediately starts to doubt OP's assumption/implementation of the "where it works great"

Then again am sure there maybe reasons/context/constraints unaware to us.

Eg - the ingestion is spiky, with the possibility of all 7500 in a few secs/minute, you would want to first make sure that the http traffic can scale before getting to the point where it can actually connect and push to the queue

Possible reason#2 - an intern who just finished up their first Kafka task; just got freed up

#3 - or this was the only infra available and a choice had to be made with the time available at hand

#4 - or this was an experiment to see for yourself

A majority may not agree with your view, but none of us really are in your shoes. So I applaud you for sharing your thoughts anyway.

Why do so many engineers end up having such a struggle with an event sourcing system, yet the system itself remains highly popular I don’t know. I theorize the following:

- Its flexible enough to do things like receive events (messages) and sending downstream events from those received

- it can ingest events fast. A well tuned instance is very fast and can handle a lot of volume

- it often allows a middleware log point (or other work types) for things happening throughout your whole system

Perhaps all of these things (and more) are hard to attain using a different technology

For example, if you use Kafka or streaming solutions like Flink or Spark, you should change your mental model to (possibly large), (possibly resplayable) streams of events and look for simple ways to get these event streams going and good ways to consume them. And then you need to let the design push you where it wants you to go.

Like, at work, we recently had a discussion how it was so storage-expensive for a project to store all events of a day and how the query to count all of these events per tenant was taking so long. While they are using a streaming event processor in front of it. Like, what the hell - think in streams, tally up these events on the fly and persist that every hour?

Also Avro is great but like Kafka you were probably holding it wrong.

I do prefer Protobuf in these particular scenarios as Protobufs features more closely align with svc <-> svc RPC style communication patterns while Avro shines in longer lived scenarios where messages need to be archived and you don't want to come up with your own framing for your Protobufs.

This is because Avro has the Avro Object Container Format which is a simple block based file format, which allows for relatively efficient seeking, block based compression etc. Protobuf unfortunately doesn't define any standard file formats or even wire protocol framing. If you need to do more than simply store and scan/read in bulk you might want to use Parquet instead though.

Reading this blog post was probably a waste of my time, hopefully this comment actually helps someone though.

Some morons even write books and blog posts about this. The funny thing is this sort of shit is done in the name of scale, but the big folks never operate this way. Large scale infrastructures actively disdain keeping buffers and state in the middle of the request flow. They cannot afford the cost and latency of such systems. They do it the sane way[1].

[1] https://www.usenix.org/conference/osdi23/presentation/saokar

I have a scenario where we have hundreds of installs of an old and shitty RDBMS on customer sites, and we need to replicate changes to data to a central store. We had to come up with a bespoke event system that would capture insert, update, and delete events, ship them to a REST endpoint, who would then throw them into Kafka to be processed into the central store. Kafka’s ordered message log made it ideal for this scenario, as we can’t play events out of order (although because of poor design in the old databases, it sometimes happens and we built a retry system using additional Kafka topics, nonetheless avoiding out of order messages is critical to keep consumer lag under control).

This works mostly ok, but we have a problem when individual customers have big bursts of traffic. Ultimately, we need records to be processed in the order they happen, per customer. Naively, we could partition by customer ID, but arbitrarily adding new partitions as we add customers is not practical over time, and regardless, bulk inserts, updates, etc. could cause large amounts of latency for a customer. So, we’re doing a balancing act of trying to partition using customer ID + a “bundle name” of related tables (the net effect being activity to dependent tables for the same customer always go to the same partition and thus process in order). We’re also looking at using additional topics to create high, medium, and low priority queues, but while that may smooth out some of the problems, it really only breaks the original problem into three smaller versions of the same problem, effectively kicking the can down the road.

Ultimately, the best solution would be to get rid of the crappy RDBMS and replace with something that we can binlog or otherwise sync transactionally rather than record by record. We are working on this, but it’s slow going. In the meantime, we continue to wrestle with Kafka partitioning woes.

As an aside, we also got rid of Avro. It just didn’t have any benefits that outweighed the challenges to get it and keep it working over time. Much easier to just use plain json, a common message class library between consumers and producers, and a fast, traditional json library. I’ll fully admit that perhaps the avro woes are more an issue inexperience, but I seem to find more people who have the same experience as me than not. Either way, plain json has not caused us any problems.

https://en.wikipedia.org/wiki/Apache_Kafka

1. You cannot just use Kafka for free. It will take dev time to set up itself, dev time to code sources and sinks, dev time to handle commonly glossed over but utterly important details like idempotency, retries, duplicate messages, consumed-but-not-committed (or whatever the term is in Kafka world) network interruptions or restarts of consumers.

2. You cannot just continue to use Kafka for free. Running it has an operational cost; this is mitigated using it as a PAAS but not fully solved as you’ll still need to twiddle configurations and scramble to deal with things like “We had no idea we’d need to handle idempotency or use deadletter queues, fixed it, but now need to deal with old data before our fix”.

3. There are many ways you can implement async producer:consumer patterns that are less complex and less costly than Kafka. For example, you can write data to S3. Or you can store records in a regular RDBMS. Kafka isn’t worth it unless you really need “real-time” ingestion but can’t/won’t/shouldn’t implement an actually-real time (synchronous) system instead, like if you get large spikes and are ok with ingestion going from O(seconds) to O(minutes) when that happens.

4. There’s a good chance you don’t need an async queue at all. If consumers can horizontally scale quickly (like with Lambda) why not synchronously invoke them over HTTP/RPC and only use async queues (or a file, etc) for messages that fail multiple retries? Since external users usually aren’t directly writing to your Kafka topic, and thus you have a degree control over your ingestion and consumption, why not just combine the two services (since you can experience data loss from external world to ingestion service anyway, and in fact this is a pretty likely source of failures, your queue may not even be solving the problem you think it is). If you don’t need ordering or partition and are using queues for eg config update propagation, why not just synchronously update consumers or implement basic polling in your consumers?

5. A lot of Kafka/Confluent “features” like retries or logging you can get with so many other tools and services but for some reason these can be the actual selling point more so than the fact it’s an async queue (that also has these features).

Yes, in a FAANG design interview where it only costs you 5seconds to say “and we’ll use an async queue like Kafka between these components to handle variable load and partially consumed data” it’s a great tool that saves you a lot of time. And when you pretend integration and maintenance costs don’t exist, and don’t even know what idempotency means, and are sitting across from some slick Confluent salesperson telling you Kafka can be THE database of everything your company does with all these nice features, it sounds great to midwit managers and hasbeen architecture astronauts. In reality? The dumb unsexy alternatives probably solve your actual problem more simply

I agree with the author to an extent - kafka is overly complex for low traffic systems. It can however be a godsend in specific cases, but what those cases are can be hard to pinpoint.

The thing that makes Kafka interesting is the technique of operating from a linux disk write-buffer. That's the trick that makes it fast and scale to huge volumes. But if you don't have the scale, you can stand up a table, or RabbitMQ, or anything that manages append-only ordered log entries. There doesn't need to be a new thing... Kafka was the new thing.

Overkill in what sense? The blog post seems to suggest Kafka was already pervasive in their organization, and that they leveraged the existing infrastructure and simply added.a couple of topics. How is this overkill?

does something like that exist ???

The think Kafka really brings to the table is trusting the pipe -- in a case where writing to disk queues would occur often enough, I would run into reliability building my own system, instead Kafka handles that type of indexing.

your suggestion of single node rdbms as a replacement to kafka suggest you dont have experience with workloads that cannot be served by a single machine, yet you still need a single architecture.

agree that Confluent took a gread product that works for high load use case, and then tries to shove it to each and every average Fortune1000 enterprise use case with 100 users and traffic that could be well served by SQLite/Postgres on a single machine