Now when people argue “because decoupling,” I hear, “You don’t get as much notification that you just broke a downstream system.”

Event driven architecture, to me is itself an antipattern.

It seems like a replacement for batch processing. Replayable messages are AWESOME. Until you encounter the complexity for a system to actually replay them consistently.

As far as the authors video, while there was some truth in there, it was a little thin, compared to the complexity of these architectures. I believe that even though Kafka acts the part of "dumb pipe", it doesnt stay dumb for long, and the n distributions of Kafka logs in your organization could be 1000x more expensive than a monolithic DB and a monolithic API to maintain.

Yes it appears auditable but is it? The big argument for replayability is that unlike an API that falls over theres no data loss. If you work with Kafka long enough you’ll realize that data loss will become a problem you didnt think you had. You’ll have to hire people to “look into” data loss problems constantly with Kafka. Its just too much infrastructure to even care about.

Theres also, something ergonomically wrong with event drive architecture. People dont like it. And it also turns people into robots who are “not responsible” for their product. Theres so much infrastructure to maintain that people just punt everything back to the “enterprise kafka team”.

The whole point of microservices was to enable flexibility, smart services and dumb pipes, and effective CI/CD and devops.

We are nearing the end of microservices adoption whether it be event or request driven. In mature organizations it seems to me that request driven is winning by a large margin over event driven.

It may be counterintuitive, but the time to market of request driven architecture and cost to maintain is way way lower.

In pretty much all projects I worked with in recent years, people chop up the functionality into small separate services and have the events be serialised, sent over the network and deserialised on the other side.

This typically causes enormous waste of efficiency and consequently causes applications to be much more complex than they need to be.

I have many times worked with apps which occupied huge server farms when in reality the business logic would be fine to run on a single node if just structured correctly.

Add to that the amount of technology developers need to learn when they join the project or the amount of complexity they have to grasp to be able to be productive. Or the overhead of introducing a change to a complex project.

And the funniest of all, people spending significant portion of the project resources trying to improve the performance of a collection of slow nanoservices without ever realising that the main culprit is that the event processing spends 99.9% of the time being serialised, deserialised, in various buffers or somewhere in transit which could be easily avoided if the communication was a simple function call.

Now, I am not saying microservices is a useless pattern. But it is so abused that it might just as well be. I think most projects would be happier if the people simply never heard about the concept of microservices and instead spent some time trying to figure how to build a correctly modularised monolithic application first, before they needed to find something more complex.

I've ended up in a lot of arguments about this while we were building larger distributed systems because I've come from a more request/response oriented message passing architectures. I.e. more synchronous. What I've found is that the event driven architecture did tend to lead to less abstractions and more leaked internal details. This isn't fundamental (you can treat events like an API) but was related to some details in our implementation (something along the line of CDC).

Another problem with distributed systems with persistent queues passing events is that if the consumer falls behind you start developing a lag. Yet another considerations is that the infrastructure to support this tends to have some performance penalties (e.g. going through Kafka with an event ends up being a lot more expensive than an RPC call). Overall it IMO makes for a lot of additional complexity which you may need in some cases, but if you don't then you shouldn't pay the cost.

What I've come to realize is that in many ways those systems are equivalent. You can simulate one over the other. If you have an event based system you can send requests as events and then wait for the response event. If you have a request/response system you can simulate events over that.

If we look at things like consensus protocols or distributed/persistent queues then obviously we would need some underlying resources (e.g. you might need a database behind your request/response model). So... Semantics. Don't know if others have a similar experience but when one system is mandated people will invent workarounds that end up looking like the other paradigm, which makes things worse.

There are things that conceptually fit well with an event driven architecture and then there are things that fit well with a request/response model. I'm guessing most large scale complex distributed apps would be best supporting both models.

I’ve used it with a good degree of success in some data pipeline and spark stuff to have stuff automatically kick off, without heinous conditional orchestration logic. I also use evented stuff over channels in a lot of my rust code with great success.

However, echoing the sentiments of some other comments: most articles about event driven stuff seem to be either marketing blogspam or “we tried it and it was awful”. To be honest I look at a lot of those blog posts and about half the time my thoughts are “no wonder that didn’t work out, that’s an insane design” but is that just “you’re-doing-it-wrong-cope”?

Are there success stories out there that just aren’t being written? Is there just no success stories? Is the architecture less forgiving of poor design and this “higher bar of entry” torpedoes a number of projects? Is it more susceptible to “architecture astronauts” which dooms it? Is it actually decent, but requires a somewhat larger mindset-change than most people take to it, leading to half-baked implementations?

I can’t help but feel the underlying design has some kernels of some really good ideas, but the volume of available evidence sort of suggests otherwise.

Edit- I should say I never saw one in the wild, quick search found some academic projects https://scholar.google.com/scholar?q=event-driven+control+sy...

There is no avoiding it when dealing with, erm, events.

Events are things that happen that you cannot predict exactly when, where, and what.

The user clicked the mouse

The wind changed direction

Using Events to signal state change from one part of a system to another is a bad idea. Use a function call.

A rule of thumb is if the producer and the consu,er are in the same system then "Event Driven Architecture " is the anti pattern

I feel like a lot of teams out there can probably benefit from this simpler approach - it's probably what a lot of people are doing unwittingly.

> Commands only have a single consumer. There must be a single consumer. That’s it. They do not use the publish-subscribe pattern.

...oops.

Now the question is how much (more) time I want to spend on a(nother) rewrite.

In my experience programmers are very happy to do everything in the application (something database people often complain about). What kind of problems do you see?

> If you work with Kafka long enough you’ll realize that data loss will become a problem you didnt think you had. You’ll have to hire people to “look into” data loss problems constantly with Kafka.

Not my experience at all, and I've used Kafka at a wide range of companies, from household-name scale to startups. Kafka is the boring just-works technology that everyone claims they're looking for.

I'm no fan of microservices, but Kafka is absolutely the right datastore most of the time.

Not to mention certain observability vendors bleeding you for all those logs you now need to keep an eye on it.

Absolutely agreed on every point

Also, people need to understand that "event driven" has nothing to do with "event sourcing". Just don't keep all the events until eternity, because you can (and because some people think you should because "kafka").

But when I've done that testing, Kafka hasn't been the problem.

The problem I've run into most is that ordering is a giant fucking pain in the ass if you actually want consistent replayability and don't have trivial partitioning needs. Some consumers want things in order by customer ID, other consumers want things in order by sold product ID, others by invoice ID? Uh oh. If you're thinking you could easily replay to debug, the size and scope of the data you have to process for some of those cases just exploded. Or you wrote N times, once for each of those, and then hopefully your multi-write transaction implementation was perfect!

[0] in fairness, a lot of applications also don't guarantee that they never drop requests at all, obviously. 500 and retry and hope that you don't run out of retries very often; if you do, it's just dropped on the ground and it's considered acceptable loss to have some of that for most companies/applications.

Microservices make sense when there are very strong organizational boundaries between the parts (you'd have to reinterview to move from one team to the other), or if there are technical reasons why two parts of the code cannot share the same runtime environment (such as being written in different languages), and a few other less common reasons.

The core orchestration of the system was done via events on the bus, and nobody had any idea what was happening when a bug occurred. People would pass bugs around, “my code did the right thing given the event it got”, “well, my code did the right thing too”, and nobody understood the full picture because everyone was stuck in their own silo. Event driven architectures encourage this: events decouple systems such that you don’t know or care what happens when you emit a message, until one day it’s emitted with slightly different timing or ordering or different semantics, and things are broken and nobody knows why.

The worst part is that software is basically “take user input, do process A on it, then do process B on that, then do process C on that.” It could have so easily been a simple imperative function that called C(B(A(input))), but instead we made events for “inputWasEmitted”, “Aoutput”, “Boutput”, etc.

What happens when system C needs one more piece of metadata about the user input? 3 PR’s into 3 repos to plumb the information around. Coordinating the release of 3 libraries. All around just awful to work with.

Oh and this is a very high profile piece of software with a user base in the 9 figure range.

(Wild tangent: holy shit is hard to get iOS to accept “do process” in a sentence. I edited that paragraph at least 30 times, no joke, trying every trick I could to stop it correcting it to “due process”. I almost gave up. I used to defend autocorrect but holy shit that was a nightmare.)

I can attest to this case study being 100% true. Our platform has been using EventStore as our primary database for 9 years going strong, and I'm still very happy with it. The key thing is that it needs to be done right from the very beginning; you can't do major architecture reworks later on and you need an architect who really knows what they're doing. Also, you can't half-ass it; event sourcing, CQRS, etc all had to embraced the entire time, no shortcuts.

I will say though, the biggest downside is that scaling is difficult since you can't always rely on snapshots of data, sometimes you need to event source the entire model and that can get data heavy. If you're standing up a new projector, you could be going through tens of millions of events before it is caught up which requires planning. It is incredible though being able to have every single state change ever made on the platform available, the data guys love it and it makes troubleshooting way easier since there's no secrets on what happened. The biggest con is that most people don't really understand it intuitively, since it's a very different way of doing things, which is why so many companies end up fucking it up.

I also tried doing it in a property setting, where profit margins were tight. The effort needed wasn’t worth the cost, and clients didn’t really care about the value proposition anyway. We pretty much replaced the whole layer with a more traditional crud system.

In web or business systems it works well for some(!) parts. You just shouldn’t do everything that way - but often people get too exited about a solution and then they tend to overdo it and apply it everywhere, even when not appropriate.

Always chose the golden middle path and apply patterns where they fit well.

Event driven and CQRS "entities" made logic and processing much easier to create/test/debug.

Primary issues: 1. Making sure you focus on the "Nouns" (entities) not the "Verbs". 2. Kafka requiring polling for consumers sucks if you want to "scale to zero". 3. Sharding of event consumers can be complicated. 4. People have trouble understanding the concepts and keep wanting to write "ProcessX" type functions instead of state machines and event handlers. 5. Retry/replay is complicated, better to reverse/replay. Dealing with side effects in replay is also complicated (does a replay generate the output events which trigger state changes in other entities?)

Been running now for 6 years, minimal downtime except for maintenance/upgrades.

In the process of introducing major new entity and associated changes, most of the system unaffected due to the decoupling.

These system are working fine, but maybe a common ground : * very few services * the main throughput is "fact" events (so something that did happen) * what you get as "Event carried state transfer" is basically the configuration. One service own it, with a classical DB and a UI, but then expose configuration to all the system with this kind of event (and all the consumers consume these read only) * usually you have to deal with eventual consistency a lot in this kind of setup (so it scales well, but there is a tradeoff)

The WAL is an event log, and when you squint at its internal architecture, you’ll see plenty of overlap with distributed event sourcing.

Our users are small-businesses with organisation numbers, and we mostly think of them as unique. But they strictly aren't, so we 'overwrote' some companies with other companies.

Once we detected and fixed the bug, we just replayed the events with the fixed code, and we hadn't lost any data.

Every use I've seen sent events after database transactions, with the event not part of the transaction. This means you can get both dropped events, and out of order events.

My current company has analytics driven by a system like that. I'm sure there's some corrupted data as a result.

The main issue being people just don't know how to build and test distributed systems.

So we ended up using protobufs over a local MQTT broker and adopted a macro-service architecture. This suited the project very well because it had a handful of obvious distinct parts and we took full advantage of Conway's law by making each devs work the part where their strengths and skills were maximized.

We made a few mistakes along the way but learned from them. Most of them relating to inter-service asynchronous programming. This article put words on concepts we learned through trial and errors, especially queries disguised as events.

I think it works well when it's the only thing that can work.

- Producer and consumer are decoupled. That’s a good thing m right? Good luck finding the consumer when you need to modify the producer (the payload). People usually don’t document these things

- Let’s use SNS/SQS because why not. Good luck reproducing producers and consumers locally in your machine. Third party infra in local env is usually an afterthought

- Observability. Of rather the lack of it. It’s never out of the box, and so usually nobody cares about it until an incident happens

JavaScript

When I say increased, I mean we want the best answer but there are some answers the bank can’t know. If someone has transferred money into your account from another bank but we don’t know that yet, optimising for absolute correctness is pointless because the vast majority of wrong answers are baked in to the process. We can send you a message and you might read it a day later. Unless we delete the message from your phone, we can’t guarantee the message you read is fully consistent with our internal state.

Frankly our system is much better than the batch driven junk that is out of sync a second after it has executed. “Hey you have a reward.” “No I used it 2 hours ago you clowns.”

Note this isn’t cope. In some cases we started fully sync but relaxed it where there are tradeoffs that gave us better outcomes and we weren’t giving anything material up.

I'd put it the other way: event driven architecture makes it safer to expose more internal details for longer, and lets you push back the point where you really need to fully decouple your API. I see that as an advantage; an abstract API is a means not an end.

> Another problem with distributed systems with persistent queues passing events is that if the consumer falls behind you start developing a lag.

Isn't that what you want? Whatever your architecture, fundamentally when you can't keep up either you queue or you start dropping some inputs.

> If you have a request/response system you can simulate events over that.

How? I mean you can implement your own eventing layer on top of a request/response system, but that's going to give you all the problems of both.

> If we look at things like consensus protocols or distributed/persistent queues then obviously we would need some underlying resources (e.g. you might need a database behind your request/response model).

Huh?

> Don't know if others have a similar experience but when one system is mandated people will invent workarounds that end up looking like the other paradigm, which makes things worse.

I agree that building a request/response system on top of an event sourcing system gives you something worse than using a native request/response system. But that's not a good reason to abandon the mandate, because building a true event-sourcing system has real advantages, and most of those advantages disappear once you start mixing the two. What you do need is full buyin and support at every level rather than a mandate imposed on people who don't want to follow it, but that's true for every development choice.