High-performance, exactly-once, failure-oblivious distributed programming (2018)(christophermeiklejohn.com) |
High-performance, exactly-once, failure-oblivious distributed programming (2018)(christophermeiklejohn.com) |
Even within centralized, monolithic systems with no outside interaction, laws of physics and reality still apply - a pull on the cord, an earthquake, a flooding or a myriad of other things may interrupt message processing resulting in either exactly zero or more times a message will be processed even if the message were being processed within the confines of an embedded micro-controller using hand crafted assembler code.
In critical sectors like banking, finance and payments, designing systems in well understood and boring manner is absolutely critical while hoping for the best based on shiny brochures and marketechture is a sure recipe for disaster.
"Exactly once semantics" is semantics. It's at least once with idempotency, which may or may not be able to be guaranteed on the part of the system depending on actual implementation details which the marketing fluff will invariably leave out if they're saying "exactly once". And that's a major problem when relying on such 'semantics'.
I bet analyzing those messages using this framework would reveal exactly why compensating actions are needed.
The takeaway I am getting from this paper is that, you're not just looking for idempotency. Monotonicity is important, as well as commutativity and associativity. If it cannot be expressed in that way, then coordination is required.
There are additional efficiencies if the operators are commutative and associative in addition to being idempotent.
It's now relatively simple for a developer to implement a system with exactly once guarantee as long as you take care of the world that is not inside a Kafka transaction (integrations with third parties and such), which is still not super easy sometimes, but less so then the distributed transaction that will happen inside Kafka.
Kafka hides the complexity really well from my use of it so far is very reliable with the "new" semantics.
Everybody and their uncles seem to have picked up on this trend of advertising at-least-once systems as exactly-once, then burying somewhere in the docs that you're expected to guarantee idempotency yourself to get the appearance of exactly-once. That was the state of the art decades years ago, it's the state of the art now, and it's pretty damn dishonest to sell quality-of-life improvements as a fundamental shift in the guarantees/properties of these systems.
What baffles me even more is why the above is apparently not generally considered good-enough, elegant-enough --- and as a bonus, not violating the laws of physics either? Both sides of the coin are quite tameable and implementable. And together deliver what was wanted in the first place, and effectively. Curious in any subtle edge-cases I might have missed here!
Databases have been handling exactly-once semantics for decades now. What Kafka is doing is not new and actually gives you a false sense of security when it comes to these kinds of things.
What Kafka supports is exactly-once processing which has been supported in other stream processing frameworks such as Apache Storm years before Confluent's marketing. Duplicates are possible in Kafka with the current implementation of exactly-once, if one uses Kafka's consumer api it will de-dedupe on the processing side.
So no, there is no such thing as exactly-once in distributed systems.
The idea of a replayable log seems to be able to convert a disordered sequence of events into something that is ordered. Whereas, the Bloom(L) stuff constructs algorithms that only requires partial order. An event stream can be disordered because the functions being used are monotonic, and the compositions of the data structure uses operators that are commutative, associative, and idempotent. (Thus, there is no requirement for exactly-once guarantee, or an ordered event stream).
AFAIK nowhere in the Kafka documentation does it use the term "queue", and unless you only have one consumer per consumer group it's impossible to guarantee FIFO behavior. Maybe call me a nitpicker but I've seen this "queue" language lead to completely wrong assumptions about kafka.
People insist on this messaging precisely because that crowd is somewhat vulnerable to this sort of shenanigans.
I've seen teams try to continue programming the way they always have and try to throw difficult problems over the wall. It is exactly-once in a non-threaded, monolithic, stateless web app, so why shouldn't it always be like that? It's part of a programmer's mindset to try to create abstractions and reason within a simpler problem space.
Or it can be modeled as event stream where on {id=2, t=2, +$300} was posted, and {id=2, t=4, -$300} was declined. If you want to distinguish between a "retry" or a "remainder", you add that into the tuple within the event stream.
"Send it again" doesn't mean "processed again". Isn't exactly-once built on at-least-once just binding the result to a future? Then any subsequent accesses or attempts to update will simply return the bound result, which was processed exactly once.
Even if it does persist, does it persist the identifier on receipt of the message, or on sending it to you for processing? If the former you run the risk of crash and never having handled it; you really have no guarantee of delivery to your processor. If the latter, what happens if the receiver crashes before it hands it off to be processed? If simultaneous, is 'processing' atomic? Probably not; what happens if you crash midway through processing the thing? Etc.
That's my point; you need more details to make the system robust. You don't get "exactly once delivery" out of the box; you get a system that attempts it by deduplicating, but there be gremlins, and the fact you're not saying "it's at least once delivery with (details)" means I'm not hearing a technical pitch, but a marketing one.