Most consensus libraries (Raft, Paxos) treat the state machine as a pure black box. This is fine until your state machine needs to actually do something, like charge a credit card, fire a webhook, or send an email. If a leader crashes after the side effect but before committing it, you get duplicates. This is my attempt at fixing this problem from first principles ish: build chr2 to make crash-safe side effects first-class citizens. mechanism: Replicated Outbox: Side effects are stored as "pending" in replicated state. Only the leader executes them under a fencing token. Durable Fencing: A manifest persists the highest view using atomic tmp+fsync+rename. This ensures a "zombie" leader can't wake up and double-execute stale effects. Deterministic Context: Application code receives a deterministic RNG seed and block_time from the log, ensuring 1:1 state transitions during replay. Strict WAL: Entries are CRC’d and hash chained. it is designed to prefer halting on mid-log corruption over guessing. The Trade-offs: Side effects are intentionally at-least-once; "exactly-once" requires stable effect IDs for sink-side deduplication. It’s a CP system safety over availability. Repo: https://github.com/abokhalill/chr2 if you’ve ever had “exactly once” collapse the first time a leader died mid flight, you know exactly why I built this. |