Circuit Breakers in Elixir(blog.rokkincat.com) |
Circuit Breakers in Elixir(blog.rokkincat.com) |
Try iTunes
Catch exception
Try local search
I guess in a bad state it makes your results faster. How often does that need to happen to justify the complexity?
Circuit Breakers provide three things not provided by your above scheme:
First, there is a configurable policy on how many errors to tolerate before breaking. This policy is not baked into your code, but lives outside, often in configuration files. Some of the work that is currently going on is the support for more advanced policies and more advanced ways of ramping up connectivity again on the flip side.
Second, there is no resource buildup. In your example, every request to iTunes will wait and use resources while it is waiting. Once the circuit breaks, you immediately respond with an error. In a system with 10k req/s the buildup is pretty serious if you have a timeout of 5 seconds, say, since it will effectively be 50k reqs waiting. Which could be 50k network sockets.
Third, fuse has a monitoring system built in. Any fuse you create will post its current state to an event manager which can be used to build monitoring applications (essentially this is a low-volume pub/sub pattern). Rolling your own, you have to provide this monitoring yourself, but using fuse, you get a nice way to plug into the fabric. This is used by Riak's Search system yokuzuna for instance.
Finally, what makes fuse special from the other circuit breakers is that it has a full QuickCheck specification. I.e., we have a pseudo-formal account of fuse working as intended according to the specification. In particular, I tend to generate random fuse scenarios for a couple of hours before releasing new versions. This amounts to a couple million random test cases, and we approach a full model-check of the code as we spend more time generating test cases. There are some novel work in there with respect to handling randomness and time in test cases via Erlang's QuickCheck's excellent mocking system. As a result, there have been few bug reports and likewise few fatal errors reported.
(Edit: for completeness, all the code of fuse is online here: https://github.com/jlouis/fuse including documentation)
Now I need to find one for python to tinker with.
Additionally some third party API's have rules where they expect you to back off exponentially if there are errors. You'll get your account suspended if you don't play by the rules.
In a timeout situation, you know relatively little.
If the circuit tells you it is broken, you know that the backend system is deemed down and trying again right now is probably futile.
Each client for a server (on which the fuse runs) will be rather blind to the experiences of other clients. But the server, and thus the fuse, will know what happens to all the clients. This allows you to act globally on the server side. But this in turn empowers each client locally since it now also has global information.