Eric Brewer on Why Banks are BASE, Not ACID – Availability Is Revenue(highscalability.com) |
Eric Brewer on Why Banks are BASE, Not ACID – Availability Is Revenue(highscalability.com) |
I don't think so. I support ATM client software for a large bank in the US and we certainly don't do this. This may be true for "remote" ATMs that are installed in convenience stores on POTS. I can't say I've ever actually heard of it though - the main problem with this idea is that cards cannot be authenticated without network access, and just spewing out money to every piece of plastic calling itself a card when your network connection has been dropped isn't really a recipe for success. Fraud is a real problem.
The ATM client software I support cannot do any transactions without a connection with its authorization system. That authorization system though, can stand-in for the various accounting systems and external networks up to pre-defined limits. So for example if for some reason we can't reach the checking account system we'll authorize up to $xxx total for the day on a stand-in basis. The transaction with the authorization system is definitely ACID; the ATM will not get a response code authorizing a withdrawal unless the transaction has been recorded in the authorization system. The account system may well be caught up later. The funny thing is, ACID is a property of individual database systems and it has absolutely nothing to with a question of whether two separate ledgers are guaranteed to be changed together or not at all. That would be the job of a distributed transaction coordinator - and those really are not used very much in banking. Instead there is a protocol of credits and debits and a settlement process to work out the exceptions. Maybe this is what the article was trying to say up to a point but they sort of confused the issue between the point of view of the ATM and the accounting systems of record.
Well they do this in Australia... http://www.news.com.au/money/banking/computer-glitch-hits-cb...
> "People were running past me screaming 'Free money! Free money!'," Punchbowl Pharmacy manager Feriale Zakhia said of the people using a nearby ATM.
> "Everyone was so happy. They were running around with huge smiles."
> [A technical problem] forced the bank to put all of their ATM machines into offline mode. Customers had no access to their account balance but were still able to withdraw money - more than their accounts held.
> Those withdrawal limits are up to $2000 a day for holders of keycards and debit Mastercards.
> "No one has received free cash," Mr Fitzgerald said. "What they've done is overdrawn their accounts. We will be following those people up and recovering that money."
In that case, the ATM was disconnected from the accounting system and allowed withdrawals up to a set limit ($2000), but it (probably, the article is unclear) was still connected to the authorization system.
It (probably) still checked your PIN, and checked whether your card had been cancelled, etc. It just didn't connect through to check your balance.
As Jeremy said if for some reason we can't reach the checking account system we'll authorize up to $xxx total for the day on a stand-in basis
In this case some reason == "[A technical problem]" and xxx == $2000
I'm sure that, under the hood, there are a lot of ACIDic transactions going on, but, stepping back, it still looks pretty BASEic. When I hit "withdraw $200", there is no guarantee that my actual account has a transaction commit for that amount. Instead, there may just be a log message saying "SoftwareMaven withdrew $200".
I don't really think it is useful to try and use the terms ACID/BASE to refer to the aggreggate process behavior of an entire industry's technology. Not that it is wrong, it just really doesn't mean anything at that point. Most data interchange that takes place between thousands of different parties is going to have similar characteristics. Maybe a market/exchange is a good counter-example but I can't think of many others.
I believe some of the first ATMs actually worked offline (the PIN was encoded on the magentic stripe), but networked models came out a few years later. Of course, this was in the late 1960s, when card readers (and the expertise to use them) were far harder to obtain.
Exactly--the ecosystem as a whole is BASE, but the individual systems are ACID, and the "eventual consistency" aspect is implemented as first-class logic in applications/processes.
I think BASE is a good ecosystem-level principle, but when it gets into datastores, then each individual system doesn't know what it's true opinion of the world is, much less how it can effectively coordinate with other similarly potentially confused systems.
Source: I am a former software engineer for a credit card transaction system vendor.
Banks are one software kludge after another in attempt to not rewrite something new or offer the consumer anything of value... while paying out the butthole to whatever vendor has his arm shoved so far up your ass you can't ever migrate from their platform without colon replacement surgery.
So while it's a nice thought "Hey look Banks/ATMs are BASE" this was by complete accident through years of incompetence and corporate bureaucracy, not by any sort of engineering choice.
Banks used to work on paper an did fine, software migration, although can take some years, is no problem for them.
The reality is that you and Eric are right.
Banks depend on techniques based on double book keeping accounting to reconcile accounts at end of day. Different data about transactions are stored in different places by different organizations and they compare books to make sure that balances are correct.
You cannot depend on every transaction to be recorded perfectly. You must have the ability to compare books and reconcile accounts. This is simply how the world works.
Trying to make every perfect and depending on storing data in a central place with the assumption that it's always going to be consistent is too much of a liability. It doesn't work because the systems required by modern financial systems are incredibly complex and availability during markets is the highest priority. You ARE going to have faults and you ARE going to have problems. The ability to take hits gracefully and give yourself time later on to fix stuff after the fact must be built into your systems.
The ATM network is distributed and eventually consistent, and financial transactions in general are not real time.
Within an ATM or within a bank you can be damn sure transactions are used widely to prevent inconsistent data.
If you draw it at the boundary of the central general ledger, it's going to be ACID.
If you look at the way transactions pass through several intermediate systems (each of which is ACID) en route, I guess it could be called BASE.
Whether that is true or not is a much deeper discussion that three or four paragraphs on a blog.
I don't think this follows. While the system as a whole might be eventually consistent (where consistency is defined as: what's in the General Ledger), it doesn't follow that you relax constraints on the parts.
The individual components are generally ACID and the steps to move data are ACID as well. The only thing that prevents the whole system from being ACID is that transactions don't go immediately from POS/ATMs/card clearance/cheque clearance into the General Ledger, but instead must go through a series of intermediate transactions. But those intermediate transactions must, themselves, be ACID.
That's why I talked about how the boundaries matter. The final central accounts are ACID, the subsystems are ACID and the data transfers are ACID.
edit: though to contradict myself, I expect that there will be counterexamples in different banks where some stores or steps will not be strictly ACID, but will have been "good enough" or with sufficiently-acceptable workarounds that they haven't been upgraded. I don't think this fatally breaks my argument, but YMMV.
However, banks are one of the biggest purchasers of ACID systems... They still bet heavily on oracle, and when they want their accounting systems to run, and balance transactions, they dont rely on "BASE" systems. Banks are also heavily dependent on business cycle and batch processing. Daily batches are common in credit card processing (end of business day settlement for example), and also in general bank systems.
Also eventual consistency means different thing. Some systems have a "eventually inconsistent" property to them (eg: Cassandra/Dynamo), and I'm pretty sure banks would NOT be ok with that.
I respect Brewer, he is a smart guy, but he is extrapolating too much from a small fact that is ATMs will sometimes dispense cash (of what amounts? $200? $1000? or maybe just $20?) when remote communications are interrupted or broken.
I'm pretty sure "eventual consistency" doesn't mean "eventual inconsistency" under any reasonable interpretation.
As for the eventual inconsistency remark, this is from the original authors of Cassandra at Facebook. They did not extend the use of Cassandra because node flaps and packet losses caused nodes to be inconsistent and have old data. Bringing that data back up to date was very difficult, since the anti-entropy algorithms were too expensive to run frequently. They also found that the R=W=2 was too costly in terms of performance, and well you know the rest :-)
I've never heard of financial systems being eventually consistent, certain not ATMs which need to know your exact balance and how much money you've taken out already today.
I studied ATM's at Uni, and they have been like that for over 26 years.
As someone else on the thread pointed out, they do authenticate you, and record the transaction in a aci(Durable!) database before they give you money. But the system clearly has a degraded mode when the exact balance is unavailable.
Also why do you need a balance to process a deposit?
You can think of the (rare nowadays, but possible) offline, paper-based card transactions (physical imprint of the card number + signature) as a very strong example of eventual consistency - your card will be billed for that amount sometime later when the documents are processed and all the balances will be correct, but for many days the "online visible" card balance as known to the bank will be different from the "real" legal/accounting balance of that card.
It's why your bank account can go into negative territory.
Double-entry is more of an internal (financial) system implementation detail, and so an orthogonal concern to intra-system audits.
(It's not like one side of the entry is in one bank's IT system, and the other side of the entry is in the other bank's IT system.)
(...speculating further, I really doubt the OLTP systems of banks are double-entry anyway.)
If a deal involves two banks, then the authoritative "other side" of the entry will be in the other bank's IT system. Of course, you'll maintain some records of what the entry should be in your opinion, but they won't always match as you don't have full info and you'll reconcile with data you get from the other bank's IT system by (for example) SWIFT network.
And I've seen only double-entry OLTP's for the core system that includes general ledger. Maybe no all of them are built that way, but I haven't seen such examples.
Authorization: An ATM does not issue money without authorization which is done by some 'central authority', not the ATM.
Limits: In some corner cases you may be able to exceed your (daily, weekly, monthly) limits. But, as the article points out, this doesn't imply financial inconsistency.
Or, as in my disputed case, $20 cash and a $140 balance, because the deposited check hadn't actually cleared by close-of-business, so end-of-day processing assessed a $40 negative balance fee despite the fact that the deposit "eventually" cleared. The first manager I discussed this with had the audacity to claim it was my fault for not somehow recognizing that the portion of the deposited funds the ATM made available for immediate withdrawal by design were not, in fact, available for immediate withdrawal.
Banks have chosen consistency over availability regularly as they've been able to rebuild their systems over the decades. 40 years ago, if you had a credit card the place just took it, copied it down and trusted you and the bank were good for it. Try to get anyone to take your credit card if the network is down now.
IMO banking culture and standards probably were direct motivators of many of the ways traditional systems were designed. They were some of the earliest adopters of IT. It may not be accurate to say banks or ACID. It may be more accurate to say ACID is banking.
With BitCoin, such an architecture could be abused rather badly.
ATM software has nothing to do with the backend storage of the data. My point is that I bet when the data is actually written to durable storage in the backend, the set of data being written will be wrapped in an ACID transaction.
BTW I've been scammed out of money by ATMs before - money was withdrawn from my account but some system jammed and I didn't get the money - and the bank was awfully uncooperative.
So far I've had more luck with the "money under the mattress" method than with banks - and I wasn't trapped in the "Corralito" or other bank-aided money-stealing schemes
Personally, I feel that ACID is an abstraction achievable only within single, non-distributed systems. Not a very compelling insight, I hear you say.
Well no, but ACID is a tremendously advantageous state of affairs and I feel it should be surrendered only begrudgingly. I think it is better to repair it than to abandon it wholesale at the first sign of mild inconvenience.
Even though it is, in a physical sense, untrue, it is a useful untruth. Newtonian physics is wrong. It's also what we use to build bridges.
Though, I do need to emphasise a point:
> subsets of the transaction
Each step in moving the information from the ATM to the General Ledger is itself a transaction. There are no "subsets of a transaction". If it's divisible, then it's not a transaction (this is the atomicity requirement of ACID).
We see this pattern all over the place. The primary example is any place that accepts checks as payment. Availability is immediate, but consistency is not necessarily there.
You see it with a business making a PO. Lots of places will still take credit cards which are only processed daily (ok, this is becoming rare where there is cell service.)
Ecuador and Brazil also had their own versions. In the Uruguayan version, they didn't forcibly exchange the money, but they froze all bank assets for 3 years (losing out on interest, investment opportunities, exchange rates, etc...).
Uruguay is more trustworthy (especially with foreign investment) but it's not above such things.
Currently there's a big scare due to the huge exchange rate disparity with Argentina - which has an "official" exchange rate and a "real" exchange rate which is almost double the official one, and makes Uruguay non-competitive.
Edit: you're from Argentina, that's obviously not news for you :)