
Eric Brewer on Why Banks are BASE, Not ACID – Availability Is Revenue - abhijitr
http://highscalability.com/blog/2013/5/1/myth-eric-brewer-on-why-banks-are-base-not-acid-availability.html
======
jeremyjh
>If an ATM is disconnected from the network and when the partition eventually
heals, the ATM sends sends a list of operations to the bank and the end
balance will still be correct.

I don't think so. I support ATM client software for a large bank in the US and
we certainly don't do this. This may be true for "remote" ATMs that are
installed in convenience stores on POTS. I can't say I've ever actually heard
of it though - the main problem with this idea is that cards cannot be
authenticated without network access, and just spewing out money to every
piece of plastic calling itself a card when your network connection has been
dropped isn't really a recipe for success. Fraud is a real problem.

The ATM client software I support cannot do any transactions without a
connection with its authorization system. That authorization system though,
can stand-in for the various accounting systems and external networks up to
pre-defined limits. So for example if for some reason we can't reach the
checking account system we'll authorize up to $xxx total for the day on a
stand-in basis. The transaction with the authorization system is definitely
ACID; the ATM will not get a response code authorizing a withdrawal unless the
transaction has been recorded in the authorization system. The account system
may well be caught up later. The funny thing is, ACID is a property of
individual database systems and it has absolutely nothing to with a question
of whether two separate ledgers are guaranteed to be changed together or not
at all. That would be the job of a distributed transaction coordinator - and
those really are not used very much in banking. Instead there is a protocol of
credits and debits and a settlement process to work out the exceptions. Maybe
this is what the article was trying to say up to a point but they sort of
confused the issue between the point of view of the ATM and the accounting
systems of record.

~~~
Maxious
> just spewing out money to every piece of plastic calling itself a card when
> your network connection has been dropped isn't really a recipe for success.

Well they do this in Australia...
[http://www.news.com.au/money/banking/computer-glitch-hits-
cb...](http://www.news.com.au/money/banking/computer-glitch-hits-cba-
customers/story-e6frfmcr-1226014261756)

> "People were running past me screaming 'Free money! Free money!'," Punchbowl
> Pharmacy manager Feriale Zakhia said of the people using a nearby ATM.

> "Everyone was so happy. They were running around with huge smiles."

> [A technical problem] forced the bank to put all of their ATM machines into
> offline mode. Customers had no access to their account balance but were
> still able to withdraw money - more than their accounts held.

> Those withdrawal limits are up to $2000 a day for holders of keycards and
> debit Mastercards.

> "No one has received free cash," Mr Fitzgerald said. "What they've done is
> overdrawn their accounts. We will be following those people up and
> recovering that money."

~~~
timv
Leaving aside the fact that I wouldn't trust a newspaper to for the technical
details of something like this, nothing in the article contradicts what
_jeremyjh_ said.

In that case, the ATM was disconnected from the accounting system and allowed
withdrawals up to a set limit ($2000), but it (probably, the article is
unclear) was still connected to the authorization system.

It (probably) still checked your PIN, and checked whether your card had been
cancelled, etc. It just didn't connect through to check your balance.

As Jeremy said _if for some reason we can't reach the checking account system
we'll authorize up to $xxx total for the day on a stand-in basis_

In this case _some reason_ == "[A technical problem]" and _xxx_ == $2000

~~~
diroussel
Which in turn agrees with the point of the original article. ATMS use BASE not
ACID as it's more profitable to be available.

~~~
GFischer
Those humungous overdraft fees will definitely be profitable :)

BTW I've been scammed out of money by ATMs before - money was withdrawn from
my account but some system jammed and I didn't get the money - and the bank
was awfully uncooperative.

So far I've had more luck with the "money under the mattress" method than with
banks - and I wasn't trapped in the "Corralito" or other bank-aided money-
stealing schemes

<http://en.wikipedia.org/wiki/Corralito>

~~~
gbaygon
I see that you are from Uruguay, where you in Argentina at Corralito's time?
ITT Uruguay is more trustworthy in banking terms.

~~~
GFischer
Uruguay had a smaller Corralito (and I was just starting out at the time, so I
had no money in the bank).

Ecuador and Brazil also had their own versions. In the Uruguayan version, they
didn't forcibly exchange the money, but they froze all bank assets for 3 years
(losing out on interest, investment opportunities, exchange rates, etc...).

Uruguay is more trustworthy (especially with foreign investment) but it's not
above such things.

Currently there's a big scare due to the huge exchange rate disparity with
Argentina - which has an "official" exchange rate and a "real" exchange rate
which is almost double the official one, and makes Uruguay non-competitive.

Edit: you're from Argentina, that's obviously not news for you :)

------
exabrial
Sorry, but Eric Brewer is wrong. He seems to be implying that there was some
sort of intelligent design behind the software at banks that lead them to
choose BASE. This couldn't be further from the truth.

Banks are one software kludge after another in attempt to not rewrite
something new or offer the consumer anything of value... while paying out the
butthole to whatever vendor has his arm shoved so far up your ass you can't
ever migrate from their platform without colon replacement surgery.

So while it's a nice thought "Hey look Banks/ATMs are BASE" this was by
complete accident through years of incompetence and corporate bureaucracy, not
by any sort of engineering choice.

~~~
felipesoc
That's not true. I've worked for a company that makes banking software and we
did tons of migrations. The company itself changed its own software from RPG
in as400 to windows forms and now it's all web with servers in Java and .Net
(they use a middleware language which generates in every new platform). They
even made a transitional install for one branch of one of the most important
banks. The branch used my former company's solution for a few years until they
could use their own software, which had to be adapted for the new market.

Banks used to work on paper an did fine, software migration, although can take
some years, is no problem for them.

~~~
natermer
Each bank is different. Some banks have shitty IT and other ones have dynamic
IT that can adjust to fit changing realities.

The reality is that you and Eric are right.

Banks depend on techniques based on double book keeping accounting to
reconcile accounts at end of day. Different data about transactions are stored
in different places by different organizations and they compare books to make
sure that balances are correct.

You cannot depend on every transaction to be recorded perfectly. You must have
the ability to compare books and reconcile accounts. This is simply how the
world works.

Trying to make every perfect and depending on storing data in a central place
with the assumption that it's always going to be consistent is too much of a
liability. It doesn't work because the systems required by modern financial
systems are incredibly complex and availability during markets is the highest
priority. You ARE going to have faults and you ARE going to have problems. The
ability to take hits gracefully and give yourself time later on to fix stuff
after the fact must be built into your systems.

------
aneth4
This is a little misleading. Transactions are used primarily to prevent
inconsistent data, not global data consistency.

The ATM network is distributed and eventually consistent, and financial
transactions in general are not real time.

Within an ATM or within a bank you can be damn sure transactions are used
widely to prevent inconsistent data.

------
jacques_chester
This is one of those cases where deciding on whether the system is ACID or not
depends entirely on where you draw the system boundaries.

If you draw it at the boundary of the central general ledger, it's going to be
ACID.

If you look at the way transactions pass through several intermediate systems
(each of which is ACID) en route, I guess it could be called BASE.

~~~
SoftwareMaven
I think the point is that people who say it is impossible to build a banking
system without full transaction support for every action are ignoring the
reality that transactions are not guaranteed to occur, but the actions
themselves are. If, at a level of abstraction, the system can be said to be a
BASE, then it is probably true that the underlying data stores are not
required to be ACIDic. Whereas everything I've read has always said "you can't
do banking without ACID".

Whether that is true or not is a much deeper discussion that three or four
paragraphs on a blog.

~~~
jacques_chester
> _If, at a level of abstraction, the system can be said to be a BASE, then it
> is probably true that the underlying data stores are not required to be
> ACIDic._

I don't think this follows. While the system as a whole might be eventually
consistent (where consistency is defined as: what's in the General Ledger), it
doesn't follow that you relax constraints on the parts.

The individual components are generally ACID and the steps to move data are
ACID as well. The only thing that prevents the whole system from being ACID is
that transactions don't go immediately from POS/ATMs/card clearance/cheque
clearance into the General Ledger, but instead must go through a series of
intermediate transactions. But those intermediate transactions must,
themselves, be ACID.

That's why I talked about how the boundaries matter. The final central
accounts are ACID, the subsystems are ACID and the data transfers are ACID.

edit: though to contradict myself, I expect that there will be counterexamples
in different banks where some stores or steps will not be strictly ACID, but
will have been "good enough" or with sufficiently-acceptable workarounds that
they haven't been upgraded. I don't think this fatally breaks my argument, but
YMMV.

~~~
dllthomas
I think "if the system as a whole doesn't require ACID, maybe the pieces
don't" is correct and useful, but it still requires looking at the pieces and
seeing if that's the case. In this case, I think that the system is relying on
the ACIDity of some components to ensure Eventual consistency - it's
conceivable that an alternate method might not, but one would have to be
proposed and evaluated.

~~~
jacques_chester
I think this is reasonable; but then the problem becomes that while
consistency may be _eventual_ there are nevertheless fixed deadlines to meet.
Soft realtime consistency isn't good enough when the annual report has to be
printed.

Personally, I feel that ACID is an abstraction achievable only within single,
non-distributed systems. Not a very compelling insight, I hear you say.

Well no, but ACID is a tremendously advantageous state of affairs and I feel
it should be surrendered only begrudgingly. I think it is better to repair it
than to abandon it wholesale at the first sign of mild inconvenience.

Even though it is, in a physical sense, untrue, it is a _useful_ untruth.
Newtonian physics is wrong. It's also what we use to build bridges.

------
ryanobjc
I'd tend to agree on a general sense - yeah when faced with failure in a WAN
environment, it makes sense to attempt to continue when at all possible.

However, banks are one of the biggest purchasers of ACID systems... They still
bet heavily on oracle, and when they want their accounting systems to run, and
balance transactions, they dont rely on "BASE" systems. Banks are also heavily
dependent on business cycle and batch processing. Daily batches are common in
credit card processing (end of business day settlement for example), and also
in general bank systems.

Also eventual consistency means different thing. Some systems have a
"eventually inconsistent" property to them (eg: Cassandra/Dynamo), and I'm
pretty sure banks would NOT be ok with that.

I respect Brewer, he is a smart guy, but he is extrapolating too much from a
small fact that is ATMs will sometimes dispense cash (of what amounts? $200?
$1000? or maybe just $20?) when remote communications are interrupted or
broken.

~~~
seanmcdirmid
I don't think Brewer is stating an opinion, rather a fact. Nor is it very
surprising. I'm sure banks have huge IT operations where parts rely on ACID or
BASE depending on need. But any massive distributed system can't really expect
ACID to work very well.

I'm pretty sure "eventual consistency" doesn't mean "eventual inconsistency"
under any reasonable interpretation.

~~~
ryanobjc
I dont think anyone would claim a massively distributed system should be fully
consistent or ACID.

As for the eventual inconsistency remark, this is from the original authors of
Cassandra at Facebook. They did not extend the use of Cassandra because node
flaps and packet losses caused nodes to be inconsistent and have old data.
Bringing that data back up to date was very difficult, since the anti-entropy
algorithms were too expensive to run frequently. They also found that the
R=W=2 was too costly in terms of performance, and well you know the rest :-)

~~~
standel
Do you have the reference?

------
Aloisius
Citation needed.

I've never heard of financial systems being eventually consistent, certain not
ATMs which need to know your exact balance and how much money you've taken out
already today.

~~~
artsrc
[http://www.abc.net.au/news/2011-03-01/cash-spews-out-of-
comm...](http://www.abc.net.au/news/2011-03-01/cash-spews-out-of-commonwealth-
atms/1962848)

I studied ATM's at Uni, and they have been like that for over 26 years.

As someone else on the thread pointed out, they do authenticate you, and
record the transaction in a aci(Durable!) database before they give you money.
But the system clearly has a degraded mode when the exact balance is
unavailable.

Also why do you need a balance to process a deposit?

------
stephen
A nit, but the remark of "auditing == everything is written down twice ==
double-entry accounting" is cute, but doesn't seem applicable to availability.

Double-entry is more of an internal (financial) system implementation detail,
and so an orthogonal concern to intra-system audits.

(It's not like one side of the entry is in one bank's IT system, and the other
side of the entry is in the other bank's IT system.)

(...speculating further, I really doubt the OLTP systems of banks are double-
entry anyway.)

~~~
PeterisP
Funny, but "It's not like one side of the entry is in one bank's IT system,
and the other side of the entry is in the other bank's IT system." is actually
false in interbank deals such as correspondent accounts, money market or forex
deals.

If a deal involves two banks, then the authoritative "other side" of the entry
will be in the other bank's IT system. Of course, you'll maintain some records
of what the entry should be in your opinion, but they won't always match as
you don't have full info and you'll reconcile with data you get from the other
bank's IT system by (for example) SWIFT network.

And I've seen only double-entry OLTP's for the core system that includes
general ledger. Maybe no all of them are built that way, but I haven't seen
such examples.

------
ExpiredLink
The article is not entirely wrong but misleading. The two important aspects
are authorization and limits.

Authorization: An ATM does not issue money without authorization which is done
by some 'central authority', not the ATM.

Limits: In some corner cases you may be able to exceed your (daily, weekly,
monthly) limits. But, as the article points out, this doesn't imply financial
inconsistency.

------
jasomill
As someone who's had to dispute NSF charges for reordered transactions, I
question the author's claim that ATM operations commute. Specifically, assume
a $0 beginning balance. Then "withdraw $20 then deposit $200" might yield an
error, $0 cash, and a $200 balance where "deposit $200 then withdraw $20"
would net $20 cash and a $180 balance.

Or, as in my disputed case, $20 cash and a $140 balance, because the deposited
check hadn't actually cleared by close-of-business, so end-of-day processing
assessed a $40 negative balance fee despite the fact that the deposit
"eventually" cleared. The first manager I discussed this with had the audacity
to claim it was _my fault_ for not somehow recognizing that the portion of the
deposited funds the ATM made available for immediate withdrawal _by design_
were not, in fact, available for immediate withdrawal.

------
trotsky
This seems like a pretty bad strategy to sell nosql etc. What are we supposed
to believe, that every ATM downloads account numbers, pin hashes and balances
for every account in their network?

Banks have chosen consistency over availability regularly as they've been able
to rebuild their systems over the decades. 40 years ago, if you had a credit
card the place just took it, copied it down and trusted you and the bank were
good for it. Try to get anyone to take your credit card if the network is down
now.

IMO banking culture and standards probably were direct motivators of many of
the ways traditional systems were designed. They were some of the earliest
adopters of IT. It may not be accurate to say banks or ACID. It may be more
accurate to say ACID is banking.

------
rjempson
This is an example of why analogies should never be used (except perhaps if
trying to explain a concept, not something tangible).

ATM software has nothing to do with the backend storage of the data. My point
is that I bet when the data is actually written to durable storage in the
backend, the set of data being written will be wrapped in an ACID transaction.

------
chiph
I think it's funny that the photo in the article is of two guys _stealing_ an
ATM (note the pantyhose masks).

------
EGreg
Banks can have availability over consistency because we have enough laws and
protections in place to reverse any fraudulent transactions.

With BitCoin, such an architecture could be abused rather badly.

