Scaling Financial Reporting at Airbnb(medium.com) |
Scaling Financial Reporting at Airbnb(medium.com) |
(Disclosure: I used to work at Oracle on Financials Cloud.)
"Any sufficiently complicated financial reporting system contains an ad-hoc, informally-specified, bug-ridden implementation of double-entry accounting."
It's such a powerful yet simple way of thinking about money that anyone who builds a billing or accounting system should be intimately familiar with it.
Though whether SQL is a functional language (or a programming language at all, if you're talking ANSI SQL) is a subtle question, I would at the very least not describe it as a traditional imperative programming language. I think this is an important distinction for the article because, contrary to what this article suggests, I've found SQL to be quite helpful for understanding functional and declarative programming concepts. That said, it might be a lot easier to express the types of tasks in the article as straight-forward functions rather than getting wrapped up in all this set-based talk in SQL.
When you program directly against Spark, you're effectively building SQL plans explicitly. It's both more indirect - instead of writing a program that does stuff, you write a program that creates a data flow graph that does stuff; and you have more responsibility for performance, for good and bad.
I think to get good performance, you simply can't think on a per-item basis. You need to orient your thinking towards what can be efficiently performed at the bulk level. Whether it's column scanning in HDFS, or index scanning in a RDBMS, you need to be aware of the engineering properties of the operators you're applying. Doing lots of things per-item is a recipe for blowing your budgets, whether it's cache, memory, I/O, whatever. You want to iteratively do a little work to lots of items, and then join, rather than lots of work to each item one at a time.
I've written something similar to the first part - extract raw data out of various source DB's using SQL queries then push it to our organisation's ERP product (SAP) using A2A messaging.
From my view SAP is black box but it handles the actual accounting/ financial logic part i.e Ledgers, product tracking, inventory management etc. Our Accountants all seem pretty comfortable using it.
We have been using it at the finance company I work for to maintain customer's ledgers.
Can it?
This post is a fascinating outlook of how things can be ran at big corp. I always wondered how big companies did accounting. This stuff can't be outsourced since it's so tightly coupled to your business logic.
Is there any tech conferences geared toward this stuff?
Any accounting saas that can scale indefinitely? As opposed to Xero?
Need to wait for AWS to release a version of Airbnb's system. ;)
Many big corporations use SAP or Oracle.
The one thing I hate about AirBnB.... Looking to rent a nice place for 10k 9 months in advance? Ok...we'll charge it RIGHT NOW.
Why not do it like Amazon and charge when the package is shipped (trip date has arrived)? I get the cashflow thing but come on... (100 bookings and counting...)
Of course AirBnB could forward the money to the host but doing it they way they do cuts out the problem entirely.
If you were talking about exporting the data from the event based system, then that's still possible, and I think it's something that our finance team may still be evaluating, but I can't speak for them.
In other words - because they can and should (and can afford to and not be hooked up to some third party vendor milking them for the rest of their companies existence).
In my opinion Superset actually fills a niche that was open for far too long.
AirBnB is using an extract, load and transform architecture. No mention of the hardware, data through put, whether they have a message broker/queue to ease the burden of peak volume but work.
I have a strong feeling that they could have 1.) Kept the system exactly how it is and done some performance tuning. But that's not sexxy anymore. Things are just supposed to scale. Which brings me to
2.) Moved transformation logic to its own server or multiple servers using a message broker and queue to aid the transfer of data between systems. It would have been more readable and could have been done in a mo the or less.
In summary I believe they should have put some effort in to keep SQL. Especially for the purpose of accounting because spark does not lend itself to readable logic.
Booking date (I understand that on that day, the booking is confirmed, but the guest's payment method has not yet been charged)
Dt Receivables from guests 100
Cr Future host payout 90 (Problematic: how can you pre-recognise a liability on your balance sheet? Unless this is an off-balance sheet account...)
Cr Deferred income 10 (Problematic: deferred revenue arises when you receive a pre-payment from customers and have a standing obligation to them to render the services)
Dt Cash 100
Cr Receivable 100
Dt Deferred revenue 10
Dt ??? ??? (seems to be missing to balance the double-entry)
Cr Payable 90
Booking date - no accounting entries, no effect on books. You did not render a service to the customer yet, nor fulfil or incur any obligations yet.
Payment received from guest - you received a prepayment for future services to be rendered, so now have a liability to the customer to fulfil this obligation, i.e., deferred revenue.
Dr Cash 100
Cr Deferred revenue 10
Cr Payables to host 90 (unsure about this one, as this goes into the whole gross vs. net revenue recognition discussion for marketplace-type businesses)
Dr Deferred revenue 10
Cr Revenue 10We still use a general ledger to book the outputs of our new financial pipeline, but I don't think we have a traditional business model (no traditional inventory management). That's more in the finance and accounting department though, and I can't speak much to that.