Distributed Computing at Airbnb

Distributed Computing at Airbnb(nerds.airbnb.com)

60 points by AirbnbNerds 13 years ago | 10 comments

uptown 13 years ago |

I'm always impressed by the technical write-ups that come from Airbnb. I'm also always astounded by the complexity of what's happening on the back-end for what appears, on-the-front-end, to be a relatively straight-forward CRUD-type system.

old-gregg 13 years ago | |

It is a relatively straight-forward CRUD-type system, it's the amounts of data it deals with coupled with (I am speculating) high volume of traffic what makes it tricky to continue to look like a simple CRUD app from the outside.

The complexity could have been reduced dramatically if they were hosting on a modern dedicated hardware configured for their workloads. Successfully reaching this kind of scale on AWS (or any other massive cluster of exhausted and unreliable virtual generics with limited I/O) requires this kind of software wizardry on the back-end.

It would have been even more complicated if all of AirBnB was running on a cluster of randomly dying first generation iPhones.

Want to avoid layers and layers of software complexity and paying for brainy guys to run it? Pay for a nicer hardware then. Sometimes it's cheaper (and easier), sometimes it's not.

xal 13 years ago | | |

I wish I could upvote this twice. Easily one of the most insightful comments I've seen on HN recently.

keeptrying 13 years ago | | |

This is really true and most every financial services company knows this.

They spend so much money on buying huge machines which makes writing performance intensive code similar to writing regular code - ie if its sucks it'll still perform pretty well.

Think of the biggest financial services companies you know - all of them do this.

xiaoma 13 years ago | |

The front end is pretty cool too. Check this out, just released:

https://github.com/airbnb/rendr

lightyoruichi 13 years ago |

You need to move away from Posterous. It's closing down in less than 15 days.

AirbnbNerds 13 years ago | |

We've got something in the works.

jaredwilliams 13 years ago |

I'm curious what other components you use around these services. ZooKeeper for properties? What do your Storm spouts feed from (Kestrel, Kafka, ...), and what do your bolts write to? Are you using something for distributed locking?

The article is a good start, but leaves a lot to the imagination!

Diamons 13 years ago |

Storm Framework links to Hadoop, just an FYI.

AirbnbNerds 13 years ago | |

Thanks, fixed.