Migrating Messenger storage to optimize performance(code.facebook.com) |
Migrating Messenger storage to optimize performance(code.facebook.com) |
Until sometime in 2016/2017 it was possible to look for specific things at whatever time (e.g. 2011) and now even things from <3y ago don't return anything.
I'm a bit unclear as to how this process resulted in 90% decreased storage use.
As far as I understand, Google doesn't have a bunch of tools that merely work together, they have one huge system with different bits that _live_ together, so much that separating and open-sourcing them is cool but won't give you the same thing as being from inside:
- They use Blaze, a build system that integrates directly with the object store. They open-sourced Bazel as a kind of equivalent, but the build system won't shine unless you have an integrated object store and an integrated vcs client - They have open-sourced Kubernetes, a successor of what they were using for they were using internally for cloud management - They have open-sourced LevelDB, a successor of the fundamental brick they are using for BigTable
So in a way LevelDB isn't used as-is inside Google, but its spirit is in use at a fundamental level by pretty much everyone
A Brooklyn based artist did this recently and made a book of his conversation history (and covered an apartment with some of the pages). You can find a bit more information and pictures at https://www.halfspace.uk/messages-2011-2017
We've asked you many times to stop posting unsubstantive comments to HN. If you can't or won't stop, we will ban you.
[1] https://code.facebook.com/posts/989638804458007/introducing-...
I get that MyRocks is truly amazing, but I'm wondering what issues they were facing with HBase. I heard HBase was picked over Cassandra (developed at FB) because it was strongly consistent vs Cassandra's eventual consistency.
I also heard they had an issue with network flaps causing unrepairable inconsistency. I never got the full scoop on this.
I had worked with the engineers directly responsible for the HBase choice back when. All water under the bridge now it seems.