Ask HN: What are some good projects/resources to learn distributed systems?

54 points by gyani95 8 years ago | 14 comments

I'm a kind of person who likes to hack on projects to learn things. If I am learning a new language, I'll think of a problem that I want to solve. After reading some syntax I'll start hacking on a solution.

I have decided to learn distributed systems.

I think this weekend would be a good start to hack on things. I can continue later in the week I understand I won't pickup distributed systems over a weekend.

What resources do you recommend for one trying to learn distributed systems on their own?

jxub 8 years ago |

I'd recommend the book Designing Data Intensive Applications, by Martin Kleppmann and as for online resources, check out http://book.mixu.net/distsys/single-page.html for a throughout overview.

gyani95 8 years ago | |

Thanks for the suggestion. I've added it to my list. Do you recommend some good first apps for learning distributed systems?

daoran 8 years ago |

Beside Martin Kleppmann's wonderful book, I'd also recommend MIT 6.824 distributed system course, you can find lots of resources on the course website.

cottonseed 8 years ago |

You might find some of these references useful:

Resources for Getting Started with Distributed Systems https://caitiem.com/2017/09/07/getting-started-with-distribu...

Distributed systems theory for the distributed systems engineer http://www.the-paper-trail.org/post/2014-08-09-distributed-s...

Distributed Systems for Fun and Profit http://book.mixu.net/distsys/

Notes on the Theory of Distributed Systems http://cs-www.cs.yale.edu/homes/aspnes/classes/465/notes.pdf

simon_acca 8 years ago |

Aphyr, of Jepsen fame, published an outline for a distributed systems class that makes for a good reference on what is relevant for most modern distributed systems: https://github.com/aphyr/distsys-class

gyani95 8 years ago | |

Thanks for this! This sounds lovely and is 12-16 hours long. Fits my schedule :D

iampims 8 years ago |

This Github repo is a pretty good start to get familiar with distributed systems:

https://github.com/donnemartin/system-design-primer

gyani95 8 years ago | |

I've been through this. It's a good starting point and can be fairly useful for interviews. It gives a good high level overview of systems, enough for you to talk about them.

Maybe I can go through this again and dive deeper into topics that interest me.

convolvatron 8 years ago |

do you want to learn how to design distributed systems from first principles or build systems from pre-existing distributed components?

unfortunately these are both called 'distributed systems'

#1 is alot more interesting. #2 is what people are hiring for

gyani95 8 years ago | |

I'd like to say both! I guess I can get started with the second one and then move onto the first one?

Maybe after playing with a few components I can look into their internals.

dsnuh 8 years ago | | |

For #2 - Maybe try setting up your own Kubernetes cluster on a cloud service provider, without using shortcuts? Perhaps etcd and the Raft protocol would be interesting and helpful in some cases, but leans more towards #1.

For #1 - CAP Theorem, read through aphyr's blog, google concepts/details/jargon you find in white papers on different of the various distributed consensus protocols, etc.

lambda_func 8 years ago |

Some stuff you can build to learn more about distributed systems (you can start simple, do research, and then iterate to build a more comprehensive solution): build a distributed file system, a distributed lock server, a consensus algorithm (paxos, raft, etc.), two/three phase commit protocol, distributed hash table, etc.

srkiranraj 8 years ago |

MapReduce paper, though published in 2004, it is foundational, relevant and very well written paper.

https://ai.google/research/pubs/pub62

P.S: it has close to 25k citations.

brudgers 8 years ago |

Erlang?