I've been working withe a friend on a pytorch implementation of DownpourSGD, a distributed optimization algorithm. I ended up writing a blog post about our implementation and some problems we ran into along the way. It was a pretty interesting project to work on, would appreciate if y'all checked it out!

Post: https://jcaip.github.io/Distbelief/

Github: https://github.com/ucla-labx/distbelief