Ask HN: Is distributed systems research slowing down?

22 points by civicsquid 1 year ago | 10 comments

In the very late 1990s to the early 2010s, it seemed like the volume of distributed systems research was enormous.

If I search for papers during that period, I see everything from theoretical stuff like Chord to industry results like Google BigTable/Spanner and Facebook Cassandra/TAO, and of course people trying to figure out how to implement Paxos properly. Figuring out how to build systems to support web scale and mobile was both necessary and cool.

Now, I don't hear about distributed systems as a 'frontier' to be explored so much as an implementation detail.

It feels like there isn't as much innovation, but maybe that's because I'm not looking in the right places / in the right circles. I imagine distributed systems would be extremely relevant given the focus on AI and distributed consensus (crypto).

Could anyone better read than myself point to the 'hot' distributed systems work that is going on to support AI / distributed consensus today? Perhaps I just need better keywords.

warner25 1 year ago |

Regarding AI and distributed systems, this might not be what you have in mind, but take a look at federated learning. I'm currently a computer science PhD candidate at a small school, and a couple of our graduates in the past year worked on the fundamentals and applications of federated learning.

I came into this PhD program thinking that I wanted to work on stuff like the distributed databases that you listed, or the stuff they're built on like clock synchronization. I did my master's degree in 2017-2018 and I was fascinated by an "advanced databases" class that covered these things. Unfortunately, nobody in my department works on such things, and I agree with you that I don't hear much about that area anymore.

civicsquid 1 year ago | |

Good suggestion! I’ve seen federated learning come up before but have not explored it much.

This seems like a good direction for sure.

JSDevOps 1 year ago |

No, AI has nothing to do with this. The late 1990s to early 2010s were a massive period for distributed systems research, but that surge was driven by advancements in computing infrastructure and the growing needs of industry giants—not AI. From theoretical frameworks like Chord to practical implementations like BigTable, Spanner, Cassandra, and TAO, the focus was on solving complex problems in scalability, consistency, and fault tolerance. The efforts to properly implement Paxos were part of that journey. AI had little to do with the sheer volume of breakthroughs that came from this time; it was all about improving distributed systems, not artificial intelligence. Not everything needs to loop back to AI.

slau 1 year ago | |

Maybe there was an edit, but I don’t see how OP said anything was due to AI. They’re just asking whether distributed systems aren’t even more relevant today than during the 90s or 2000s as it seems AI could benefit from being distributed.

austin-cheney 1 year ago | | |

There was an edit. The topic reads much differently now.

civicsquid 1 year ago | |

Hm, I’m not sure what you saw in the post before my edits, but I think this answers “did AI motivate / help discover the breakthroughs we saw 20 years ago?” which I definitely agree would be a “no”.

Either way, before and after my edits the intent was to identify areas in which distributed systems researchers moved their focus to support areas such as (but not exclusively) AI.

The question comes from me supposing that “pure” distributed systems research has slowed.

musicale 1 year ago |

Everything is a distributed system now, and system designs are changing in important ways. As you indicate, we you are currently seeing a redesign of basically all computer systems from mobile to datacenter/cloud/hpc to better support AI workloads. I expect there are still many opportunities for distributed systems research.

austin-cheney 1 year ago |

If anybody needs a practical launching point to study distributed systems maybe I can help.

I have a browser/Node.js application that I worked on for several years that is a browser based peer-to-peer file system sharing tool. The application is massive in size, but its easy to tinker with, the build takes about 8 seconds, and the test automation is decent enough for anybody interested to get started. The idea is to invert the transmission model of the web from anonymous connections towards a server instead to trusted connections between owned computers and even restricted connections to different users (and their various devices) all in real time using sockets and streams and a tiered distribution model. https://github.com/prettydiff/share-file-systems

I am currently working on a web server application that allows spinning up many different web servers quickly and a monitoring dashboard that shows the various servers, their connected sockets, port management, and various other things. This all started with the idea that you can proxy and redirect anything super efficiently where a proxy in Node.js is as simple as:

    proxy.pipe(socket);socket.pipe(proxy);

fragmede 1 year ago |

I'm sure the next generation of supercomputers (eg anything upcoming with Nvidia's GB200 architecture, Tesla's Dojo, etc) will have their own share of challenges that need to be solved, at their specific scale.

ldjkfkdsjnv 1 year ago |

Its a solved problem, unless youre at FAANG scale, you can just use cloud solutions to build distributed systems. Use their NO SQL dbs, queues, serverless functions, etc.