Ask HN: Can you help me monitor the Internet?

8 points by adrienjarthon 9 years ago | 7 comments

Hi, I want to build a service monitoring the Internet and I need your help to build the best tool possible.

Below is a rough idea of what I have in mind, please comment with any idea or opinion to refine my draft. How would you do it? What kind of tech/tools do you recommend? What did I miss? Any help will be greatly appreciated ;)

Draft: - A couple of servers would run various checks and store the results in a central database, a public page would be available at internet.rip and display the status of the Internet with details on ongoing outages and their estimated impact. - Check most DNS servers (TLDs, ISPs, Public), public routers status (not sure how to do this, looking glass?), public BGP routes for instabilities or missing routes, Internet Exchanges status, route, latency and packet loss to most IPs (best would be to find the smallest set of IPv4 to monitor passing through the highest number of different routers to be able to draw a map of network paths and their status) - Keep the infrastructure simple: don't want to need a server on each AS so it would best to monitor from the outside as much as possible, otherwise this could be a different approach with an open-source deamon people can install on their servers to contribute, but that's much more complicated. - Make internet.rip (the website) as resilient to network issues as possible and for this I'm thinking CDN, multiple DNS, very long caching of the public page, service workers and multiple API endpoints to fetch the data reachable through JS directly using their IP so it could even work without any DNS if you have the site in cache. - Keep internet.rip free by limiting the cost, the infrastructure will be supported by me. Maybe add a discrete donation button but no ads.

You can also reach me at adrien@updown.io if you have anything to discuss privately. Feel free to register on the landing page to know when it launches (http://internet.rip)

jonny_storm 9 years ago |

This is a hard problem. Internet tomography aside (an even harder problem), route-servers and looking glasses everywhere expressly forbid automation, and any network administrator, virtually anywhere, can break your reachability testing. Add to this the myriad complications of BGP selectivity (Why do people insist that "net neutrality" is in danger when it never existed as stated?), and network operator participation becomes almost unavoidable.

While I understand that tools such as http://www.internettrafficreport.com and http://internetpulse.keynote.com are not enough, it isn't at all clear to me how any single actor could provide better monitoring. I admit this sounds pessimistic, but I've thought deeply about this and similar topics for many years, and I'm all too familiar with the obstacles to (and poor architecture underlying) Internet polling.

However, I do see a clear need for aggregated monitoring data--many sources, re-published succinctly in one easy-to-find place--and I would certainly donate to such a cause.

adrienjarthon 9 years ago | |

Yes indeed, that's what I noticed too and if I can do it with existing sources, just presenting them in a cleaner way that's definitely what I will do. I don't want to re-invent the wheel, just do as much as I can to make the information available.

About the routing part other people mentioned https://atlas.ripe.net/ which seems to be doing a very good job at that thanks to their 4000 probes distributed across the world, If I can use their data it'll already solve a good part of the equation.

Thanks for your feedback jonny_storm!

kirankn 9 years ago |

You may want to check out ripe.net

adrienjarthon 9 years ago | |

Yes, I already had a quick look and there's a lot of interesting stuff here, I'll definitely need it. Thanks!

benzimmer 9 years ago | | |

You especially might want to have a look at https://atlas.ripe.net/.

Here you can find a podcast interview with Internet Hall-of-famer Daniel Karrenberg[1] about RIPE and the ATLAS project (don't worry the page being in german, the interview is in english language): https://requestforcomments.de/archives/280

[1](http://internethalloffame.org/inductees/daniel-karrenberg)