I’d try to use that first, because sadly Nebula and ZeroTier don’t have a relay/TCP/HTTPS fallback option and still “have no plans of implementing one” as of Mar 2023, which leaves you out of luck as soon as you encounter NATs or try to access your cluster from an airport/hotel wifi

Maybe you know of other mesh/p2p VPNs that do support TCP fallback though? Would be great to see some alternatives in this area

I evaluated it for one of my projects but it was not fitting the bill for reasons outside of their control, https://www.defined.net/

The common wisdom on dist is drastically wrong. I was at WhatsApp and going to Erlang conferences and people would say don't run dist clusters with more than N hosts (N ~~ 100), and we were running dist clusters with 10N.

You probably want to adjust socket buffers and dist buffers.

WhatsApp did use something they called wandist to provide cross-cluster connections and more specifically to determine which clusters communicated with which other clusters. That's useful if the node counts are high enough and memory per node low enough that socket buffers are significant and you have a lot of nodes that don't communicate with each other; dist wants to make a full mesh, but you might be able to avoid it other ways? Wandist was also useful because pg2:join had scaling challenges because of contention on cluster-wide locks from the global module, but the new pg (originally contributed by WhatsApp) addresses that. Wandist also added regional affinity for the cross-cluster equivalent of process groups, and as mentioned below, reliable messaging layered on top of multiple connections.

Dist doesn't hide any imperfections of your network though. If you routinely have tcp connections that can't make progress, you will have dist reconnects and have to deal with the unknowns there (wandist layered multiple connections and reliable messaging on top of dist, if a connection stopped functioning, messages would be sent on a second connection; at the cost of using more memory). If your network gets throughput limited, you can see net_adm:ping times of tens of minutes, which is exciting too. All that said, you really do want a reliable network when running distributed systems of any flavor, and building mostly reliable networks is possible, but it's a choice. You'll still need to deal with ocassional issues even with a mostly reliable network; WhatsApp used to routinely catch network problems at our hosting provider before they did; and we had to build a lot of blackbox monitoring and diagnostics for their network.

Tailscale makes the ec2 instances feel just as local as the LAN machines. I love having easy SSH connectivity to my laptop and EC2s from my windows machine. There's no tunneling or bastioning or key management or any of that nonsense. And without extra network hops (nodes direct-connect whenever possible) I don't have extra nodes in the route to steal bandwidth time or cost.

But maybe small-scale isn't what you're after : I haven't done the work to figure out the at-scale bootstrapping process (I still install the systemd or whatever units and google auth each machine manually).