Globally Distributed Elixir over Tailscale(richardtaylor.dev) |
Globally Distributed Elixir over Tailscale(richardtaylor.dev) |
Reminds me of the intended use case of Nebula, which seems very similar to this. If you're interested in a bare-bones and totally self-hosted option, it could be a good choice here. https://github.com/slackhq/nebula
I’d try to use that first, because sadly Nebula and ZeroTier don’t have a relay/TCP/HTTPS fallback option and still “have no plans of implementing one” as of Mar 2023, which leaves you out of luck as soon as you encounter NATs or try to access your cluster from an airport/hotel wifi
Maybe you know of other mesh/p2p VPNs that do support TCP fallback though? Would be great to see some alternatives in this area
Edit: Yep, just found a reference to it: https://docs.zerotier.com/zerotier/troubleshooting/ (Sorry, no direct link, so you'll need to Ctrl-F and look for Relay)
I evaluated it for one of my projects but it was not fitting the bill for reasons outside of their control, https://www.defined.net/
The common wisdom seems to be to do federation of clusters rather than clustering multiple data centers together.
The common wisdom on dist is drastically wrong. I was at WhatsApp and going to Erlang conferences and people would say don't run dist clusters with more than N hosts (N ~~ 100), and we were running dist clusters with 10N.
You probably want to adjust socket buffers and dist buffers.
WhatsApp did use something they called wandist to provide cross-cluster connections and more specifically to determine which clusters communicated with which other clusters. That's useful if the node counts are high enough and memory per node low enough that socket buffers are significant and you have a lot of nodes that don't communicate with each other; dist wants to make a full mesh, but you might be able to avoid it other ways? Wandist was also useful because pg2:join had scaling challenges because of contention on cluster-wide locks from the global module, but the new pg (originally contributed by WhatsApp) addresses that. Wandist also added regional affinity for the cross-cluster equivalent of process groups, and as mentioned below, reliable messaging layered on top of multiple connections.
Dist doesn't hide any imperfections of your network though. If you routinely have tcp connections that can't make progress, you will have dist reconnects and have to deal with the unknowns there (wandist layered multiple connections and reliable messaging on top of dist, if a connection stopped functioning, messages would be sent on a second connection; at the cost of using more memory). If your network gets throughput limited, you can see net_adm:ping times of tens of minutes, which is exciting too. All that said, you really do want a reliable network when running distributed systems of any flavor, and building mostly reliable networks is possible, but it's a choice. You'll still need to deal with ocassional issues even with a mostly reliable network; WhatsApp used to routinely catch network problems at our hosting provider before they did; and we had to build a lot of blackbox monitoring and diagnostics for their network.
https://www.erlang.org/blog/otp-22-highlights/#fragmented-di...
There are also apparently other bottlenecks like funnelling everything through a single gen_server. (https://erlang.org/pipermail/erlang-questions/2016-February/...). Again, this may have been fixed since then.
I use it at home, and I'm not sure I need yet another third-party product to do some average networking setup, but I have to admit it's pretty decent at what it does.
So... is anyone using Tailscale on their servers?
Tailscale makes the ec2 instances feel just as local as the LAN machines. I love having easy SSH connectivity to my laptop and EC2s from my windows machine. There's no tunneling or bastioning or key management or any of that nonsense. And without extra network hops (nodes direct-connect whenever possible) I don't have extra nodes in the route to steal bandwidth time or cost.
But maybe small-scale isn't what you're after : I haven't done the work to figure out the at-scale bootstrapping process (I still install the systemd or whatever units and google auth each machine manually).
There's several talks from Erlang conventions; Rick Reed did several overview type presentations, and we had a couple other presenters on different bits and pieces. But, post 2014, we didn't present nearly as much (got kind of bogged down in post-acquisition stuff).