RaptorCast: Designing a Messaging Layer(category.xyz) |
RaptorCast: Designing a Messaging Layer(category.xyz) |
I’m not seeing how the new design affects the throughput needs, but I’ll say this:
Except for highly controlled environments (OS, NICs etc), you will run into perf issues with UDP-based protocols much sooner than with TCP, even if you’re just pushing zeroes. Packet switching is much more difficult to optimize.
If you only use sporadic messages without backpressure, and you’re willing and able to handle out-of-order messages and retransmission logic, by all means, use UDP. Like for realtime multiplayer games, it makes sense.
For high throughput on diverse platforms and hardware, the story is very different. Yes, even with Quic. I learnt this the hard way.
All that said, I’m very curious what the results are. Is this designed fully deployed, and if so in what kind of environment and traffic patterns? Even better: benchmarks/stress tests would be fantastic.
RaptorCast uses erasure coding to break a block proposal into smaller pieces with plenty of redundancy to allow for omissions. This means that if you receive sufficiently many chunks, you can decode the block proposal (no matter which of the chunks you received). The redundancy factor can be tweaked, but it’ll likely be >2x, to allow for networking issues and faulty/malicious nodes. Furthermore, the blockchain can make progress as long as >2/3 of the validators receive the block proposal and are honest. This means that at least in theory, you should be able to tolerate a lot of packet losses.
Re throughput: Monad has 2 blocks / s, each 2MB in size. So even with a redundancy factor of 3x, each validator only has to send 12MB per second.
Re backpressure: Not really an option for blockchains. If you have 100 peers and one of them is too slow, what are you going to do? If you back pressure to slow down consensus, you slow down the entire blockchain even though most peers are fast. There’s a recent paper about this problem: https://arxiv.org/abs/2410.22080.
What’s important is that the amount of bandwidth required per validator remains constant in RaptorCast, no matter how many validators are part of the network. And you always just need one round-trip to broadcast a block proposal, as opposed to Gossip protocols that may involve more steps and have higher latency.
Detect, and remove them from the set, so that if the bottle-neck is upstream of them you're relaxing pressure on switches etc that may be shared with nearby nodes, reducing the chance that two of them are suddenly too slow...
If your packet loss is due to your traffic overwhelming a queue at any intermediate hop, sending more redundant packets would be aggravating the problem instead of solving it.
Are you running this on top of something providing congestion control?
Really, the only thing that's missing from UDP is (optional) backpressure.
A lot of software can handle out-of-order datagrams with no performance penalty (like file uploads, etc.). This is especially annoying when you're operating in an environment with link aggregation where the interface insists on limiting your bandwidth to a single link.
For instance, TLS is stateful. If you want to deliver unannounced (stateless) datagrams to a server, you’d technically be able to use the servers pubkey but you’d have to sacrifice forward secrecy. Making business logic with dropped packets and OOO delivery is also very difficult depending on domain.
Personally I would like to see faster bootstrap and especially resumes of streams to eliminate handshake cost. For instance:
- resume with last shared secret for 0rtt handshake (I believe Quic supports this)
- allow the dialer to queue/send data before ack (this is an API-breaking change on sockets, but worthwhile imo - it makes socket FSM on app layer easier to use because it never needs to be nil
The lack of congestion control seems significant too. Most message-oriented protocols layered on top of UDP end up adding that back at the application layer as a consequence.
I've built a decoder for online codes (very similar) in the past and learned that it's best to just to a Gauss elimination step regardless of the degree of equation/packet/message you get. Also, since the system is very sparse it's best to not have a matrix representation, even if you're using bitvectors, as you'll be scanning rows which have mostly zeros. Just use a representation for sparse sets.
Anyway, just my 2cts