Show HN: Syq – copy files between machines fast (better than rsync)

Show HN: Syq – copy files between machines fast (better than rsync)(greaber.github.io)

41 points by greaber 4 days ago | 51 comments

I got frustrated with the slowness of rsync and made an alternative that works faster by using multiple parallel connections, direct encrypted TCP when available, and other optimizations. I also added cool features like the ability to maintain a persistent ssh connection to the server for fast one-offs, the ability to download to your laptop while working in an ssh shell on a server, and the ability to do direct remote-remote transfers without forwarding your ssh agent (by using restricted ssh keys on the receiver that will only execute a specific request signed with the key on your laptop). It is also designed to be more robustly/flexibly scriptable than alternatives.

hxseven 3 days ago |

I thought I'd share a more positive comment. I work on a tool in a similar area (moving stuff around) but on a higher level. I currently use rsync or rclone for the file moving parts. So to me your tool looks interesting and promising, thanks for sharing it :)

Some suggestions for improvement for your website/Github:

- I wanted to understand how it differs from rsync. A comparison page would be helpful.

- A comparison to rclone would also be interesting, as it also uses multiple parallel connections.

- As it target more advanced users I think it would be helpful to provide more advanced technical insights on how your transfers exactly work. Also how are edge cases handled, what tests are done? Moving and copying files is critical, and you need to inspire trust in your project among users.

greaber 3 days ago | |

Thanks! Yeah, I was a little surprised (maybe I shouldn't have been) at how negative some of the comments were. I will think about what additional explanations I can provide that will help people. Regarding rclone, I have also spent a bunch of time looking at faster transfers to and from S3-compatible storage, but that problem is somewhat different, and I didn't look in depth at how rclone is implemented. FWIW, in my experience, s5cmd is generally a bit faster than rclone even if you tune rclone options. (But rclone is more flexible.) My guess is that s5cmd is already close to the performance ceiling, but I don't really know.

As for edge cases, I will also try to document that more. In general, when I'm not sure what semantics to go for, I try to either copy what rsync does or do something safer. For instance, one thing I am looking at now is the best way to handle cases where a source path cannot be represented on the target filesystem or where two source paths would collide (e.g. due to unicode normalization or case insensitivity). Tentatively, my inclination is to try to fail before copying any files when possible. Currently, syq doesn't do this (but neither does rsync).

rdbl27 3 days ago | | |

Don't despair; new tool rollouts are always a hard sell -- especially when replacing a beloved classic.

For me, the killer feature would be "ease of use."

Sure, I _can_ make rsync do any number of rarely-used optimizations, if I feel like studying the manpage for half an hour and figuring out how to fit it to my exact use case.

If your tool has the same features but is automatically adaptive -- I'd use it.

Copying thousands of tiny files in deeply nested subdirs? Just works. Copying a huge file that's already encrypted? Just works. Copying a mix? Just works.

No special flags to set, zero config. It just works, optimally, every time.

"Reducing the user's cognitive load" is the killer feature here.

unsnap_biceps 3 days ago |

> I also added cool features like the ability to maintain a persistent ssh connection to the server for fast one-offs

rsync already supports this via OpenSSH's ControlMaster directive. Bonus, it speeds up every connection to that server rather then a single tool's.

> the ability to do direct remote-remote transfers without forwarding your ssh agent (by using restricted ssh keys on the receiver that will only execute a specific request signed with the key on your laptop).

You can control your agent forwarding in your local ssh client configuration. And with modern ProxyJump, you don't need to forward your agent at all regardless of how many bastion hops are between you and your final target.

greaber 3 days ago | |

Yeah, ControlMaster is cool and was an inspiration, but since the syq client is always talking to the same receiver on the server, we can save a few round trips that ControlMaster still needs, so syq persist feels faster. This architecture is also what enables reverse mode, where you can download to your laptop while working on the server.

ProxyJump doesn't help when you want to copy files from server A to server B without giving server A an agent that can do arbitrary things on server B. There is really no alternative to using a restricted authorized key on server B for this scenario.

AceJohnny2 3 days ago |

Thank you for showing this. I'll echo what some others say: if you're claiming to be faster than rsync, you should demonstrate why front-and-center. There have been decades of documentation on how rsync optimizes transfers, so it's a lot to live up to.

As an aside, It's perhaps an indictment of our networking landscape that multiple parallel connections between 2 specific machines would accelerate a transfer. I would've expected a single TCP connection to be able to saturate a line. Or perhaps the parallel connections seeks to amortize the per-file setup overhead?

cynicalsecurity 3 days ago |

I'm eager to replace battle-tested rsync with a new half-baked tool that no one uses for a tiny speed improvement. I'm also going to enable automatic updates of that tool too.

greaber 3 days ago | |

I'm seeing 5x speedups on my real workloads, and you can see some synthetic benchmarks here or run your own https://greaber.github.io/syq-bench/

throw1234567891 3 days ago | | |

what's a "synthetic benchmark", claude came up with those numbers by looking at the code?

xorcist 3 days ago | |

"Tiny" speed improvement? According to the benchmarks, it's almost 10x the speed of rsync!

Really serious benchmarks never contain any information about network characteristics, options used or buffer sizes, and these are indeed very serious. The "details" button produces huge bar graphs. No disappointment there.

Absolutely looking forward to 10x my speed, perhaps even transferring gigabytes per second on my gigabit link.

xz18r 3 days ago | |

You live up to your username!

mika6996 3 days ago |

Why is this better than rsync?

Zenul_Abidin 3 days ago |

Wow, interesting! How is the performance compared to rclone though?

greaber 2 days ago | |

I did some benchmarks against rclone today. https://greaber.github.io/syq-bench/rclone.html Syq was sometimes much faster and never more than a few percent slower. I didn't do any tuning of syq since it is meant to perform well without any special options, but I did tune a couple of rclone options. It's possible that more extensive rclone tuning would yield better results though. Please let me know if there is anything I need to try!

hikarudo 3 days ago |

Very cool!

How does the direct TCP mode work?

greaber 3 days ago | |

Currently, there is just a fixed range of ports that it will use if they are open and you don't pass `--no-tcp`. It does its own encryption over the TCP connections. Also worth knowing is that for some connections (e.g. if there are any dropped packets), you may get much better performance if you can enable BBR congestion control. https://greaber.github.io/syq/server-tuning.html#test-conges...

QuiCasseRien 3 days ago |

rclone has been the real fully featured ultra power alternative to rsync over the past 4-5 years.

How do you compare Syq to rclone ?

greaber 2 days ago | |

As for speed, see https://greaber.github.io/syq-bench/rclone.html. As for other stuff, it depends what you use rclone for.

jttnr 3 days ago |

Open page, read headline, read first sentence, emdash, close page.