Some more meta-level questions if don't mind: 1. People can tar + ztsd infinitely many files and upload that archive normally and avoid the small-file problem altogether. If somebody cannot use this approach, I guess their use case involves much more real-time reading of each file after creating it. Your pipeline involves quite a lot of moving parts. Does the tool/sdk have enough support for observability/failure recovery? 2. Pardon my dumb question. The final multi-part upload in S3 still uses the same number of HTTPs connections as if the file were sent from clients, right?
Anyway, interesting read. Learned something new.
I don’t have benchmarks for anything like that runtime. 100 files of 100K is the biggest. For raptor the biggest issue is picking the ‘—rate-mbps’ which controls how fast packets are sent. If defaults to a pretty low value so if you have a Gbps connection it will need to be specified to go fast.
> Do you have benchmarks for the total round trip time as well, rom sending to full reconstruction?
One use case I can talk a bit about involved transforming a large (100s of millions of records) hash database to individual S3 objects. Each was around 2kb. It took eons, even with Lambda executions sharing the work at scale.
The full end-to-end time can be observed using ‘—confirm FILE’ which will make raptor wait until the multipart upload is completed (and confirmation notification received). The baseline number is around .5s but it varies. Larger blocks with repair packets in use take longest.
> People can tar + ztsd infinitely many files and upload that archive normally and avoid the small-file problem altogether.
Sure, is that works for the use case. But sometimes small files i S3 are the desired outcome. For example, to serve as web pages or a static JSON API.
> Your pipeline involves quite a lot of moving parts.
It seems simple to me. It’s just a couple queues and a couple Lambda functions (one very simple). The FIFO queue is important and does double duty - ensuring only one completion event gets through, and keeping them in order. I didn’t add much error recovery. Resending files is often the fastest path. There are some ways to get there though. If you use the ‘—no-prefix’ flag only incomplete blocks will be processed on repeated uploads — not a solution, but something that could be built on.
> Pardon my dumb question. The final multi-part upload in S3 still uses the same number of HTTPs connections as if the file were sent from clients, right?
Yes, you have this right. Since we don’t control the S3 API all we can do is show how much better it could be if S3 had an API like this. It could save an insane amount of electricity.
~ $ time aws s3 cp data/1kb/file_1.bin s3://${BUCKET_NAME}/test-upload/ --quiet
real 0m0.720s
user 0m0.593s
sys 0m0.070s
$ time ./s3upload data/1kb/file_1.bin s3://${BUCKET_NAME}/test-upload/ --quiet
Uploaded data/1kb/file_1.bin
real 0m0.327s
user 0m0.136s
sys 0m0.062s
~ $ time ./raptor data/1kb/file_1.bin $RAPTOR_ENDPOINT --server-key $RAPTOR_SERVER_KEY --client-key $RAPTOR_CLIENT_KEY --silent
real 0m0.303s
user 0m0.013s
sys 0m0.014s
Code for the s3upload tool is here: https://gist.github.com/mlhpdx/aaf99a468c06747dbea12f9639102...
The raptor tool has another nice advantages not possible with the AWS API:
Optional Acknowledgement. The `--confirm` option allow skipping waits for ACKs, which takes the run time down to just what it takes to put the outbound packets on the network. That makes uploads very fast, and the reliability is tunable with the `--overhead` option. On networks with significant packet loss this can make urgent uploads much, much faster.
The `--rate-mbps` controls the transmission rate. Sometimes slower is better, and having this option means not needing to setup bandwidth control using `cgroup` or similar.