Tus.io: Open Protocol for Resumable File Uploads

Tus.io: Open Protocol for Resumable File Uploads(tus.io)

72 points by chrisfarms 10 years ago | 28 comments

_urga 10 years ago |

It will need to monitor and induce at most 20-50ms bufferbloat when uploading, regardless of whether the traffic is high or low priority.

Try this demo for yourself:

1. Run "ping google.com" from another computer on your LAN.

2. Upload a 10-20MB file via Gmail or Dropbox from your computer.

3. Watch the ping times on the other computer skyrocket from around 100ms to upwards of 5-10 seconds.

4. Try a Google search from any other computer on your LAN while this is happening.

As an example, Apple's software update actually uses a variant of LEDBAT (the delay sensitive congestion avoidance algorithm from BitTorrent's protocol) when downloading software updates to avoid inducing bufferbloat in the downlink.

terminado 10 years ago |

When you think about it, it's pretty weird that we've had resumable, buffered streaming video modules for server-side stacks for years now, but nothing standardized for sending data in the other direction.

I guess getting things into the server stack can roll out slowly, with very limited and spotty adoption.

Browser support on the other hand...

Well, need I continue?

Suffice to say, that even without the open warfare between rival browser projects, and nightmare-mode security vulnerabilities in common browser plug-ins, it will only make a difference if hundreds of millions of users actually latch onto a new protocol, before it becomes something of a norm.

advisedwang 10 years ago |

If anyone is interested in prior art, take a look at Google Cloud Storage's resumable uploads: https://cloud.google.com/storage/docs/json_api/v1/how-tos/up...

It's very domain specific and relies on GCP's JSON API so probably isn't suitable to apply broadly.

kvz 10 years ago | |

Hi, one of the authors here. Indeed, resumable uploads are not a thing we invented. Nevertheless, we think it's handy to have a document describing the approach and client and server implementations. Amazon, Dropbox and Google offer some in the way of describing how they do it, but nothing to address the problems that wider audience could encounter, and no platform for collaboration to make it better. tus was built under MIT and on GitHub and will continue to evolve through contributions from the community (although we consider that what we cemented into 1.0.0 ready for adoption)

thescriptkiddie 10 years ago |

Resumable file transfers in JS is really cool.

But why a totally new protocol? Why not use bittorrent, rsync, or even ftp? A lot of effort has gone into making those reliable in the face of network congestion. Trying to shove a lot of bits down a single TCP connection with no traffic shaping is just asking for bufferbloat to strike.

auscompgeek 10 years ago | |

Because these protocols are unusable from the perspective of JS on a webpage in a browser. Yes, browsers do support FTP, but I don't believe there are any browsers that support FTP uploads (and even if there are any, I doubt it'd be possible to upload from JS).

HTTP also has the advantage of being allowed through virtually every firewall.

It seems that webapps are paving the way of the future, for better or worse.

kvz 10 years ago | |

Hi, one of the authors here. Thanks for raising a valid concern. Vimeo in fact already offers FTP uploads for premium users, but helped us write this as the advantages of something HTTP based are significant. We can:

- be sure that browsers and servers already speak it (so small additional libraries required on both ends)

- gracefully degrade to regular form POSTs should tus support not be possible (IE8, etc)

- easily write hooks for progress bars, and e.g. encoding of the uploaded material

- use existing holes in firewalls. FTP requires many ports (for PORT, DATA, PASV) that are blocked on airports, public libraries, large corporations

- add existing HTTP components to make tus better (loadbalancing, intrusion detection, auth, proxies, etc). Some of these apply to FTP as well, but the HTTP options are more (advanced).

Bittorrent is not really client-server oriented, which is what this protocol is trying to solve. Rsync has many the same disadvantages as FTP listed above.

As for a single TCP connection with no traffic shaping, tus support splitting up a file into multiple parts and uploading them in parallel (as regular tus uploads, so profiting from checksums, retries, and resumability) and stitching them together on the server-side by issuing a 'concat'. This means you can have as many connections as you like, and the individual chunks can complete in any order as well.

Traffic shaping is not part of the protocol. By default tus should saturate available connections.

That said we're writing down some recommendations in a separate 'developer guide' document that offers best practices for implementing and deploying tus, traffic shaping could be part of it. Feel free to weigh in here https://github.com/tus/tus-resumable-upload-protocol/pull/68

LeoPanthera 10 years ago | |

rsync was the first thing that came to my mind as well. You can even use it for entire directories of stuff.

daurnimator 10 years ago |

The 100-Continue idea mentioned in the 3 year old top comment seems like a much better idea. The issues on the tus issue tracker seem to discard this prematurely.

kvz 10 years ago | |

Hi, one of the authors here. 100 Continue wasn't discarded that quickly and is part of the protocol. More info here: https://github.com/tus/tus-resumable-upload-protocol/issues/...

daurnimator 10 years ago | | |

I saw that issue. As I read it, it discards using 100 continue to solve the original problem.

nodesocket 10 years ago |

Do you have to use the tus server written in go (https://github.com/tus/tusd)? Possible to use a node.js backend?

mintplant 10 years ago | |

You could implement the protocol in node.js. Here's the standards document: https://github.com/tus/tus-resumable-upload-protocol/blob/ma...

edit: Looks like someone's already done that: https://github.com/vayam/brewtus (found on http://tus.io/implementations.html, which also has links to Ruby, PHP and CoffeeScript server implementations, and Qt C++, PHP, Go, and Python clients)

kvz 10 years ago | |

Hi, one of the authors here. We're currently working on an official implementation in Node.js ES6 now here: https://github.com/tus/tus-node-server - so yes :)

voltagex_ 10 years ago |

As much as I'd like to see this using an existing standard, I'm very glad - my upload bandwidth is <1 megabit, and I get several disconnections a day (ADSL desync)

kvz 10 years ago | |

Hi, one of the authors here. I really hope the sites that you're using a lot will all implement tus :) I'm confident that could avoid some frustrations

amelius 10 years ago |

Does this allow traffic shaping on the client? Or will this "stuff the pipe" once an upload starts?

kvz 10 years ago | |

Hi, one of the authors here. It would, but traffic shaping is not part of the protocol itself and by default tus will saturate available connections.

amelius 10 years ago | | |

I think traffic shaping is very desirable when using a websocket for a separate command-channel. The last thing you want is an upload blocking all communications over that command-channel.

Furthermore, browsers can open only a maximum number of connections at the same time (I believe the maximum on some browsers is even just 2). What if the application already uses one connection for a websocket? Will there be only 1 connection left for the upload? And what if the browser needs to download other resources in the background, such as images, fonts, etc.?

Just some concerns, good luck with the project :)

AReallyGoodName 10 years ago |

How long till you guys get a Java server library out?

Currently using AWS API for resumable uploads here.

kvz 10 years ago | |

Hi, one of the authors here. You are the first one to request it, I've documented that here https://github.com/tus/tus-resumable-upload-protocol/issues/....

None of the current core members have experience running Java servers in the real world so it's hard to see what would be involved. We're definitely open to discussing it in the issue if you like.

cwt137 10 years ago |

Why are they not prefixing their headers with "X-"?

seivan 10 years ago | |

I recall it became unnecessary at some point(?)

Might be relevant. https://tools.ietf.org/html/rfc6648

kvz 10 years ago | | |

Hi, one of the authors here. Correct, we added that to the protocol FAQ too: https://github.com/tus/tus-resumable-upload-protocol/blob/ma...

The only exception is X-HTTP-Method-Override, as people have been standardizing on this, and the header is specifically meant to let people deploy tus in environments that don't support all of HTTP and are apparently hard to change.

cwt137 10 years ago | | |

Thanks

seivan 10 years ago |

Out of curiosity why did you decide to use NSURLConnection instead of NSURLSession? I recall NSURLSession also had some code related to resume functionality for both uploads and downloads but my memory is a bit foggy right now.

salgernon 10 years ago | |

Particularly since NSURLConnection is deprecated in the current SDK. NSURLSession has been available since IOS 7.

While NSURLSession does support resumable downloads, it isn't able to resume uploads.

The needNewBodyStream: delegate is the closest thing, and that is called when first uploading from an NSInoutStream or when a transmission error occurs for an idempotent request, or an auth challenge occurs when some of the stream has been sent. (Streams cannot be rewound.)

Background NSURLSessions can take a file reference to upload, and ideally will perform the upload "at a good time", hopefully while charging and connected to a stable network. (But it won't resume an upload, just retry if possible.)

Disclaimer: I'm "involved", but not speaking for any parties involved. Mostly because I don't get invited to those sorts of parties.