Transfering Files with gRPC

52 points by CommonGuy 112 days ago | 21 comments

sluongng 112 days ago |

https://github.com/googleapis/googleapis/blob/master/google/... is a more complete version of this. It supports resumable uploads, and the download can start from an offset within a file, allowing you to download only part of the file instead of the whole.

Another version of this is to use grpc to communicate the "metadata" of a download file, and then "side" load the file using a side channel with http (or some other light-weight copy methods). Gitlab uses this to transfer Git packfiles and serve git fetch requests iirc https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/sidec...

pipo234 111 days ago | |

I understand some of the appeal of grpc, but resumable uploads and download offsets have long be part of plain http. (E.g. RFC 7233)

Relying on http has the advantage that you can leverage commodity infrastructure like caching proxies and CDN.

Why push protobuf over http when all you need is present in http already?

avianlyric 111 days ago | | |

Because you may already have robust and sensible gRPC infrastructure setup and working, and setting up the correct HTTP infrastructure to take advantage of all the benefits that plain old HTTP provides may not be worth it.

If moving big files around is a major part of the system you’re building, then it’s worth the effort. But if you’re only occasionally moving big files around, then reusing your existing gRPC infrastructure is likely preferable. Keeps your systems nice and uniform, which make it easier to understand later once you’ve forgotten what you originally implemented.

sluongng 111 days ago | | |

The evolving schema is much more attractive than a bunch of plain text HTTP headers when you want to communicate additional metadata with the file download/upload.

For example, there are common metadata such as the digest (hash) of the blob, the compression algorithm, the base compression dictionary, whether Reed-Solomon is applicable or not, etc...

And like others have pointed out, having existing grpc infrastructure in place definitely helps using it a lot easier.

But yeah, it's a tradeoff.

ithkuil 111 days ago | |

I like implementing this standard gRPC interface (of I already have a gRPC based project) because it allows me to reuse a troubleshooting utility I wrote that uses it:

https://github.com/mkmik/byter

augusteo 111 days ago |

Building on sluongng's point about schema evolution: we ended up in a weird middle ground on a project where we used gRPC for metadata and presigned S3 URLs for the actual bytes.

The metadata schema changed constantly (new compression formats, different checksum algorithms, retry policies). Having protobufs for that was genuinely useful. But trying to pipe multi-gigabyte files through gRPC streams was painful. Memory pressure, connection timeouts on slow clients, debugging visibility was terrible.

S3 presigned URLs are the boring answer, but they work. Your object storage handles the hard parts (resumability, CDN integration, the actual bytes), and your gRPC service handles the interesting parts (authentication, metadata, business logic).

jeffbee 111 days ago | |

Sending bulk data by reference is a common pattern. Even inside Google when I was there bulk data was sometimes placed on ephemeral storage and sent by reference, and 100MB was considered a "dangerously large" protobuf that would log a warning during decode.

kruador 111 days ago |

I would add a further advantage of plain HTTP (REST) compared to gRPC. Splitting the response into blocks and having the client request the next block, as in the gRPC solution, causes round-trip delays. The server can't send the second block of data until the client requests it, so the server is essentially idle until the client has received all packets of the first block, parsed them and generated the next request.

In contrast, while HTTP/2 does impose framing of streams, that framing is done entirely server-side. If all one end has to send to the other is a single stream, it'll be DATA frame after DATA frame for the same stream. The client is not required to acknowledge anything. (At least, nothing above the TCP layer!)

It probably wasn't noticeable in this experiment as, if I'm reading it correctly, the server and client were on the same box, but if you were separated by any significant distance, plain HTTP should be noticeably faster.

CamouflagedKiwi 111 days ago |

I've done this before, using Google's semi-standard ByteStream messages. It works, but is a bit of work, and I really don't love how you're building on top of a protocol that completely solves streaming contents of arbitrary size, which gRPC drops, and you have to reinvent again in the application layer.

I know it's not easy to solve given how protobuf-centric it is, but this is the worst piece of gRPC for me. The 4MB limit is a terrible size, it's big enough to rarely hit in test cases but small enough it can hit you in production. If you control it all you can just lift that number to something arbitrarily big to avoid most things just failing (although you probably don't want to use that as an actual solution for streaming files of any size), but in practice a lot of "serious" setups end up contorting themselves remarkably to try to avoid that limit.

aktau 112 days ago |

Perhaps worth mentioning: https://github.com/stapelberg/rsync-over-grpc.

matttproud 111 days ago |

Method signatures in gRPC present a pandora's box of questions: https://matttproud.com/blog/posts/grpc-method-discipline.htm....

The questions aren't unique to gRPC, however; gRPC forces you to confront them early and explicitly IMO, which is not a bad thing.

jasonjei 111 days ago |

I have PTSD from Google Protobufs. Sometimes the cost of a less-efficient protocol or traditional REST is worth it over an overengineered solution. Protobufs can be fine, but it's largely overkill. Debugging with protobuf was the price we paid for an "efficient" protocol

tuetuopay 111 days ago |

And then a C programmer comes in and slams sendfile. That’s the main advantage of HTTP/1.1. Of course TLS throws a wrench in it, but once kTLS is actually good (ahem), it’ll work.

In all seriousness, don’t do large file transfers over gRPC, except in a pinch for small files. As soon as e.g API gateways are introduced in the mix, stuff can go south very quickly: increased allocation, GC pressure, network usage, etc. Just use presigned S3 URLs.

aurumque 111 days ago | |

S3 also gives you multipart parallel uploads. Each part gets stored and then when you're done the concatenation is performed close to the storage layer.

tuetuopay 110 days ago | | |

Indeed. So many wheels in HTTP/1.1 that needs to be reinvented with gRPC.

profsummergig 111 days ago |

Apparently the correct spelling is "transferring".