You can run Git on object storage if you re-make packfiles(tigrisdata.com) |
You can run Git on object storage if you re-make packfiles(tigrisdata.com) |
For example: https://andrewaylett.github.io/rgitweb/ is a repository viewer that runs against a static git repo. It doesn't care what you're storing the pack in, just that the data is there to make the right range requests.
Great to have issues, PRs, etc in git, but requires each participant to set up a fork to participate.
No CI though, but an extension to capture job logs is on the roadmap.
I found the idea of a git-proxy interesting where you start with proxying an existing forge and then later adopt the object storage backend
never finished, because stopped having personal use case, but had exactly same idea(and a bit more features)
I think that makes more sense than some spam campaign for this.
Works pretty awesomely with git workloads, even Linux sized trees.
What you are saying sounds like an assertion that the problem described here[1] - by Vicent Marti - that packfiles aren't contiguous blocks of data and so every fetch fans out to more fetches from the client - is not actually a problem? Or is there an innovation in zerofs that somehow solves this for git? From what I can see it offers NFS and NBD interfaces and GitHub tried and rejected both of those very early on.
The problem he described seems consistent with known patterns from datalake storage architectures, its a big part of why columnar storage is the only game in town for most use cases.
This is why TFA has a lot of interest here - he has changed the storage layout of git packfiles to a columnar model to address this challenge.
[1] https://cursor.com/blog/git-at-any-scale#git-without-packfil...
If Git repos were routinely terabytes in size, this would be a much harder problem. But the vast majority of repos are <1GB, and at that scale caching dominates. Even with imperfect locality, you can cache enough of the working set that those reads don’t turn into s3 round trips.
In a low-latency environment, I haven’t found filesystem round trips themselves to be an issue for Git workloads either.
https://github.com/awslabs/git-remote-s3
This works seamlessly so you can run `git remote add origin s3://my-git-bucket/my-repo`
It only really makes sense to tarball repos into cold storage on S3 or Glacier, but short of GitLab cloud I suspect there aren’t enough repos cold enough to be worth the dev costs.
S3 that doesn’t have wild costs for git objects sounds like it’s just NFS
Works great to this day. And I was far from the first.
Yes, it was not "object storage", but Git can be served from a dumb http server as static files. Now someone figured out that object storage can also serve static files via http. Wow!
https://github.com/Nickforall/walgit-elixir https://github.com/tobi/walgit (https://news.ycombinator.com/item?id=49420598) https://github.com/tuist/micelio/
BLAKE3 has a whole mechanism for this, and as an added bonus it'd be substantially faster.
And then "dotgit" was never seen again :)
Author thinks that delta encoding is there for no reason. Good luck, kid.