The GitHub registry public beta is live(help.github.com) |
The GitHub registry public beta is live(help.github.com) |
https://hn.algolia.com/?query=Software%20Heritage&sort=byPop...
And GNU Guix at least will transparently fallback to them:
> Since Software Heritage archives source code for the long term, Guix can fall back to the Software Heritage archive whenever it fails to download source code from its original location. The way this fallback has been designed, package definitions don’t need to be modified: they still refer to the original source code URL, but the downloading machinery transparently comes to Software Heritage when needed.
https://www.softwareheritage.org/2019/04/18/software-heritag...
I wonder if it'd be too crazy to add git-awareness to the Nix utils. Like tell nix to install version X of some package, so in the nixpkgs repo it would check the history of the corresponding file to find the commit where it last was that version, checkout that commit, build the package and its dependencies, and then return to the branch it was at.
In theory, that database could also just be included in the channel so it could be queried locally, but I don't know how large it would end up being.
I would prefer doing it based on the 6-month release channels, so you get multiple version for every 6 months. You end up with some gaps between versions, but also have more guarantees everything actually works together. Basically "nix search" with multiple channels.
I actually had to do something similar with GHC versions for a project of mine. It turns out you can run Nixpkgs all the way back to the first release in 13.10 (LC_ALL=C is needed). Obviously not that long ago right now, but it should continue to work as time goes on & give us 10+ years.
https://gist.github.com/matthewbauer/ecdb665f84b7f6100f3f6e3...
What you suggest in the second paragraph sounds great and definitely doable!
You can, on Windows. The secret is the system keeping backwards compatibility in mind for its core APIs, something that most desktop APIs on Unix (outside of X11) do not do.
I am... really confused by this.
Isn't this just Github? Github is a hosting service that allows you to host your packages and code in one place. It has testing and publishing pipeline support, you can add artifacts/releases, make your packages private or public, host different types of software at the same time, and it's compatible with most existing dependency systems, including NodeJS.
I can see this has more download statistics, which is nice. And it has a policy that artifacts can't be deleted, which is very nice.
Is that it though? I know I have to be missing something; what can I do now that I couldn't already do with Github as is?
This seems to limit the utility for ruby. I'm not sure if other supported platforms have similar issues?
You could already do a lot of what github registry for ruby does by using an existing feature where you could already point to a git repo (not just GH) in your `Gemfile`. What this adds is just the ability to resolve multiple versions from github using ordinary rubygems resolution. The existing feature forced you to manually specify a tag (hoping there was a predictable tag for a version) or SHA, or use whatever is on master HEAD.
And then removed 16 months later: https://github.blog/2009-10-08-gem-building-is-defunct/
Hopefully this one lasts longer.
I'd love to be able to host wheels for my python projects, or {rpm, deb, flatpack, etc...} for effectively arbitrary code. Is that in the works?
For example, Python has a distinction between distributions (the actual file downloaded, e.g. a tarfile or a manylinux1 wheel) and versions that doesn't exist in most other languages.
And, I suspect (from python's noteworthy absence), degree to which the language's users were already (mis)using GitHub releases (or sites) for this purpose.
pip install git+https://github.com...
git+ssh://git@bitbucket.org/foo/bar.git@fixit/atemp69#egg=hotshitDon't know but the formats do not seem to match the Azure DevOps package feeds formats (some overlap of course but some missing in one, some missing in the other) so it's not from shared code.
0: https://hub.packtpub.com/surprise-npm-layoffs-raise-question...
1: https://github.com/npm/cli/pull/125#issuecomment-474127391
The problems like with rubygems from yesterday and npm a few weeks back would be gone with something like that.
Oh, and no pip registry :(
GitHub is already really great about alerting you with critical issues. Whenever there's a security bug, it pops up in our repo (and with Dependabot, it's become automatic).
I just looked up Dependabot and linked it with a repo that I already have robust testing and CI pipeline for. Preliminarily Dependabot is great!
It automatically updates by dependencies to the latest versions and submits individual PRs. Since I have TravisCI hooked up to this particular repo, I can see all the test results for each PR and can (confidently) merge the changes into master without manually firing up my personal dev machine(s) and manually performing what Dependabot just did.
Anyway, thanks for the tip!
They should have a release and snapshot branches.
Would be interesting to see how it compares to Docker Hub for hosting private images.
(No, I'm not related to that company in any way. I just saw it yesterday and thought it seemed like a neater solution.)
git+ssh://git@bitbucket.org/foo/bar.git@fixit/atemp69#egg=hotshit
We forked some things into a private DevPI instance at present for that reason (well, also for latency)
On top of that, for orgs, you'd be paying monthly for each user you add.
Right now, all the major package manager are indirectly making each other better, they experiment, improve and borrow good ideas from each other. It's open-source and there is a little barrier for developers to contribute.
If n years from now GitHub becomes the defacto standard for package managers and replaces all the existing ones the further innovation will be much slower.
It might transform into "Want to improve package managers? You have to work for Microsoft"
This is a replacement for npmjs.com (the hosting service, which is not the same thing as npm the package manager), or for rubygems.org (again, the hosting service, not the gem tool), or for Docker Hub.
If anything, this may actually improve the collaboration between package manager developers because there will now be a large development team that will be working with the backend of various package registries and will have better insight into what each one is doing wrong and what each one is doing right.
Even immutable repos are going to have fun with DMCAs.
>GitHub Package Registry allows you to develop your code and host your packages in one place. You can use packages from GitHub Package Registry as a dependency in your source code on GitHub.
"Package registry" is a fairly generic term, so to me it would be natural to refer to this product as "the Github package registry" (capitalized or not).
Is there a name for deliberately avoiding "the" in this way?
"Limited" should have been in the title, because it makes it a not so public beta.
We've been looking for a simple way to streamline releases. Right now everything we have at my job is on GitLab and I use GitLab personally (though I have a github account, of course).
I prefer GitLab in every way, but this feature alone might be a good enough reason to switch. It would make releases just so darn easy. The only thing I hope (which is not made clear) is that the stipulation that you can't easily delete a package on the registry (According to the link, its only for GDPR requests and legal reasons) is something that, for instance, an Enterprise account wouldn't have. I already have our purchasing team looking into it, thats how serious this is.
If the API for hitting these packages is any good, its gonna be so hard to resist.
I really hope GitLab has a good response to this.
To wit, since GitLab is custom hosted, I wonder how hard it would be to add this into the CE edition....
With all that said, I wonder what the hidden limits will be. Imagine if instead of NPM maintaing all of its servers, it was just a thin database that had better routing to github releases? Would that fall afoul with GitHub?
I mean, whats the point of maintaining your own distribution server when GitHub can front all the hosting costs and all you have to do is map the name of a package to its Github Package Release URL. I could see NPM, PyPI et. al. just doing that, instead of having their own servers. Maybe its a good idea to run additional cache nodes, but GitHub being the main place where release code lives for you package index would cut the bills significantly no?
Is Artifactory immutable? Or I guess that it handles versioning/publishing better?
[0]: https://help.github.com/en/articles/linking-to-releases
Before this new service how would you use GitHub as a source for installing, for example, Maven packages?
Does Maven do something more complicated like automatically figure out which platform binary to pull?
http://www.lordofthejars.com/2011/09/questa-di-marinella-e-l...
The immutability of the packages is also handy as you pointed out by the hope and a prayer that a tag stays static.
Is there not a global config for rubygems that would specify a list of registries to search for a package instead of having to add them to each project?
So you'd still need to add a separate source for each dependency hosted on github to your own project Gemfile. Including for each indirect dependency, knowing which indirect dependencies exist that need a github repo source.
If you could list this for the entire project... it'd probably be a performance issue as rubygems/bundler check every repo source you list for every dependency (including every indirect dependency; a Rails app has hundreds, still an order of magnitude or two less than a react JS project heh).
Even if you could only list "github's ruby registry" once (per project? for your account? and keep in mind this is hypothetical, you can't), it would still mean any gem expressing a dependency on another gem hosted on github would have to include in it's instructions "oh, if you use this, you need to manually make sure to add github to your sources. Or you'll get an error that says some gem you've never heard of can't be found, and have no idea how to fix it." Unless it's a bid to get _everyone_ to do that, and basically make github ruby registry a standard part of the ecosystem that everyone just always adds to every project.
I don't think there's enough/any value added by the github ruby registry to get the ecosystem to shift like that. It's unclear what it does that the 'standard' rubygems.org gem source doesn't do already (unless rubygems.org can't solve their recent severe compromised account security problems... but as it is, with the indirect dependency problem, I think github registry will be too painful to use even if you'd like to to escape rubygems.org security issues).
https://help.github.com/en/articles/configuring-rubygems-for...
If you want to disrespect customers by treating them like disposable guinea pigs (and not even giving them the courtesy of notifying them they're part of an experiment), don't be surprised if they start to catch on and treat your company as if it's disposable, too.
It's not clear to me what the value added for ruby specifically is now. (Yes, I know rubygems.org has problems; but this has feature problems with indirect dependencies compared to rubygems.org hosting).
I know Azure Pipelines is becoming the sort of defacto automated CD/CI pipeline though (used to be Travis for so long) and I've heard nothing but good things about that. Might have to take a look.
At GitLab, the CE edition currently offers a container registry, that allows users to build, push and share images using the Docker client and/or GitLab CI/CD.
The EE edition offers an NPM and Maven registry, that allows users to publish, download and share dependencies. Both also integrate with GitLab CI/CD. We are currently working on Conan (C/C++) and NuGet (.NET). We are evaluating moving these features to CE as well.
We also offer a proxy to for Docker images (which will be extended to each registry) that improves reliability and performance and (in the future) will help mitigate and remediate open source risk.
If you end up trying GitHub's registry, I'd love to hear more about what you thought.
GitLab can also work as a proxy to upstream registers for performance and in the future for security.
You can use Gradle, which supports source repositories, instead of Maven:
Even if GitHub "wins" there will still be a lot on Docker Hub.
Also, it's already possible to use other registries like Amazon's ECR or Google Cloud's GCR, or even your own private one.
(now the asterisk: in terms of functionality above and taking an average desktop distribution in mind you'd probably find more of it in Linux, but applications can't rely on most of that functionality being available on all distros as a "standard" and even for the stuff they could expect, way more often than not they rely on specific version ranges - e.g. an application cannot rely on "Gtk", it can only rely on "Gtk <version>.xxx", as Gtk itself isn't backwards compatible across major versions).
We do something similar (and the way more manual) here[0].
[0]: https://github.com/dapphub/dapptools/blob/master/overlay.nix...
https://gist.github.com/matthewbauer/7c57f8fb69705bb8da9741b...
It keeps makes past versions of packages available to you, but does not really help with discoverability.
Also, to maintain package building scripts in a database outside of git would require either making 1 script per package that builds every version of that package ever. That sounds inconceivably hard. Another option is to have separate scripts per version, which sounds redundant. Another option is for newer scripts to import code from previous versions' and override parts, which sounds convoluted. If you use git, the script only has to worry about the current version. Not forgetting how to build previous versions would be automatically handled by git.
Re-reading your comment thought, I think you're talking about pre-built packages. If that's the case, then I think I agree. I'm talking about the repo containing the scripts for building such packages.
> If you maintain such a "database", you'd have to explicitly state version info for the dependencies of every package.
I don't know what you mean. If I say "I want git v2.10.0" I don't care about dependencies; whatever nixpkgs tarball I download that has that will have the dependencies too. There will be a whole range of nixpkgs tarballs that contain the requested git version of course, but with such a database I could also say "find me a channel tarball that contains both git v2.10.0 and some-dependency v7.23" if I want that particular combination (assuming the two coexisted in the same nixpkgs at some point).
> For a particular version of a package, the versions of the dependencies it's compatible with are those that are present in the same commits.
This is true of downloading a nixpkgs channel tarball too.
> Also, to maintain package building scripts in a database outside of git would require either making 1 script per package that builds every version of that package ever.
I don't know what you mean by this either.
> Re-reading your comment thought, I think you're talking about pre-built packages.
No I'm not.
What I'm talking about is just any time hydra builds a channel, it can run a script that collects the pname/version combination for every derivation in nixpkgs (or at least, every searchable derivation; we probably don't need to collect this from all the invisible helper derivations). This can then be appended to a database from the previous version, such that we end up with this information for every single tarball.
In fact, this is pretty much exactly what you'd do for git, except you'd build it incrementally rather than running a script that processes the entire git history from scratch (which is pretty much building it incrementally for every commit).
EDIT:
> The default way to consume nixpkgs is through channels which today does not involve git. And a nixpkgs git checkout is over 1GB (my .git dir is currently sitting at 1.4GB on this machine). So that's not great.
One could also add tags to the repo in the form of pname-version for every package. I wonder how well git can handle that many tags...
In any case, the advantage of being able to do this from the git repo is that you wouldn't depend on someone forever hosting every version of a channel. I would think one would discard old channels before they discard git history.
Some short-term conveniences come at a very high price in the long run.
If you could... they wouldn't have built this.
I'll take your word on it though. I don't know much about how Java package management works, and like you said, I assume the Github team wouldn't waste their time building something that wasn't necessary.
I guess if nothing else it would be a pain in the neck to have to know in advance how release files had to be laid out.
How are you going to recreate that directory structure with GitHub releases? You can't even have any custom directories - they're just release-name/file-name.
I mean just try recreating it yourself and see how far you get.
You could try use GitHub pages instead, but GitHub very actively pester you if it even looks like you're distributing binaries there.
They can’t, at least not older packages, I believe they need to contact NPM if they wish to unpublish packages. https://docs.npmjs.com/cli/unpublish
[0] https://chiselapp.com/user/rkeene/repository/hashcache/ [1] http://hashcache.rkeene.org/
SHAs can't be changed, but they can be deleted. And on GitHub, entire projects, usernames, orgs can be deleted. Or renamed. In case of a user rename, GitHub does maintain redirects for awhile. Until that username is taken by somebody else.
So yea, does not seem to be a problem that actually exists.
Fortunately a copyright takedown request is not a typical scenario, but it does happen, even with "immutable" repositories like maven-central.
In other words a "copyright takedown request" isn't valid, unless the author was in violation of the copyright of somebody else while publishing those packages and this was decided in a court of law.
It might happen, but I have never heard of Maven Central packages being removed.
But I do see GitHub repos being renamed or removed all the time and I have seen NPM packages removed, for no reason other than the author wanted so, screwing the entire JavaScript ecosystem.
The DMCA process is law. Maven Central (like anyone else who hosts things) have to respond to valid takedown requests (which means taking down content long before any court case; even if a counter-notice is filed the content still has to be taken down temporarily) or else they'd become liable for infringement themselves. It's less common than on github or NPM, sure (which I suspect has more to do with the complexity of maven central's registration process than anything else), but it happens and any host on the scale of maven central needs a process in place for doing it.
Surely you must be joking.
Yes it is a big concern and the solution is to use repositories that aren't so volatile.
Granted this is quite the esoteric edge case... at least for now. ;-)