Comparison of Programming Language Package Managers(docs.google.com) |
Comparison of Programming Language Package Managers(docs.google.com) |
pip, for instance, eagerly resolves the highest version of a dependency first found in the tree and doesn't backtrack AFAIK [1]. Cargo sounds as though it behaves similarly, but will actually backtrack [2]. I've also seen package managers like 0install that will use SAT solvers to solve dependency constraints [3].
I've found lack of backtracking on conflicts to be frustrating, but I'm curious how the other two (and other conflict resolving strategies) compare in practice.
[1]: https://github.com/pypa/pip/issues/988
https://github.com/rust-lang/cargo/issues/2064
and graydon wrote a POC of using the Z3 solver to find a solution where semver constraints were satisfied, optimizing for the latest versions within those constraints:
https://github.com/graydon/z3-rs/blob/master/tests/semver_te...
Glide has a dumb solver in that it goes with the first best version it finds. It will tell you if there is a later conflict that you can manually resolve.
There is a package, discussed here called gps, that is being prepped to merge into Glide. This solver has a back-off strategy. It's loosely based on the solver in dart. The idea is to find the latest newest version that works for the complete tree.
We've been incrementally working to get a good solver and are almost there.
1. RubyGems is the package manager. Bundler is a meta-layer on top of that which does full dependency resolution to find the right version before installing. Bundler builds on the capabilities provided by RubyGems (and will be integrated into RubyGems in the future).
2. Bundler does support vendoring. It is widely discouraged, but I take advantage of it in my app packager for deploys at work (Cartage: https://github.com/KineticCafe/cartage, specifically cartage-bundler https://github.com/KineticCafe/cartage-bundler).
3. RubyGems and Bundler can use alternative sources (my work applications use both public and private sources simultaneously).
One thing I don’t see that would love to see tracked and solved is authenticity verification. RubyGems has support for signed gems but it’s not widely used and hasn’t really been validated as Correct.
They should probably also look at CocoaPods and Carthage (iOS and macOS build dependencies) and whatever Swift provides.
It is also widely encouraged. Some engineers:
1. Don't trust dependencies to always be there.
2. Write apps that need to work in disconnected environments -- ie with no internet connectivity.
I work on the Cloud Foundry buildpacks team for Pivotal. Being able to stage and launch apps in a disconnected environment is A Big Deal for a lot of companies.
Heroku mostly does this right with its 'slug' system, and I made Cartage specifically to be able to make deployable packages that use dependencies. You have a build machine that’s connected to the network and can reach RubyGems (and, in the future with some plug-in work, other packaging systems—I expect to add Node, Elixir, and Lua soon) and that will vendor the resulting build and give you a tarball that is just installable with all of the dependencies of your package.
Using a new language is no longer a matter of cracking open a book, installing a compiler, and firing up vim. You've got to change your whole lifestyle now.
https://www.dartlang.org/tools/pub
It hews pretty closely to Bundler's model (versions, version ranges, lockfiles, shared dependencies), though. So I don't know if it gives you much insight beyond "Yes, another language team things Bundler is on the right track."
Personally, I have a hypothesis that states that all programming languages either have a package manager that works like Bundler or will end up getting one.
https://metacpan.org/diff/file?target=ETHER%2FURI-1.71%2F&so...
This information is also available through an API for integration into command line tools.
In fact, the original creator of Glide (Go package manager) wrote about Perl and CPAN when talking about Go at http://technosophos.com/2015/09/02/dont-let-go-be-condemned-....
Other things from the Perl ecosystem that should be copied are:
- CPAN Testers which automatically tests every package on multiple systems from Windows to Solaris. This helps identify portability, backcompat, and regression issues.
- CPAN mirrors which ensure that there isn't a single point of failure. This might not be as important now with fast networks and high uptimes, but it also ensures that everyone can replicate all of the code at anytime. I believe R's CRAN does this.
All modern packagers provide now such src urls, esp. go went ahead with this idea.
Fwiw pip isn't even able to enforce versions correctly (packages are installed as the file is read, and can conflict with previously expressed constraints). Or report installed versions correctly (it's possible that packages are half-installed or installed but not reported as such by pip commands).
$ pip freeze -r requirements.txt > constraints.txt
$ pip install -r requirements.txt -c constraints.txtWe were sad.
* All current package managers are either language or OS specific. What if you have an application with code written in multiple languages?
* NPM didn't have any kind of integrity checks for its packages, and I assume most package managers don't either. If you download a corrupt package, for example, you won't have any idea and it will still install.
* Some package managers do better than others with regards to managing packages. I found NPM encourages dependency hell and very little management tools for dependent or installed packages.
* A lot of package managers seem to intermix packaging, distribution, and a registry. The registries tend to have limited names to pick from (like real estate) and can result in legal problems. Also if registration to the service catalog is required you cannot self-host or self-manage the distribution of your application.
I am trying to work on a solution to these problems at https://github.com/prettydiff/biddle
I'm not arguing the point just trying to understand the decision.
At least, that's my understanding of what they meant.
Probably because of this[0]:
> Note that Cargo does not yet support vendoring in a first-class fashion, but this is a hotly desired feature and coming soon!
[0] http://doc.crates.io/faq.html#how-can-cargo-work-offline
CPAN, RubyGems, etc. all exist because system packaging systems are (a) specific to a particular distribution, (b) generally many versions behind the development versions, (c) really hard to get into unless you are already a system packager for a particular distribution, and (d) effectively don’t exist for two of the largest platforms where you will find users.
I’d suggest making some sort of meta-system packaging system that works for all the languages, but xkcd 927 (and I wouldn’t use it myself, anyway, because it would probably not install Ruby packages correctly and would solve things worse than NPM already does).
- It does mean that the common libraries are installed once - conflicts between common libraries are more transparent - System tools make sure that package stays up-to date
* building on not-linux -- you know, where almost all personal computers exist. Gee, wouldn't it be nice if most computers could actually compile your code?
* Having only one build config, only having to maintain one publication (ugh need to maintain deb, rpm, nix, chocolatey, brew, whatever the hell arch and suse use...)
* reproducible builds -- distros are well-known to randomly modify libs, making "foo x.y.z" in yum and apt basically random. It's not even clear to me I'll get the same thing from apt on different days!
* Iterating and collaborating on fledgling projects which rightly wouldn't be accepted into a distro -- or just iterating on a timescale smaller than "months"
* local deps -- using system-global deps (as preferred by most linux package managers) is known to be a huge nightmare for any language package manager that's tried. Hence pyvenv. Hence why so many have thrown up their hands in disgust and just started dockering all the things.
(inb4 someone comes to explain that Nix or whatever probably magically solves all these problems)
Like all things in languages, the world needs to be reimplemented. Which is too bad, but I also never see a great alternative. Eventually people want everything they work on to be in the same language, which of course has the advantage that anyone in that language can hack on it, but the disadvantage that we always start anew every time a language gets created.
Containers don't "solve" everything, but some of the primitives used to build them can be used to severely restrict process and user privileges -- to the level that it's much more secure than a "standard" unix-y shared environment.
I wish devs would just tell me the libraries I need, and I'll get the packages myself.
Unless your on the bleeding edge of development, you don't need the newest versions of a lib. And you probably shouldn't be developing on the bleeding edge anyway. There's a reason apt-get is so stable, and is used everywhere.
The caveat to all that is: I hate making packages for apt-get. It's a terrible process, and I completely understand why so many devs want to avoid it.
This is a decent overview on technical details: http://vincent.bernat.im/en/blog/2016-pragmatic-debian-packa...
[#] I assume you meant the friction with the tools, not the fact that package builder needs to chase down all the dependencies and generally prepare many source tarballs to be included in a repository.
I've seen this argument come up a few times now, and I have a hard time reconciling this rosy view of system package managers at the center of a development process with my own experiences. That process was never as easy as "cracking open a book, installing a compiler, and firing up vim" and usually entailed a fair amount of trial and error, environment variable fiddling, and sadness from having to chase down a derelict PPA (or the equivalent) hosting the actual dependency version I need. Then more sadness as the number of projects I work on increases and my dependency versions bifurcate.
Some peers of mine recently experimented with building a Nix buildpack and concluded that it won't work. Nix has hardcoded paths for its core directories, and won't work in a security-constrained container.
Apt is no more standard than pip, gem, or any of the others.
Try maintaining packages for Debian, Red Hat, Arch, Gentoo, OSX, Windows, and a dozen others, for a while, and see how quickly you change your mind.
> You've got to change your whole lifestyle now.
Oh please.
guix and nix both work cross-language and cross-distro. Still OS specific though since only linux AFAIK. Also, containers partially solve this problem.
> * NPM didn't have any kind of integrity checks for its packages, and I assume most package managers don't either. If you download a corrupt package, for example, you won't have any idea and it will still install.
Any package manager that doesn't do integrity checks is a bad package manager. The only one I know of currently that doesn't is npm, but I haven't looked deeply into every available package manager.
> * A lot of package managers seem to intermix packaging, distribution, and a registry. The registries tend to have limited names to pick from (like real estate) and can result in legal problems. Also if registration to the service catalog is required you cannot self-host or self-manage the distribution of your application.
What package managers don't let you self host? I'm truthfully not aware of any. Even NPM does according to a quick google.
> I am trying to work on a solution to these problems at https://github.com/prettydiff/biddle
From your readme: "biddle is inspired by the incredible awesomeness of NPM". Since NPM is literally the worst package manager I have ever used, that line makes me want to stop reading and never touch biddle. I'd word it differently.
Edit: Reading biddle further. Dependency management and central hosting are some of the primary reasons to have a package manager. At least for me, that kills any interest at all. I imagine there's a niche market though?
(Aside from the lack of package integrity check, which I'll grant, sucks).
It's already very hard to find a programmer that writes a lot in more than one language and cares about packaging/deploying software.
I really, really don't get why this isn't copied to the rest of the open source language environments. (Masochism? :-) )
They removed a lot of complexity by doing that and I bet more than a few people didn't notice.
var foo = require('foo');
a. I maintain ~13 Ruby gems. I can generate new versions and verify their install on any system that I can run Ruby on. Under “use a system package manager”, I now have to maintain at least four unique package management formats (deb, RPM, emerge, Pacman) just for Linux distributions and at least one for the BSDs. Thanks, you’ve just increased my workload at least five-fold, although I no longer have to worry about Windows or macOS users (wait, I’m a macOS user).
b. Sure I can use a non-distribution mirror (like a PPA), but now you have to have at least six of those mirrors (see point 1) and, again, too bad about Windows and macOS users since they obviously aren’t smart enough to use an OS that has an in-built packaging manager.
c. In truth, you haven’t increased my package load five-fold, you’ve increased it much more than that. I’ve followed instructions for making debs and RPMs…and the only thing that works for me is using FPM because the system package managers are so bloody complex and built for a single particular use-case.
d. You still don’t have a solution for Windows or macOS users because neither of those OSes have system package managers, and no neither ports nor homebrew counts. RubyGems, pip, cabal, npm, etc. may increase the cognitive load but they work for people who don’t use Ubuntu Zany Zebra.
You’ve also introduced a new problem:
e. OS package managers (except probably nix) do not solve the multiple-installed-version problem. RubyGems allows (even encourages) multiple versions of a gem to be installed and applications (through their Bundler Gemfile) can lock to a particular version. I have packages with 1.x, 2.x, and 3.x releases where all three versions are still in use despite some backwards incompatibilities (mime-types 2.x dropped support for Ruby 1.8, deprecated some methods, and changed the default data format and loading mechanism; 3.x dropped support for Ruby 1.9 and removed the deprecated methods, and changed the default data format).
This is not possible in an OS package manager without renaming the installed library in some way. This is mostly fine when you‘re installing compiled software…but it’s not even perfect there. I’ve been working with language-specific package managers for a long time, and I’ve been dealing with how distribution package managers break some of the packages that I work with for just as long. I generally prefer the mess we have now to waiting for the perfect system package manager, especially since there are systems which will never get them.
[fixed formatting]
[1]: https://www.well-typed.com/blog/2014/09/how-we-might-abolish...
[2]: https://www.well-typed.com/blog/2015/01/how-we-might-abolish...
I think Rust is the greatest programming language of our generation.
But regardless (and I meant irony, not pony in the original post), not everyone will agree with me, so everyone will want their own language. What we need is a general purpose packaging tool (I like cargo, but apt-get could serve equally well) which could support any language.
(And then there's catkin, the build system for the Robot Operating System. That's the build job from hell. ROS is a message-passing environment to which a huge number of existing packages have been adapted. Multiple languages, different underlying libraries, and no central control of versions across packages. It sort of works. Although there are bad days when it breaks the Ubuntu updater.)
I write apt-packages all the time. I get the process. And yes, it's still terrible. And when you consider that devs need to know how to build for apt-get, yum, etc, it's understandable, IMO, why they don't want to bother.
This is the place where I'm sure we're not talking about the same aspect of building packages.
I assumed that you were talking about Debian specifically, since the only hint to the topic was "apt-get" somewhere earlier. I thought that you were talking about the old, overcomplicated way of providing manually all the necessary targets (binary, build, install, clean) and specifying dh_{any} commands in debian/rules, possibly using underdocumented CDBS. This is what would be straightened up by the article I mentioned.
Instead, you seem to mean something about packaging on a higher level than that. Do you care to elaborate, what's terrible in building binary (or DEB) packages? I'm doing it for years and I'm yet to encounter a more sensible way of deploying software, including web applications (which are one huge inconsistent and fragile mess).
> [...] when you consider that devs need to know how to build for apt-get, yum, etc, it's understandable, IMO, why they don't want to bother.
What magic do they need to know? There are very little things for DEBs and RPMs. One thing (a common one) is to have a simple way to compile the project offline. Another (again, a common one, and not limited to packaging) is to know how your runtime and compiler (if any) allows to use libraries. Then you need to know there can be post-install scripts, which are meant to e.g. add user account for a daemon, and possibly how to write an init script. And there are two build rules skeletons to know, but those are quite short. This is the only thing which differs in a significant way. Or have I missed something in your opinion?
We certainly have some work to do on Cargo's docs. Please don't hesitate to open a thread on users.rust-lang.org if you run into some kind of issue. In addition, I actually live in Brooklyn; if Pivotal ever needs a hand with Rust stuff, just let me know, I'd be happy to swing by your office and talk about whatever.
Email me on ... uh ... wednesday, maybe? We could get you to visit to check the buildpack work we did last week.
However, as I noted above, this model breaks for disconnected environments, in which neither the staging container nor the runtime container have internet connectivity.
Heroku's ruby buildpack code runs bundler, Cloud Foundry's buildpack is a soft fork of Heroku's, so either you vendor your dependencies before sending it to Cloud Foundry for staging, or you get a failed staging step when the code in the staging container can't dial out to a remote repo.
To me it "looks" like a buildpack, insofar as you are taking something, injecting its dependencies and producing an artifact that's ready to run by itself. If you ever whack bin/detect, bin/compile and bin/release onto it, it'll probably work well enough as a buildpack in connected environments.
My major complaint about npm is the choice to allow version range operators on dependency declarations. We know the node.js ecosystem places a high value on composability, so using lots of tiny modules which themselves depend on lots of tiny modules is the norm. This is a problem though because range operators get used liberally everywhere, so getting reproducible builds is like winning the lottery.
There are other things I don't like about using npm: node_modules/ is big and has a lot of duplication (even with npmv3), it's pretty slow, historically it has been unstable, its still crap on Windows, etc. - but for someone who has 'ensures reproducible builds' as part of their job description, the way its modules get versioned is its worst feature.
The range operators are important, else you'd never be able to resolve 2 packages that want a similar versioned sup-dependency e.g. jquery 1.12 because without range operators those 2 packages would have declared minor version differences (1.12.1 and 1.12.3) depending on when they were published. This would mean you'd always end up with duplicated dependencies.
I'd argue 'node_modules is big' is not a fault of npm. If the package or app you're trying to install generates a large node_modules dir, that is something you should take up with the package maintainer. See buble vs babel - buble has a way smaller dep tree.
npm is only slow in the ways that all other package managers are, when installing large dependency trees or native dependencies (like libSass) and it is way faster than say pip and rubygems in this regard. When I 'pip install -r requirements.txt' at work, I literally go and make a coffee.
Also never experienced any instability, though I may have been lucky. Certainly it has been very stable for the last year or so when I've been working with a lot. Could you elaborate on why it is crap on Windows? I did think all major issues (e.g. deep nesting problem) were now fixed ...
It shrinkwraps everything in your current node_modules directory.
This includes platform specific dependencies that may not work on other platforms but now will cause npm install to fail instead of just printing a message about it.
So our current workflow has to be:
1. Update package.json 2. rm -rf node_modules/ 3. npm install --production # This doesn't include any of those pesky platform specific packages 4. npm shrinkwrap 5. npm install # Get the dev dependencies
As far as the other comments about npm, I just generally have more problems with it than rubygems/bundler and the general OS package managers.
Deep nesting is not 'solved' it just doesn't happen 100% of the time anymore. If you have conflicts, you still have deep trees. I suppose range operators help with this a little, but looking at what gets installed it doesn't seem to help that much; I still have duplicated dependencies.
I was mentally comparing npm to tools like maven, ivy and nuget, all of which are faster but also not interpreted. Not a fair comparison I guess.
https://nixos.org/wiki/How_to_install_nix_in_home_(on_anothe...
I've not personally tried it. From reading that document, it seems inconvenient but possible.
My understanding is that the hard-coded paths are needed to ensure compatibility of the binary caches, since store paths are often embedded in other artifacts (e.g. RPATH in binaries, shebang line in scripts). You could always run your own cache using another prefix if that's a problem.
http://lethalman.blogspot.ca/2016/04/cheap-docker-images-wit...
Other language managers don't hard-code and will allow stuff to run from a local directory.
> As far as the other comments about npm, I just generally have more problems with it than rubygems/bundler and the general OS package managers.
I generally don't :)
Not sure you're aware of the suggested flow (see here [1]), but it isn't ridiculous. Use 'npm outdated' to see which packages are out-of-date and 'npm update --save' to update a dep (and update the shrinkwrap file).
Keeping track of stale sub-dependencies is a problem in and of itself, but again that exists with any package manager. (Because you will always need to pin dependencies before you go to prod right). So that 'lockfile' will get out of date pretty fast. Node at least has solutions for this that other communities don't [2] (I haven't tried this service).
[1] https://docs.npmjs.com/cli/shrinkwrap#building-shrinkwrapped... [2] https://greenkeeper.io/
https://github.com/crohr/pkgr is the only thing that’s nearly similar, and it does a few things differently than Cartage (https://github.com/KineticCafe/cartage for anyone else following this subthread). Pkgr is a little more opinionated on using the OS environment and making sure that OS-level dependencies are fully declared…I leave that to Ansible, for the most part.
Some rely on gcc, but again, it can be run from a local directory. Various dynamic libraries can be looked up by setting environment variables or passing parameters.
I think Nix is a great idea, but I'm not aware of an argument for hardcoding those root paths which truly holds water. It makes sense of NixOS, sure, but that's because it's an environment with top-to-bottom control.
The rest of us have to be good and share our toys.
As for why those paths have to be hard-coded: they don't it's a config variable you can change. When you change it you give up using other people's binaries so you have to build everything yourself, which is exactly like every other distribution system that relies on hard-coded /usr/whatever paths for binaries. The FHS standard is incompatible with Nix's guarantees and dropping it is where it gets all of its power, so it has to use something different, and it chose to make that path /nix/store/whatever