I wish there was a way to either shebang something like this or build a wheel that has the full venv inside.
uv has support for it: https://docs.astral.sh/uv/guides/scripts/#running-a-script-w... (which only helps if your team is all in on uv, but maybe they are)
https://discuss.python.org/t/pep-722-723-decision/36763 contains the reasoning for accepting 723 and rejecting 722.
> A user facing CLI that is capable of executing scripts. If we take Hatch as an example, the interface would be simply hatch run /path/to/script.py [args] and Hatch will manage the environment for that script. Such tools could be used as shebang lines on non-Windows systems e.g. #!/usr/bin/env hatch run
https://micro.webology.dev/2024/08/21/uv-updates-and.html shows an example with uv:
> With this new feature, I can now instruct users to run uv run main.py without explaining what a venv or virtualenv is, plus a long list of requirements that need to be passed to pip install.
That ends:
> PEP 723 also opens the door to turning a one-file Python script into a runnable Docker image that doesn’t even need Python on the machine or opens the door for Beeware and Briefcase to build standalone apps.
Pyenv + poetry already gives you ability to "pull in local dependencies". Yes, you have to create a virtual environment and it's not "ad-hoc".
But if you're going to pull in a bunch of libraries, WHY would you want to invoke python and all your work dependencies on a one liner? Isn't it much better and easier to just spell-out the dependencies in a pyproject.toml? How "ad-hoc" are we talking here?
Previously I could allocate a whole week to setup initial scaffold for the project. Also more tools - more failure points, so I can flex on stupid juniors how smart I am. Now I can’t even go to pee with how fast and easy this freaking uv is. WTF.
I do like the idea of getting rid of pyenv though. And since poetry has failed to become as widespread as I hoped, maybe uv has a better shot?
So I can just do uv {package} for a quick and dirty global install. I'm so used to pip install being global by default just making this shorthand makes things a bit easier.
i would highly recommend only using —-system in a docker container or similar
uv run -q --with pandas==2.1.4 python -c "import pandas; print(pandas.__version__)" 2.1.4
Examples include, quant libraries, in-house APIs/tools, etc.
Conda packages general binary packages, not just python packages. uv is just python packages.
When we talk about local dependencies in the Python packaging ecosystem, it's usually adding some package on your file system to your environment. The existing title made me think this would be about the `[tool.uv.sources]` feature.
Really, it's about how we create environments on-demand and make it trivial to add packages to your environment or try other Python versions without mutating state.
not with this attitude of getting scared of things by watching someone doing something, for sure
[0]: https://flox.dev/
aka how to say that you've never really tried learning Nix without saying it directly.
I use Windows, and not WSL. Nix does literally nothing for me.
Worked great except for conda making the terrible mistake of compressing package files with bzip2 which took forever to decompress for huge packages.
I see no reason you can't install any kind of non-Python thing that a Python system wants with uv because a wheel is just a ZIP file, so long as it doesn't need to be installed in a particular place you can just unpack it and go.
The problem is all the things that do need to be installed in a particular place. Linux seems to have a lot of those, especially if they're build-time dependencies for something else. Hence the need for, and slow development of, https://peps.python.org/pep-0725/ (relevant background, though I'm sure you know this sort of stuff: https://pypackaging-native.github.io/ ).
- pip doesn't handle your Python executable, just your Python dependencies. So if you want/need to swap between Python versions (3.11 to 3.12 for example), it doesn't give you anything. Generally people use an additional tool such as pyenv to manage this. Tools like uv and Poetry do this as well as handling dependencies
- pip doesn't resolve dependencies of dependencies. pip will only respect version pinning for dependencies you explicitly specify. So for example, say I am using pandas and I pin it to version X. If a dependency of pandas (say, numpy) isn't pinned as well, the underlying version of numpy can still change when I reinstall dependencies. I've had many issues where my environment stopped working despite none of my specified dependencies changing, because underlying dependencies introduced breaking changes. To get around this with pip you would need an additional tool like pip-tools, which allows you to pin all dependencies, explicit and nested, to a lock file for true reproducibility. uv and poetry do this out of the box.
- Tool usage. Say there is a python package you want to use across many environments without installing in the environments themselves (such as a linting tool like ruff). With pip, you need to install another tool like pipx to install something that can be used across environments. uv can do this out of the box.
Plus there is a whole host of jobs that tools like uv and poetry aim to assist with that pip doesn't, namely project creation and management. You can use uv to create a new Python project scaffolding for applications or python modules in a way that conforms with PEP standards with a single command. It also supports workspaces of multiple projects that have separate functionality but require dependencies to be in sync.
You can accomplish a lot/all of this using pip with additional tooling, but its a lot more work. And not all use cases will require these.
pip install
I worked at a place where the engineering manager was absolutely exasperated with the problems we were having with building and deploying AI/ML software in Python. I had figured out pretty much all the problems after about nine months and had developed a 'wheelhouse' procedure for building our system reliably, but it was too late.
Not long after I sketched out a system that was a lot like uv but it was written in Python and thus had problems with maintaining its own stable Python enivronment (e.g. poetry seems to trash itself every six months or so.)
Writing uv in Rust was genius because it eliminates that problem of the system having a stable surface to stand on instead of pipping itself into oblivion, never mind that it is much faster than my system would have been. (My system had the extra feature that it used http range requests to extract the metadata from wheel files before pypi started letting you download the metadata directly.)
I didn't go forward with developing it because I argued with a lot of people who, like you, thought it was "the perfect being the enemy of the good" when it was really "the incorrect being the enemy of the correct." I'd worked on plenty of projects where I was right about the technology and wrong about the politics and I am so happy that uv has saved the Python community from itself.
and I think a killer feature is the ability to inline dependencies in your Python source code, then use: uv tool run <scriptname>
Your script code would like:
#!/usr/bin/env -S uv run --script # /// script # requires-python = ">=3.12" # dependencies = [ # "...", # "..." # ] # ///
Then uv will make a new venv, install the dependencies, and execute the script faster than you think. The first run is a bit slower due to downloads and etc, but the second and subsequent runs are a bunch of internal symlink shuffling.
It is really interesting. You should at least take a look at a YT or something. I think you will be impressed.
Good luck!
It also automatically creates venvs if you delete them. And it automatically updates packages when you run something with uv run file.py (useful when somebody may have updated the requirements in git). It also lets you install self contained (installed in a virtualenv and linked to ~/.local/bin which is added to your path)python tools (replacing pipx). It installs self contained python builds letting you more easily pick python version and specify it in a .python-version file for your project (replacing pyenv and usually much nicer because pyenv compiles them locally)
Uv also makes it easier to explore and say start a ipython shell with 2 libraries uv run --with ipython --with colorful --with https ipython
It caches downloads. Of course the http itself isn't faster but they're exploring things to speed that part up and since it's written in rust local stuff (like deleting and recreating a venv with cached packages) tends to be blazing fast
Also I don’t recognise errors and I don’t know which python versions generally work well with what.
I’ve had it happen so often with pip that I’d have something setup just fine. Let’s say some stable diffusion ui. Then some other month I want to experiment with something like airbyte. Can’t get it working at all. Then some days later I think, let’s generate an image. Only to find out that with pip installing all sorts of stuff for airbyte, I’ve messed up my stable diffusion install somehow.
Uv clicked right away for me and I don’t have any of these issues.
Was I using pip and asdf incorrectly before? Probably. Was it worth learning how to do it properly in the previous way? Nope. So uv is really great for me.
I saw a discourse reply that cited some sort of possible security issue but that was basically it and that means that the only way to get that functionality is to not use pip. It's really not a lot of major stuff, just a lot of little paper cuts that makes it a lot easier to just use something else once your project gets to a certain size.
However, the speed alone is reason enough to switch. Try it once and you will be sold.
now though, yes unequivocally you are missing out.
it does virtualenv, it does pyenv, it does pip, so all thats managed in once place.
its much faster than pip.
its like 80% of my workflow now.
I've used pip, pyenv, poetry, all are broken in one way or another, and have blind spots they don't serve.
If your needs are simple (not mixing Python versions, simple dependencies, not packaging, etc) you can do it with pip, or even with tarballs and make install.
Using uv means your project will have well defined dependencies.
The only way to justify VC money is a plot to take over the ecosystem and then profit off of a dominant position. (e.g. the Uber model)
I've heard a little bit about UV's technical achievements, which are impressive, but technical progress isn't the only metric.
> I haven't felt like it's a minor improvement on what I'm using
means that this:
> I'd love if we standardized on it as a community as the de facto default
…probably shouldn’t happen. The default and de facto standard should be something that doesn’t get put on a pedestal but stays out of the way.
It would be like replacing the python repl with the current version of ipython. I’d say the same thing, that it isn’t a minor improvement. While I almost always use ipython now, I’m glad it’s a separate thing.
The problem is that in the python ecosystem there really isn't a default de facto standard yet at all. It's supposed to be pip, but enough people dislike pip that it's hard as a newcomer to know if it's actually the standard or not.
The nice thing about putting something like this on a pedestal is that maybe it could actually become a standard, even if the standard should be simple and get out of the way. Better to have a standard that's a bit over the top than no standard at all.
…Not with IPython. But with an implementation written in Python instead of C, originating from the PyPy project, that supports fancier features like multi-line editing and syntax highlighting. See PEP 762.
I was apprehensive when I heard about it, but then I had the chance to use it and it was a very nice experience.
This to me is unachievable. Perfection is impossible. On the the way there if the community and developers coalesced around a single tool then maybe we can start heading down the road to perfectionism.
What people seem to miss about Pip is that it's by design, not a package manager. It's a package installer, only. Of course it doesn't handle the environment setup for you; it's not intended for that. And of course it doesn't keep track of what you've installed, or make lock files, or update your `pyproject.toml`, or...
What it does do is offer a hideously complex set of options for installing everything under the sun, from everywhere under the sun. (And that complexity has led to long-standing, seemingly unfixable issues, and there are a lot of design decisions made that I think are questionable at best.)
Ideas like "welllll I use poetry but pyenv works or you could use conda too" are incoherent. They're for different purposes and different users, with varying bits of overlap. The reason people are unsatisfied is because any given tool might be missing one of the specific things they want, unless it's really all-in-one like Uv seems like it intends to be eventually.
But once you have a truly all-in-one tool, you notice how little of it you're using, and how big it is, and how useless it feels to have to put the tool name at the start of every command, and all the specific little things you don't like about its implementation of whatever individual parts. Not to mention the feeling of "vendor lock-in". Never mind that I didn't pay money for it; I still don't want to feel stuck with, say, your build back-end just because I'm using your lock-file updater.
In short, I don't want a "package manager".
I want a solid foundation (better than Pip) that handles installing (not managing) applications and packages, into either a specified virtual environment or a new one created (not managed, except to make it easy to determine the location, so other tools can manage it) for the purpose. In other words, something that fully covers the needs of users (making it possible to run the code), while providing only the bare minimum on top of that for developers - so that other developer tools can cooperate with that. And then I want specialized tools for all the individual things developers need to do.
The specialized tools I want for my own work all exist, and the tools others want mostly exist too. Twine uploads stuff to PyPI; `build` is a fine build front-end; Setuptools would do everything I need on the back-end (despite my annoyances with it). I don't need a lockfile-driven workflow and don't readily think in those terms. I use pytest from the command line for testing and I don't want a "package manager" nor "workflow tool" to wrap that for me. If I needed any wrapping there I could do a few lines of shell script myself. If anything, the problem with these tools is doing too much, rather than too little.
The base I want doesn't exist yet, so I've started making it myself. Pipx is a big step in the right direction, but it has some arbitrary limitations (I discuss these and some workarounds in my recent blog post https://zahlman.github.io/posts/2025/01/07/python-packaging-... ) and it's built on Pip so it inherits those faults and is that much bigger. Uv is even bigger still for the compiled binary, and I would only be using the installation parts.
Node.js's replacement for virtualenv is literally just a folder named "node_modules". Meanwhile python has an entire tool with strange ideosyncracies that you have to pay attention to otherwise pip does the wrong thing by default.
It is as if python is pretending to be a special snowflake where installing libraries into a folder is this super hyper mega overcomplicated thing that necessitates a whole dedicated tool just to manage, when in reality in other programming languages nobody is really thinking about that the fact that the libraries end up in their build folders. It just works.
So again you're pretending that this is such a big deal that it needs a whole other tool, when the problem in question is so trivial that another tool is adding mental overhead with regard to the microscopic problem at hand.
# /// script
# requires-python = ">=3.12"
# dependencies = [
# "pandas",
# ]
# ///
I never liked pyenv because I really don't see the point/benefit building every new version of Python you want to use. There's a reason I don't run Gentoo or Arch anymore. I'm very happy that uv grabs pre-compiled binaries and just uses those.
So far I have used it to replace poetry (which is great btw) in one of my projects. It was pretty straightforward, but the project was also fairly trivial/typical.
I can't fully replace pipx with it because 'uv tool' currently assumes every Python package only has one executable. Lots of things I work with have multiple, such as Ansible and Jupyterlab. There's a bug open about it and the workarounds are not terrible, but it'd be nice if they are able to fix that soon.
Realistically, the options on Linux are the uv way, the pyenv way (download and compile on demand, making sure users have compile-time dependencies installed as part of installing your tool), and letting users download and compile it themself (which is actually very easy for Python, at least on my distro). Compiling Python is not especially fast (around a full minute on my 4-core, 10-year-old machine), although I've experienced much worse in my lifetime. Maybe you can get alternate python versions directly from your distro or a PPA, but not in a way that a cross-distro tool can feasibly automate.
On Windows the only realistic option is the official installer.
1. `uvx --from git+https://github.com/httpie/cli httpie` 2. https://simonwillison.net/2024/Aug/21/usrbinenv-uv-run/ uv in a shebang
I’m sure it was already possible with shebangs and venv before, but uv really brings the whole experience together for me so I can write python scripts as freely as bash ones.
pyenv virtualenv python3.12 .venv
.venv/bin/python -m pip install pandas
.venv/bin/python
E.g. if the thing you run invokes python itself, it will use the system python, not the venv one in the first case.
uv run --python 3.12 --with label-studio label-studio
Made my life so much easier
uvx --from 'huggingface_hub[cli]' huggingface-cli
This is of course bad practice. What I would like instead is what PHP's composer does: installing stuff automatically changes pyprpject.toml (or whatever the standard will be with uv), automatically freezes the versions, and then it is on git diff to tell me what I did last night, I'll remove a couple of lines from that file, run composer install and it will remove packages not explicitly added to my config from the environment. Does this finally get easy to achieve with uv?
Every time when I have to reorganize or upgrade my AI repos, it’s yet another 50GB writes to my poor ssd. Half of it is torch, another half auto-downloaded models that I cannot stop because they become “downloaded” and you never know how to resume it back or even find where they are cause python logging culture is just barbaric.
I have used virtualenvwrapper before and it was very convenient to have all virtual environments stored in one place, like ~/.cache/virtualenvs.
The .venv in the project directory is annoying because when you copy folder somewhere you start copying gigabytes of junk. Some tools like rsync can't handle CACHEDIR.TAG (but you can use --exclude .venv)
NeutralCrane has a really helpful comment below[0], would love to have a more thorough post on everything!
Note, I use PyPI for most of my day-to-day work, so I say this with love!
The author says that a normal route would be:
- Take the proper route:
- Create a virtual environment
- pip install pandas
- Activate the virtual environment
- Run python
In addition, in my opinion this is the thing that would sucks about UV to have different functions being tied to a single tool execution.
It is a breeze to be able to activate a venv, and be done with it, being able to run multiple times your program in one go, even with crashes, being able to install more dependencies, test it in REPL, ...
I disagree though it is activated immediately, or at least to me with venv I always have to activate it explicitly.
It can be used effectively, but does not make it easy to do so.
uv makes itself harder to misuse
I do hope the situation changes one day. Python packaging is such a mess, but Poetry is good enough and actually works, so I'll stick with it for now.
Can uv work with that?
Edit: might be possible now? https://simonwillison.net/2024/Dec/19/one-shot-python-tools/
https://packaging.python.org/en/latest/specifications/inline...
Especially useful if the script has dependencies on packages in private repos.
Has been super useful to write single-file tools/scripts which LLMs can understand and run easily.
Use Nix for Python version as well as other bin deps, and virtualenv + pip-tools for correct package dependency resolution.
Waiting 4s for pip-tools instead of 1ms for uv doesn't change much if you only run it once a month.
A simple example a came across is having to rename some files:
1. you just open the shell in the location you want
2. and run this command:
uv run https://raw.githubusercontent.com/SimonB97/MOS/main/AITaskRu... "check the contents of the .md files in the working dir and structure them in folders"
There's a link to Magentic-1 docs and further info in the repo: https://github.com/SimonB97/MOS/tree/main/AITaskRunner (plus two other simple scripts).
I would want distributed projects to do things properly, but as a way to shorthand a lot of futzing about? It's excellent
python's wheels are falling off at an ever faster and faster rate
But there are a shocking number of install instructions that offer $(npm i -g) and if one is using Homebrew or nvm or a similar "user writable" node distribution, it won't prompt for sudo password and will cheerfully mangle the "origin" node_modules
So, it's the same story as with python: yes, but only if the user is disciplined
Now ruby drives me fucking bananas because it doesn't seem to have either concept: virtualenvs nor ./ruby_modules
That said, I agree that `npm i -g` is a poor system package manager, and you should typically be using Homebrew or whatever package manager makes the most sense on your system. That said, `npx` is a good alternative if you just want to run a command quickly to try it out or something like that.
This whole installing the same dependencies a million times across different projects in Python and Node land is completely insane to me. Ruby has had the only sane package manager for years. Cargo too, but only because they copied Ruby.
Node has littered my computer with useless files. Python’s venv eat up a lot of space unnecessarily too.
As long as you're in the ruby-install/chruby ecosystem and managed to avoid the RVM mess then the tooling is so simple that it doesn't really get any attention. I've worked exclusively with virtualenvs in ruby for years.
(i) wrongly configured character encodings (suppose you incorporated somebody else's library that does a "print" and the input data contains some invalid characters that wind up getting printed; that "print" could crash a model trainer script that runs for three days if error handling is set wrong and you couldn't change it when the script was running, at most you could make the script start another python with different command line arguments)
(ii) site-packages; all your data scientist has to do is
pip install --user
(iii) "python" built into Linux by default. People expected Python to "just work" but it doesn't "just work" when people start installing stuff with pip because you might be working on one thing that needs one package and another thing that needs another package and you could trash everything you're doing with python in the process of trying to fix it.
Unfortunately python has attracted a lot of sloppy programmers who think virtualenv is too much work and that it's totally normal for everything to be broken all the time. The average data scientist doesn't get excited when it crumbles and breaks, but you can't just call up some flakes to fix it. [1]
I wish they would stick to semantic versioning tho.
I have used two projects that got stuck in incompatible changes in the 3.x Python.
That is a fatal problem for Python. If a change in a minor version makes things stop working, it is very hard to recommend the system. A lot of work has gone down the drain, by this Python user, trying to work around that
Then in their main code they imported the said error.py but unfortunately numpy library also has an error.py. So the user was getting very funky behavior.
Run npm install
Delete node_modules and wait 30minutes because it takes forever to delete 500MB worth of 2 million files.
Do an npm install again (or yarn install or that third one that popped up recently?)
Uninstall/Upgrade npm (or is it Node? No wait, npx I think. Oh well, used to be node + npm, now it's something different.)
Then do steps 1 to 3 again, just in case.
Hmm, maybe it's the lockfile? Delete it, one of the juniors pushed their version to the repo without compiling maybe. (Someone forgot to add it to the gitignore file?)
Okay, do steps 1 to 3 again, that might have fixed it.
If you've gotten here, you are royally screwed and should try the next javascript expert, he might have seen your error before.
So no, I'm a bit snarky here, but the JS ecosystem is a clustermess of chaos and should rather fix it's own stuff first. I have none of the above issues with python, a proper IDE and out of the box pip.
Most packing managers are developed.
Pnpm is engineered.
It’s one of the few projects I donate to on GitHub
Similar problems with runtime version management (need to use nvm for sanity, using built-in OS package managers seems to consistently result in tears).
More package managers and interactions (corepack, npm, pnpm, yarn, bun).
Bad package interop (ESM vs CJS vs UMD).
More runtimes (Node, Deno, Bun, Edge).
Then compound this all with the fact that JS doesn't have a comprehensive stdlib so your average project has literally 1000s of dependencies.
Will also note that in my years of js experience I've hardly ever run into module incompatibilities. It's definitely gnarly when it happens, but wouldn't consider this to be the same category of problem as the confusion of setting up python.
Hopefully uv can convince me that python's environment/dependency management can be easier than JavaScript's. Currently they both feel bad in their own way, and I likely prefer js out of familiarity.
In practice I find this a nuisance but a small one. I wish there had been a convention that lets the correct version of Node run without me manually having to switch between them.
> More package managers and interactions (corepack, npm, pnpm, yarn, bun).
But they all work on the same package.json and node_modules/ principle, afaik. In funky situations, incompatibilities might emerge, but they are interchangeable for the average user. (Well, I don't know about corepack.)
> Bad package interop (ESM vs CJS vs UMD).
That is a whole separate disaster, which doesn't really impact consuming packages. But it does make packaging them pretty nasty.
> More runtimes (Node, Deno, Bun, Edge).
I don't know what Edge is. Deno is different enough to not really be in the same game. I find it hard to see the existence of Bun as problematic: it has been a bit of a godsend for me, it has an amazing ability to "just work" and punch through Typescript configuration issues that choke TypeScript. And it's fast.
> Then compound this all with the fact that JS doesn't have a comprehensive stdlib so your average project has literally 1000s of dependencies.
I guess I don't have a lot of reference points for this one. The 1000s of dependencies is certainly true though.
You basically need to just remember to never call python directly. Instead use uv run and uv pip install. That ensures you're always using the uv installed python and/or a venv.
Python based tools where you may want a global install (say ruff) can be installed using uv tool
uv itself is the only Python tool I install globally now, and it's a self-contained binary that doesn't rely on Python. ruff is also self-contained, but I install tools like ruff (and Python itself) into each project's virtual environment using uv. This has nice benefits. For example, automated tests that include linting with ruff do not suddenly fail because the system-wide ruff was updated to a version that changes rules (or different versions are on different machines). Version pinning gets applied to tooling just as it does to packages. I can then upgrade tools when I know it's a good time to deal with potentially breaking changes. And one project doesn't hold back the rest. Once things are working, they work on all machines that use the same project repo.
If I want to use Python based tools outside of projects, I now do little shell scripts. For example, my /usr/local/bin/wormhole looks like this:
#!/bin/sh
uvx \
--quiet \
--prerelease disallow \
--python-preference only-managed \
--from magic-wormhole \
wormhole "$@"I don't understand why people would rather do this part specfically, rather than activate a venv.
Also we don't have a left pad scale dependency ecosystem that makes version conflicts such a pressing issue.
we don't have a left pad scale dependency ecosystem that makes version conflicts such a pressing issue
TensorFlow.
I have to imagine that the python maintainers listen for what the community needs and hear a thousand voices asking for a hundred different packaging strategies, and a million voices asking for the same language features. I can forgive them for prioritizing things the way they have.
Get me the deps this project needs, get them fast, then them correctly, all with minimum hoops.
Cargo and deno toolchains are pretty good too.
Opam, gleam, mvn/gradle, stack, npm/yarn, nix even, pip/poetry/whatever-python-malarkey, go, composer, …what other stuff have i used in the past 12 months… c/c++ doesn’t really have a first class std other than global sys deps (so ill refer back to nix or os package managers).
Getting the stuff you need where you need it is always doable. Some toolchains are just above and beyond, batteries included, ready for productivity.
I agree that the old days of "there are 15 dep managers, good luck" was high chaos. And those who do cutesy shit like using "replace" in their go.mod[1] is sus but as far as dx $(go get) that caches by default in $XDG_CACHE_DIR and uses $GOPROXY I think is great
1: https://github.com/opentofu/opentofu/blob/v1.9.0/go.mod#L271
If you want Numpy (one of the most popular Python packages) on a system that doesn't have a pre-built wheel, you'll need to do that. Which is why there are, by my count, 54 different pre-built wheels for Numpy 2.2.1.
And that's just the actual installation process. Package management isn't solved because people don't even agree on what that entails.
The only way you avoid "worry about modifying the global environment" is to have non-global environments. But the Python world is full of people who refuse to understand that concept. People would rather type `pip install suspicious-package --break-system-packages` than learn what a venv is. (And they'll sometimes do it with `sudo`, too, because of a cargo-cult belief that this somehow magically fixes things - spoilers: it's typically because the root user has different environment variables.)
Which is why this thread happened on the Python forums https://discuss.python.org/t/the-most-popular-advice-on-the-... , and part of why the corresponding Stack Overflow question https://stackoverflow.com/questions/75608323 has 1.4 million views. Even though it's about an error message that was carefully crafted by the Debian team to tell you what to do instead.
This makes a big difference. There is also the social problem of Python community with too loud opinions for making a good robust default solution.
But same has now happened for Node with npm, yarn and pnpm.
It has a super limited compiled extensions ecosystem, plugin ecosystem and is not used as a system language in mac and linux.
And of course node is much more recent and the community less diverse.
tldr: node is playing in easy mode.
If you change your pyproject.toml file manually, uv sync [1] will update your environment accordingly.
[0]: https://docs.astral.sh/uv/guides/projects/#managing-dependen... [1]: https://docs.astral.sh/uv/reference/cli/#uv-sync
Whatever, I think I'll try it for myself later today. It's long overdue.
pyproject.toml represents an inter-project standard and Charlie Marsh has committed to sticking with it, along with cooperating with future Python packaging PEPs. But while you can list transitive dependencies, specify exact versions etc. in pyproject.toml, it's not specifically designed as a lockfile - i.e., pyproject.toml is meant for abstract dependencies, where an installer figures out transitively what's needed to support them and decides on exact versions to install.
The current work for specifying a lockfile standard is https://peps.python.org/pep-0751/ . As someone else pointed out, uv currently already uses a proprietary lockfile, but there has been community interest in trying to standardize this - it just has been hard to find agreement on exactly what it needs to contain. (In the past there have been proposals to expand the `pyproject.toml` spec to include other information that lockfiles often contain for other languages, such as hashes and supply-chain information. Some people are extremely against this, however.)
As far as I know, uv isn't going to do things like analyzing your codebase to determine that you no longer need a certain dependency that's currently in your environment and remove it (from the environment, lock file or `pyproject.toml`). You'll still be on the hook for figuring out abstractly what your project needs, and this is important if you want to share your code with others.
Sure, that's not what I meant (unless we call pyproject.toml a part of your codebase, which it kinda is, but that's probably not what you meant).
In fact, as far as I can tell from your answer, Python does move in the direction I'd like it to move, but it's unclear by how far it will miss and if how uv handles it is ergonomical.
As I've said, I think PHP's composer does a very good job here, and to clarify, this is how it works. There are 2 files: composer.json (≈pyproject.toml) and composer.lock (≈ PEP751) (also json). The former is kinda editable by hand, the latter you ideally never really touch. However, for the most part composer is smart enough that it edits both files for you (with some exceptions, of course), so every time I run `composer require your/awesomelib` it
1) checks the constraints in these files
2) finds latest appropriate version of your/awesomelib (5.0.14) and all its dependencies
3) writes "your/awesomelib": "^5.0"
4) writes "your/awesomelib": "5.0.14" and all its dependencies to composer.lock (with hashsums, commit ids and such)
It is a good practice to keep both inside of version control, so when I say "git diff tells me what I did last night" it means that I'll also see what I installed. If (as usual) most of it is some useless trash, I'll manually remove "your/awesomelib" from composer.json, run `composer install` and it will remove it and all its (now unneeded) dependencies. As the result, I never need to worry about bookkeeping, since composer does it for me, I just run `composer require <stuff>` and it does the rest (except for cases when <stuff> is a proprietary repo on company's gitlab and such, then I'll need slightly more manual work).
That is, what I hope to see in Python one day (10 years later than every other lang did it) is declarative package management, except I don't want to have to modify pyproject.toml manually, I want my package manager do it for me, because it saves me 30 seconds of my life every time I install something. Which accumulates to a lot.
Usually, you can directly make a pre-built wheel, and then an installer like Pip or uv can just unpack that into the environment. If it needs to be build on the user's machine, then you offer an sdist, which specifies its build backend. The installer will act as a build frontend, by downloading and setting up the specified backend and asking it to make a wheel from the sdist, then installing the wheel.
Poetry's build backend (`poetry.masonry`) doesn't build your external dependencies unless a) you obtain an sdist and b) the sdist says to use that backend. And in these cases, it doesn't matter what tools you're using. Your installer (which could be Pip, which is not a package manager in any meaningful sense) can work with `poetry.masonry` just fine.
If you can give a much more specific, simple, reproducible example of a problem you encountered with external dependencies and uv, I'll be happy to try to help.
> Package managers are for keeping track of which pieces of code you need in your project's environment. Build systems are for... building the code, so that it can actually be used in an environment.
This probably means something to the developers of the package managers and build systems, but to me, as a Python developer who wants to be able to publish a pure Python CLI program to PyPI, it seems like a distinction without a difference.
Personally I consider this one of uv's greatest strengths. The inflexibility and brittleness of Poetry's build system is what made me give up on poetry entirely. Had poetry made it easy to plug in a different build system I might never have tried uv.
When I first learned Python, typing python and seeing >>> and having it evaluate what I typed as if it appeared in a file was a good experience.
Now that I use python a lot, ipython is more out of the way to me than the built-in python repl is, because it lets me focus on what I'm working on, than limitations of a tool.
For many of the individual tasks, there are small Unix-philosophy tools that work great. Why is typing `poetry publish` better than typing `twine upload`? (And it would be just `twine`, except that there's a `register` command that PyPI doesn't need in the first place, and a `check` command that's only to make sure that other tools did their job properly - and PyPI will do server-side checks anyway.) Why is typing `poetry run ...` better than activating the venv with the script it provided itself, and then doing the `...` part normally?
An all-in-one tool works when people agree on the scope of "all", and you can therefore offer just one of them and not get complaints.
IME it is common to see a process tree with multiple invocations of Python in a ancestor relationship with other processes in between.
A more Pythonic way of doing this might be __pyproject__ bit that has the tiiiiny snag of needing to execute the file to figure out its deps. I would have loved if __name__ == "pyproject" but while neat and tidy it is admittedly super confusing for beginners, has a "react hooks" style gotcha where to can't use any deps in that block, and you can't use top level imports. The comment was really the easiest way.
I did notice how Homebrew sets env GEM_HOME=<Cellar>/libexec GEM_PATH=<Cellar>/libexec (e.g. <https://github.com/Homebrew/homebrew-core/blob/9f056db169d5f...>) but, similar to my node experience, since I am a ruby outsider I don't totally grok what isolation that provides
I might just blow your mind here:
$ time python -m venv with-pip
real 0m3.248s
user 0m3.016s
sys 0m0.219s
$ time python -m venv --without-pip without-pip
real 0m0.054s
user 0m0.046s
sys 0m0.009s
$ source without-pip/bin/activate
(without-pip) $ ~/.local/bin/pip --python `which python` install package-installation-test
Collecting package-installation-test
Using cached package_installation_test-1.0.0-py3-none-any.whl.metadata (3.1 kB)
Using cached package_installation_test-1.0.0-py3-none-any.whl (3.1 kB)
Installing collected packages: package-installation-test
Successfully installed package-installation-test-1.0.0
> a killer feature is the ability to inline dependencies in your Python source code, then use: uv tool run <scriptname>
Yes, Uv implements PEP 723 "Inline Script Metadata" (https://peps.python.org/pep-0723/) - originally the idea of Paul Moore from the Pip dev team, whose competing PEP 722 lost out (see https://discuss.python.org/t/_/29905). He's been talking about a feature like this for quite a while, although I can't easily find the older discussion. He seems to consider it out of scope for Pip, but it's also available in Pipx as of version 1.4.2 (https://pipx.pypa.io/stable/CHANGELOG/).
> The first run is a bit slower due to downloads and etc, but the second and subsequent runs are a bunch of internal symlink shuffling.
Part of why Pip is slow at this is because it insists on checking PyPI for newer versions even if it has something cached, and because its internal cache is designed to simulate an Internet connection and go through all the usual metadata parsing etc. instead of just storing the wheels directly. But it's also just slow at actually installing packages when it already has the wheel.
In principle, nothing prevents a Python program from doing caching sensibly and from shuffling symlinks around.
I don't care if pip technically can do something. The fact that I explicitly have to mess around with venvs and the stuff is already enough mental overhead that I disregard it.
I'm a python programmer at my job, and I've hated the tooling for years. Uv is the first time I actually like working with python.
The problems you're describing, or seeing solved with uv, don't seem to be about a problem with the design of virtual environments. (Uv still uses them.) They're about not having the paradigm of making a venv transiently, as part of the code invocation; or they're about not having a built-in automation of a common sequence of steps. But you can do that just as well with a couple lines of Bash.
I'm not writing any of this to praise the standard tooling. I'm doing it because the criticisms I see most commonly are inaccurate. In particular, I'm doing it to push back against the idea that a non-Python language is required to make functional Python tooling. There isn't a good conceptual reason for that.
If you want to distribute it to be installable by others, yes. Except that at least for now, installers will assume Setuptools by default if you don't mention anything in your `pyproject.toml`.
>but to me, as a Python developer who wants to be able to publish a pure Python CLI program to PyPI,
If you're making pure Python projects, the actual build process is trivial and every build system will do just fine. But again, the build system you choose builds your code, not your external dependencies. Whatever you put in `pyproject.toml` here has nothing to do with the packages that you install. It has to do with other people installing your package (and you taking steps to make that easier). So "external dependencies don't build the same as on Poetry" makes no sense in this context. If you need to build external dependencies (i.e. they don't come pre-built for your system), they will automatically be built with the build system that they choose.
>it seems like a distinction without a difference.
Let me try again: When you use a package manager, it's so that you can keep track of which code from other people you're using. When you choose a build system and mention it in `pyproject.toml`, it's so that other people can use your code. (For your pure Python project, you will normally run the build system locally - https://pradyunsg.me/blog/2022/12/31/wheels-are-faster-pure-... . But the overall packaging ecosystem is designed so that you can push part of that work onto the end user, when it makes sense to do so.)
With this, you can manage the dependency list via `uv add/remove` (or the `pyproject.toml` directly), and run `uv sync` to add/remove any dependencies to the managed virtual env.
Edit: adding about uv add/uv remove
https://packaging.python.org/en/latest/specifications/inline...
There was previous discussions about uncoupling the stdlib (python libraries) from the release and have them being released independently, but I can’t remember why that died off
That's the usual case, but it can definitely also happen because of the language - see https://stackoverflow.com/questions/51337939 .
> There was previous discussions about uncoupling the stdlib (python libraries) from the release and have them being released independently, but I can’t remember why that died off
This sort of thing is mainly a social problem.
Well yes, but sometimes the features provided by stdlib can feel pretty 'core'. I remember that how dataclasses work for example changed and broke a lot of our code when we upgraded from 3.8 to 3.10,
There have also been breaking changes in the C API. We have one project at work that is stuck at 3.11 since a third party dependency won't build on 3.12 without a rewrite.
For what it's worth, I think .tool-versions is slowly starting to creep into this space.
Mise (https://mise.jdx.dev/dev-tools/) and ASDF (https://asdf-vm.com/) both support it.
Big reason I prefer ASDF to nvm/rvm/etc right now is that it just automatically adjusts my versions when I cd into a project directory.
I've worked on/with some fairly large/complex python projects, and they almost never have any packaging issues that aren't just obvious errors by users. Yes, every once in a while we have to be explicit about a dependency because some dependent project isn't very strict with their versioning policy and their API layers.
I've seen someone having to do this in node like once every month, no matter which year, no matter which project or company.
As I commented here just now I never got pip. This explains it.
Node_modules doesn't support an isolated node interpreter distinct from what may be installed elsewhere on the machine. Third party tools are available that do that for node, but node_modules alone addresses a subset of the issues that venvs solve.
OTOH, its arguably undesirable that there isn't a convenient way in Python to do just what node_modules does without the extra stuff that venvs do, because there are a lot of use cases where that kind of solution would be sufficient and lower overhead.
But also, Uv uses them anyway. Because that's the standard. And if the Python standard were to use a folder like node_modules, Uv would follow suit, and so would I. And Uv would still be doing package management and workflow tasks that I'm completely uninterested in, and presenting an "every command is prefixed with the tool suite name* UI that I don't like.
There was a proposal for Python to use a folder much like node_modules: see https://discuss.python.org/t/pep-582-python-local-packages-d... . It went through years of discussion and many problems with the idea were uncovered. One big issue is that installing a Python package can install more than just the importable code; the other stuff has to be put somewhere sensible.
>So again you're pretending that this is such a big deal that it needs a whole other tool
I make no such claim. The tool I want is not for managing virtual environments. It's for replacing Pip, and offering a modicum of convenience on top of that. Any time you install a package, no matter whether you use Pip or uv or anything else, you have to choose (even if implicitly) where it will be installed. Might as well offer the option of setting up a new location, as long as we have a system where setup is necessary.
venv is a dedicated tool.
virtualenv is a different dedicated tool.
Pip isn't part of the standard library; the standard library `ensurepip` (called from `venv`) includes a vendored wheel for Pip (https://github.com/python/cpython/tree/main/Lib/ensurepip/_b...), and imports Pip from within that wheel to bootstrap it. (Wheels are zip files, and Python natively knows how to import code from a zip file, so this just involves some `sys.path` manipulation. The overall process is a bit complex, but it's all in https://github.com/python/cpython/blob/main/Lib/ensurepip/__... .)
This is why you get prompted to upgrade Pip all the time in new virtual environments (unless you preempt that with `--upgrade-deps`). The core development team still wants to keep packaging at arms length. This also allows Pip to be developed, released and versioned independently.
Npm is a pretty good example of what pip should be. Npm has had to compete with other package managers for a long time but has remained the standard simply because it actually has all the basic features that people expect out of a package manager. So other package managers can spin up using the npm registry providing slightly better experiences in certain ways, but npm covers the basics.
Pip really does not even cover the basics, hence the perpetual search for a better default.
If upgrading Spyder broke your environment, that's presumably because you were using the same environment that Spyder itself was in. (Spyder is also implemented in Python, as the name suggests.) However, IDEs for Python also like to try to do environment management for you (which may conflict with other tools you want to use specifically for the purpose). That's one of the reasons I just edit my code in Vim.
If updating dependencies breaks your code, it's ultimately the fault of the dependency (and their maintainers will in turn blame you for not paying attention to their deprecation warnings).
As for Spyder, it is included in the default Windows install of Anaconda (and linked to by the default Anaconda Navigator). As a new user doing package management via the GUI, it was not clear at all that Spyder was sharing dependencies with my project until things started breaking.
Anaconda was also half-baked in other ways: it broke if the Windows username contains UTF-8 characters, so I ended up creating a new Windows user just for that ML work. PITA.
It's so simple!
Create a image with the dependencies, then `podman run` it.
It's in their example for how to use requirements.txt: https://pip.pypa.io/en/stable/reference/requirements-file-fo...
Maybe there's some concrete example you have in mind though?
That means you either need to use pypi or do an extremely messy hack that involves adding the vendored package as a sub package to your main source, and then do some importlib black magic to make sure that everything uses said package.
https://github.com/pypa/pip/issues/6658
https://discuss.python.org/t/what-is-the-correct-interpretat...
I have yet to run into that particular case where the vendor didn't supply their own repo in favour of just providing the source directly. However I do use what are essentially vendor-supplied packages (distant teams in the org) and in those cases I just point at their GitLab/GitHub repo directly. Even for some components within my own team we do it this way.
And that's not even getting into this horseshit: https://github.com/opentofu/hcl/blob/v2.20.1/go.mod#L1 which apparently allows one to declare a repos _import_ path to be different from the url used to fetch it
I have a similar complaint about how in the world anyone would know how "gopkg.in/yaml.v3" secretly resolved to view-source:https://gopkg.in/yaml.v3?go-get=1
"Compares left pad with ml library backing the hottest AI companies of the cycle"
Left pad was ten years ago. Today is 2025-01-13 and we are still discussing how good yet another python PM presumably is. Even a hottest ML library can only do so much in this situation.
I've had this thought too. But it's not without its downsides. Users would have to know about and activate that venv in order to, say, play with system-provided GTK bindings. And it doesn't solve the problem that the user may manage dependencies for more than one project, that don't play nice with each other. If everything has its own venv, then what is the "real" environment even for?
I'm not totally sure what you're referring to, but I've definitely had a number of issues along the lines of:
- I have to use import, not require, because of some constraint of the project I'm working in - the module I'm importing absolutely needs to be required, not imported
I really don't have any kind of understanding of what the fundamental issues are, just a very painful transition point from the pre-ESM world to post.
I will note that node has embraced esm, and nowadays it's frowned upon to only publish cjs packages so this problem is shrinking every day. Also cool is some of the newer runtimes support esm/cjs interop.
But overall, computer literacy is really on the decline these days. Nowadays before you can teach programming in any traditional sense, you may have to teach the concept of a file system, then a command line...
You only have to do it once per version with this approach. Then you can create venvs from that base, and it's basically instantaneous if you do it `--without-pip`.
Here's a fun one (this was improved in 3.11, but some other names like `traceback.py` can still reproduce a similar problem):
/tmp$ touch token.py
/tmp$ py3.10
Python 3.10.14 (main, Jun 24 2024, 03:37:47) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> help()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/python/standard/lib/python3.10/_sitebuiltins.py", line 102, in __call__
import pydoc
File "/opt/python/standard/lib/python3.10/pydoc.py", line 62, in <module>
import inspect
File "/opt/python/standard/lib/python3.10/inspect.py", line 43, in <module>
import linecache
File "/opt/python/standard/lib/python3.10/linecache.py", line 11, in <module>
import tokenize
File "/opt/python/standard/lib/python3.10/tokenize.py", line 36, in <module>
from token import EXACT_TOKEN_TYPES
ImportError: cannot import name 'EXACT_TOKEN_TYPES' from 'token' (/tmp/token.py)
package organization.dns.name.this.and.that;
package this.and.that;
package their.this.and.that;
I am kind of iffy on golang's import (. "some/packge/for/side-effects") but at least it cannot suddenly mutate GOPATH[0]="/home/jimmy/lol/u/fucked" as one seems to be able to do on the regular with python
I am acutely aware that is (programmer|package|organization|culture)-dependent but the very idea that one can do that drives us rigorous people stark-raving
But, you know, top-level code (which can do anything) runs when you import any Python module (the first time), and Python code doesn't have to be in a package to get imported. (The standard library depends on this.)
Pretty much a nothing burger in Rust, so I disagree that items necessarily hate the concept. Maybe others haven’t done a good job with the UX?
If you give people an "easy way out" it is very hard to compel them to do it more rigorously. Look at the history of C++ namespaces as well as the non-acceptance of various "Modula" languages and Ada back in the 1980s.
Wait, what? Since when?
In Yarn 2/3, it needs to be added manually.
Yes. It does this because JavaScript enables it - the default import syntax uses a file path.
Python's default import syntax uses symbolic names. That allows you to do fun things like split a package across the filesystem, import from a zip file, and write custom importers, but doesn't offer a clean way to specify the version you want to import. So Pip doesn't try to install multiple versions either (which saves it the hassle of trying to namespace them). You could set up a system to make it work, but it'd be difficult and incredibly ugly.
Some other language ecosystems don't have this problem because the import is resolved at compile time instead.
[] For various broad* definitions of forever.
[*] Like, really, really broad**
[**] Maybe a week if you're lucky
In this case, instead of working with Python, you change how you manage everything!
> add about as many quality problems as they fix
Firstly, Node's `require` syntax predates the modern JS `import` syntax. It is related to some attempts at the time to create tools that could act like a module system for the browser (in particular RequireJS), but it is distinct in that a lot of the rules about how modules would be resolved were specifically designed to make sense with NodeJS and the `node_modules` system.
Secondly, although path imports are used for local imports, Node's `require` and `import` syntax both use symbolic names to refer to third-party packages (as well as built-in packages). If you have a module called `lodash`, you would write something like `import "lodash"`, and Node will resolve the name "lodash" to the correct location. This behaviour is in principle the same as Python's — the only change is the resolution logic, which is Node-specific, and not set by the Javascript language at all.
The part of NodeJS that _does_ enable this behaviour is the scoped module installation. Conceptually, NPM install modules in this structure:
* index.js (your code goes here)
* node_modules (top level third-party modules go here)
* react
* index.js (react source code)
* node_modules (react's dependencies go here)
* lodash/index.js
* vuejs
* index.js (vuejs source code)
* node_modules (vue's dependencies go here)
* lodash/index.js
* lodash
* index.js
Of course in practice, you usually don't want lots of versions of the same library floating around, so NPM also has a deduping system that attempts to combine compatible modules (e.g. if the version ranges for React, Vue, and Lodash overlap, a version will be chosen that works for all modules). In addition, there are various ways of removing excess copies of modules — by default, NPM flattens the tree in a way that allows multiple modules to "see" the same imported module, and tools like PNPM use symlinks to remove duplicated modules. And you mention custom importers in Python, but I believe Yarn uses them already in NodeJS to do the fun things you talk about like importing from zip files that are vendored in a repository.
All of the above would be possible in Python as well, without changing the syntax or the semantics of the `import` statement at all. However, it would probably break a lot of the ecosystem assumptions about where modules live and how they're packaged. And it's not necessarily required to fix the core Python packaging issues, so I don't think people in Python-land are investing much time in exploring this issue.
Basically, no, there is no fundamental reason why Python and Node have to have fundamentally different import systems. The two languages process imports in a very similar way, with the main difference being whether packages are imported in a nested way (Node) or as a flattened directory (Python).
So for me, in my specific case, the freetype-py repo has a rather big issue with Unicode paths (it will crash the app if the path is in Unicode).
There's a PR but it hasn't and probably won't get merged for dubious reasons.
The easy choice, the one that actually is the most viable, is to pull the repo with the patch applied, temporarily add it to my ./vendored folder and just ideally change the requirements.txt with no further changes (or need to create a new pypi package). But it's basically impossible since I just can't use relative paths like that.
Again it's rather niche but that's just one of the many problems I keep encountering. packaging anything with CUDA is still far worse, for example.
You can use the repo's setup to build a wheel, then tell pip to install from that wheel directly (in requirements.txt, give the actual path/name of the wheel file instead of an abstract dependency name). You need a build frontend for this - `pip wheel` will work and that's more or less why it exists; but it's not really what Pip is designed for overall - https://build.pypa.io/en/stable/ is the vanilla offering.
Take for example:
$ du -hs $HOMEBREW_PREFIX/Cellar/ansible/11.1.0_1/libexec/lib/python3.13/site-packages/* | gsort --human
103M /usr/local/Cellar/ansible/11.1.0_1/libexec/lib/python3.13/site-packages/botocore
226M /usr/local/Cellar/ansible/11.1.0_1/libexec/lib/python3.13/site-packages/ansible_collectionsBut packages that stay zipped aren't a thing any more - because it's the installer that gets to choose how to install the package, and Pip just doesn't do that. Pip unpacks the wheel mainly because many packages are written with the assumption that the code will be unpacked. For example, you can't `open` a relative path to a data file that you include with your code, if the code remains in a zip file. Back in the day when eggs were still a common distribution format, it used to be common to set metadata to say that it's safe to leave the package zipped up. But it turned out to be rather difficult to actually be sure that was true.
The wheel is also allowed to contain a folder with files that are supposed to go into other specified directories (specific ones described by the `sysconfig` standard library https://docs.python.org/3/library/sysconfig.html ), and Pip may also read some "entry point" metadata and create executable wrappers, and leave additional metadata to say things like "this package was installed with Pip". The full installation process for wheels as designed is described at https://packaging.python.org/en/latest/specifications/binary... . (Especially see the #is-it-possible-to-import-python-code-directly-from-a-wheel-file section at the end.)
The unzipped package doesn't just take up extra space due to unzipping, but also because of the .pyc cache files. A recent Pip wheel is 1.8 MiB packed and 15.2 MiB unpacked on my system; that's about 5.8 apparent MiB .py, 6.5 MiB .pyc, 2.1 MiB from wasted space in 4KiB disk blocks, and [s].8 MiB from a couple hundred folders.[/s] Sorry, most of that last bit is actually stub executables that Pip uses on Windows to make its wrappers "real" executables, because Windows treats .exe files specially.
The formula was
- Docker - Discipline = Chaos
Docker - Discipline = Chaos
Docker + Discipline = Order
- Docker + Discipline = Order
Sure today you might have five or six different databases for a site but it's not that different in my mind. Having way too many different versions of things installed is a vice, not a virtue.
Docker is great for making sure that magic bash script that brings the system up actually works again on someone else’s computer or after a big upgrade on your dev machine or whatever.
So many custom build scripts I’ve run into over the years have some kind of unstated dependency on the initial system they were written on, or explicit dependencies on something tricky to install, and as such are really annoying to diagnose later on, especially if they make significant system changes.
Docker is strictly better than a folder full of bash scripts and a Readme.txt. I would have loved having it when I had to operate servers like that with tons of websites running on them. So much nicer to be able to manage dependency upgrades per-site rather than server-wide, invariably causing something to quietly break on one of 200 sites.
My first production use of kubernetes started out because we put in the entirety of what we had to migrate to new hosting into spreadsheet, with columns for various parts of stack used by the websites, and figured we would go insane trying to pack it up - or we would lose the contract because we would be as expensive as the last company.
Could we package it nicely without docker? Yes, but the effort to package it in docker was smaller than packaging it in a way where it wouldn't conflict on a single host, because the simple script becomes way harder when you need to handle multiple versions of the same package, something that most distro do not support at all (these days I think we could have done it with NixOS, but that's a different kettle of deranged fishes)
And then the complexity of managing the stack was quickly made easier by turning each site into separate artifact (docker container) handled by k8s manifests (especially when it came to dealing with about 1000 domains across those apps).
So, theoretically discipline is enough, practical world is much dirtier though.
The trick isn't installing things, it's uninstalling them. Docker container is isolated in ways your bash script equivalent is not - particularly when first developing it, when you're bound to make an occasional mistake.
Sounds quite explicable: Docker image created by Hungarian devs perhaps?
Because I get other advantages of it. Giving in to overhead on one layer, doesn't mean I'm willing to give it up everywhere.
Can you give a concrete example, starting from a fresh venv, that causes a failure that shouldn't happen?
> but it was written in Python and thus had problems with maintaining its own stable Python enivronment
All it has to do is create an environment for itself upon installation which is compatible with its own code, and be written with the capability of installing into other environments (which basically just requires knowing what version of Python it uses and the appropriate paths - the platform and ABI can be assumed to match the tool, because it's running on the same machine).
This is fundamentally what uv is doing, implicitly, by not needing a Python environment to run.
But it's also what the tool I'm developing, Paper, is going to do explicitly.
What's more, you can simulate it just fine with Pip. Of course, that doesn't solve the issues you had with Pip, but it demonstrates that "maintaining its own stable Python environment" is just not a problem.
>Writing uv in Rust was genius because it eliminates that problem of the system having a stable surface to stand on instead of pipping itself into oblivion, never mind that it is much faster than my system would have been.
From what I can tell, the speed mainly comes from algorithmic issues, caching etc. Pip is just slow above and beyond anything Python forces on it.
An example. On my system, creating a new venv from scratch with Pip included (which loads Pip from within its own vendored wheel, which then runs in order to bootstrap itself into the venv) takes just over 3 seconds. Making a new venv without Pip, then asking a separate copy of Pip to install an already downloaded Pip wheel would be about 1.7 seconds. But making that venv and using the actual internal installation logic of Pip (which has been extracted by Pip developer Pradyun Gedam as https://github.com/pypa/installer ) would take about 0.25 seconds. (There's no command-line API for this; in my test environment I just put the `installer` code side by side with a driver script, which is copied from my development work on Paper.) It presumably could be faster still.
I honestly have no idea what Pip is doing the rest of that time. It only needs to unzip an archive and move some files around and perform trivial edits to others.
> (My system had the extra feature that it used http range requests to extract the metadata from wheel files before pypi started letting you download the metadata directly.)
Pip has had this feature for a long time (and it's still there - I think to support legacy projects without wheels, because I think the JSON API won't be able to provide the data since PyPI doesn't build the source packages). It's why the PyPI server supports range requests in the first place.
> I'd worked on plenty of projects where I was right about the technology and wrong about the politics and I am so happy that uv has saved the Python community from itself.
The community's politics are indeed awful. But Rust (or any other language outside of Python) is not needed to solve the problem.
But it commonly comes across that people think it can't be written in Python if it's to have XYZ features, and by and large they're wrong, and I'm trying to point that out. In particular, people commonly seem to think that e.g. Pip needs to be in the same environment to work, and that's just not true. There's a system in place that defaults to copying Pip into every environment so that you can `python -m pip`, but this is wasteful and unnecessary. (Pip is designed to run under the install environment's Python, but this is a hacky implementation detail. It really just needs to know the destination paths and the target Python version.)
It also happens that I care about disk footprint quite a bit more than most people. Maybe because I still remember the computers I grew up with.
$> sudo apt-get install python3.10 python3.11 python3.12
$> python3.11 -m venv .venv3.11
$> source .venv3.11/bin/activate
$> pip install -r requirements.txt
This assumes the Python version you need is available from your package manager's repo. This won't work if you want a Python version either newer or older than what is available.
> You are incorrect about needing to use an additional tool to install a "global" tool like `ruff`; `pip` does this by default when you're not using a virtual environment.
True, but it's not best practice to do that because while the tool gets installed globally, it is not necessarily linked to a specific python version, and so it's extremely brittle.
And it gets even more complex if you need different tools that have different Python version requirements.
And of course you could be working with multiple distros and versions of the same distro, production and dev might be different environment and tons of others concerns. You need something that just works across.
>True, but it's not best practice to do that because while the tool gets installed globally, it is not necessarily linked to a specific python version, and so it's extremely brittle.
"Globally" means installed with sudo. These are installed into the user folder under ~/.local/ and called a user install by pip.
I wouldn't call it "extremely brittle" either. It works fine until you upgrade to a new version of python, in which case you install the package again. Happens once a year perhaps.
The good part of this is that unused cruft will get left behind and then you can delete old folders in ~/.local/lib/python3.? etc. I've been doing this over a decade without issue.
This is simply incorrect. In fact the reason it gets stuck on resolution sometimes is exactly because it resolved transitive dependencies and found that they were mutually incompatible.
Here's an example which will also help illustrate the rest of my reply. I make a venv for Python 3.8, and set up a new project with a deliberately poorly-thought-out pyproject.toml:
[project]
name="example"
version="0.1.0"
dependencies=["pandas==2.0.3", "numpy==1.17.3"]
When I try to `pip install -e .` in the venv, Pip happily explains (granted the first line is a bit strange):
ERROR: Cannot install example and example==0.1.0 because these package versions have conflicting dependencies.
The conflict is caused by:
example 0.1.0 depends on numpy==1.17.3
pandas 2.0.3 depends on numpy>=1.20.3; python_version < "3.10"
To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip to attempt to solve the dependency conflict
> pip will only respect version pinning for dependencies you explicitly specify. So for example, say I am using pandas and I pin it to version X. If a dependency of pandas (say, numpy) isn't pinned as well, the underlying version of numpy can still change when I reinstall dependencies.
Well, sure; Pip can't respect a version pin that doesn't exist anywhere in your project. If the specific version of Pandas you want says that it's okay with a range of Numpy versions, then of course Pip has freedom to choose one of those versions. If that matters, you explicitly specify it. Other programs like uv can't fix this. They can only choose different resolution strategies, such as "don't update the transitive dependency if the environment already contains a compatible version", versus "try to use the most recent versions of everything that meet the specified compatibility requirements".
> To get around this with pip you would need an additional tool like pip-tools, which allows you to pin all dependencies, explicit and nested, to a lock file for true reproducibility.
No, you just use Pip's options to determine what's already in the environment (`pip list`, `pip freeze` etc.) and pin everything that needs pinning (whether with a Pip requirements file or with `pyproject.toml`). Nothing prevents you from listing your transitive dependencies in e.g. the [project.dependencies] of your pyproject.toml, and if you pin them, Pip will take that constraint into consideration. Lock files are for when you need to care about alternate package sources, checking hashes etc.; or for when you want an explicit representation of your dependency graph in metadata for the sake of other tooling.
> This assumes the Python version you need is available from your package manager's repo. This won't work if you want a Python version either newer or older than what is available.
I have built versions 3.5 through 3.13 inclusive from source and have them installed in /opt and the binaries symlinked in /usr/local/bin. It's not difficult at all.
> True, but it's not best practice to do that because while the tool gets installed globally, it is not necessarily linked to a specific python version, and so it's extremely brittle.
What brittleness are you talking about? There's no reason why the tool needs to run in the same environment as the code it's operating on. You can install it in its own virtual environment, too. Since tools generally are applications, I use Pipx for this (which really just wraps a bit of environment management around Pip). It works great; for example I always have the standard build-frontend `build` (as `pyproject-build`) and the uploader `twine` available. They run from a guaranteed-compatible Python.
And they would if they were installed for the system Python, too. (I just, you know, don't want to do that because the system Python is the system package manager's responsibility.) The separate environment don't matter because the tool's code and the operated-on project's code don't even need to run at the same time, let alone in the same process. In fact, it would make no sense to be running the code while actively trying to build or upload it.
> And it gets even more complex if you need different tools that have different Python version requirements.
No, you just let each tool have the virtual environment it requires. And you can update them in-place in those environments, too.
I just use pipx. Install guides suggest it, and it is only one character different from pip.
With Nix, it is very easy to run multiple versions of same software. The path will always be the same, meaning you can depend on versions. This is nice glue for pipx.
My pet peeve with Python and Vim is all these different package managers. Every once in a while a new one is out and I don't know if it will gain momentum. For example, I use Plug now in Vim but notice documentation often refers to different alternatives these days. With Python it is pip, poetry, pip search no longer working, pipx, and now uv (I probably forgot some things).
I just keep separate compiled-from-source versions of Python in a known, logical place; I can trivially create venvs from those directly and have Pip install into them, and pass `--python` to `pipx install`.
>With Python it is pip, poetry, pip search no longer working, pipx, and now uv (I probably forgot some things).
Of this list, only Poetry and Uv are package managers. Pip is by design, only an installer, and Pipx only adds a bit of environment management to that. A proper package manager also helps you keep track of what you've installed, and either produces some sort of external lock file and/or maintains dependency listings in `pyproject.toml`. But both Poetry and Uv go further beyond that as well, aiming to help with the rest of the development workflow (such as building your package for upload to PyPI).
If you like Pipx, you might be interested in some tips in my recent blog post (https://zahlman.github.io/posts/2025/01/07/python-packaging-...). In particular, if you do need to install libraries, you can expose Pipx's internal copy of Pip for arbitrary use instead of just for updating the venvs that Pipx created.
Sure, venv doesn't manage Python versions, but it's not that difficult to install the version you need system-wide and point your env to it. Multiple Python versions can coexist in your system without overriding the default one. On Ubuntu, the deadsnakes PPA is pretty useful if you need an old Python version that's not in the official repos.
In the rare case where you need better isolation (like if you have one fussy package that depends on specific system libs, looking at you tensorflow), Docker containers are the next best option.
I appreciate how thorough this was.
[1] https://github.com/astral-sh/uv/blob/main/crates/uv-python/d... (search for pgo)
It was used by rye before rye and uv sort of merged and is used by pipx and hatch and mise (and bazel rules_python) https://x.com/charliermarsh/status/1864042688279908459
My understanding is that the problem is that psf doesnt publish portable python binaries (I dont think they even publish any binaries for linux). Luckily theres some work being done on a pep for similar functionality from an official source but that will likely take several years. Gregory has praised the attempt and made suggestions based on his experience. https://discuss.python.org/t/pep-711-pybi-a-standard-format-...
Apparently he had less spare time for open source and since astral had been helping with a lot of the maitinence work on the project he happily transfered over ownership to themin December
https://gregoryszorc.com/blog/2024/12/03/transferring-python... https://astral.sh/blog/python-build-standalone
Because of this ^
UV seems like it provides a lot of that convenience for python.
Second best are OpenSSL dependencies. I sincerely hope I won’t have to deal with that again.
Having said that, I’m going to give uv a shot because I hear so many good things about it.
[1] https://pip.pypa.io/en/stable/user_guide/#constraints-files
But having tasted the sweet nectar of uv goodness, I’m onboard the bandwagon.
I'm only aware of examples where it's the fault of the packages - i.e. they specify dependency version ranges that don't actually work for them (or stop working for them when a new version of the dependency is released). No tool can do anything about that on the user's end.
> Or if you run on a different OS and the dependencies are different there (because of env markers), your requirements file won't capture that. There are a lot of gotchas that pip can't fix.
The requirements.txt format is literally just command-line arguments to Pip, which means you can in fact specific the env markers you need there. They're part of the https://peps.python.org/pep-0508/ syntax which you can use on the Pip command line. Demo:
$ pip install 'numpy;python_version<="2.7"'
Ignoring numpy: markers 'python_version <= "2.7"' don't match your environment
There are a lot of serious problems with Pip - I just don't think these are among them.
Reproducibility is important in many contexts, especially CI, which is why in Node.js world you literally do "npm ci" that installs exact versions for you.
If you haven't found it necessary, it's because you haven't run into situations where not doing this causes trouble, like a lot of trouble.
Lockfiles are an anti-pattern if you're developing a library rather than an application, because you can't push your transitive requirements onto the users of your library.
That’s very much not a lock file, even if it is possible to abuse it as such.
That's not to say they aren't downsides. https://gregoryszorc.com/docs/python-build-standalone/main/q... documents them. As an example I had to add the https://pypi.org/project/gnureadline/ package to a work project that had its own auto completing shell because by default the builds replaces the gnu readline package with lived it/edit line and they're far from a drop in replacement.
This is the part I don't understand. Why should it be the same tool? What advantage does that give over having separate tools?
Also as silly as it is I actually have a hard time remembering the venv syntax each time.
uv run after a checkout with a lock file and a .python-version file downloads the right python version creates a venv and then installs the packages. No more needing throwaway venvs to get a clean pip freeze for requirements. And I don't want to compile python, even with something helping me compile and keep track of compiles like pyenv a lot can go wrong.
And that assumes an individualindividual project run by someone who understands python packaging. UV run possibly in a wrapper script will do those things for my team who doesn't get packaging as well as I do. Just check in changes and next time they UV run it updates stuff for them
>Because its easier. Because it fits together nicer and more consistently. Also because UV is well written and written in rust so all the parts are fast. You can recreate a venv from scratch for every run.
This is the biggest thing I try to push back on whenever uv comes up. There is good evidence that "written in Rust" has quite little to do with the performance, at least when it comes to creating a venv.
On my 10-year-old machine, creating a venv directly with the standard library venv module takes about 0.05 seconds. What takes 3.2 more seconds on top of that is bootstrapping Pip into it.
Which is strange, in that using Pip to install Pip into an empty venv only takes about 1.7 seconds.
Which is still strange, in that using Pip's internal package-installation logic (which one of the devs factored out as a separate project) to unpack and copy the files to the right places, make the script wrappers etc. takes only about 0.2 seconds, and pre-compiling the Python code to .pyc with the standard library `compileall` module takes only about 0.9 seconds more.
The bottleneck for `compileall`, as far as I can tell, is still the actual bytecode compilation - which is implemented in C. I don't know if uv implemented its own bytecode compilation or just skips it, but it's not going to beat that.
Of course, well thought-out caching would mean it can just copy the .pyc files (or hard-link etc.) from cache when repeatedly using a package in multiple environments.
Also, I don't have a c compiler installed.
(By my understanding, `pyenv install` will expect to be able to run a compiler to build a downloaded Python source tarball. Uv uses prebuilt versions from https://github.com/astral-sh/python-build-standalone ; there is work being done in the Python community on a standard for packaging such builds, similarly to wheels, so that you can just use that instead of compiling it yourself. But Python comes out of an old culture where users expect to do that sort of thing.)
Sure, I guess if you have one Pip will "support" reading it.
So it’s a “lockfile” in the strictest, most useless definition: only works on the exact same Python version, on my machine, assuming no dependencies have published new packages.
>only works on the exact same Python version
It works on any Python version that all of the dependencies work on. But also it can be worked around with environment markers, if you really can support multiple Python versions but need a different set of dependencies for each.
In practical cases you don't need anything like a full (python-version x dependency) version matrix. For example, many projects want to use `tomllib` from the standard library in Python 3.11, but don't want to drop support for earlier Python because everything else still works fine with the same dependency packages for all currently supported Python versions. So they follow the steps in the tomli README (https://github.com/hukkin/tomli?tab=readme-ov-file#building-...).
>on my machine
(Elsewhere in the thread, people were trying to sell me on lock files for library development, to use specifically on my machine while releasing code that doesn't pin the dependencies.)
If my code works with a given version of a dependency on my machine, with the wheel pre-built for my machine, there is no good reason why my code wouldn't work on your machine with the same version of the dependency, with the analogous wheel - assuming it exists in the first place. It was built from the same codebase. (If not, you're stuck building from source, or may be completely out of luck. A lockfile can't fix that; you can't specify a build artifact that doesn't exist.)
This is also only relevant for projects that include non-Python code that requires a build step, of course.
>assuming no dependencies have published new packages.
PyPI doesn't allow you to replace the package for the same version. That's why there are packages up there with `.post0` etc. suffixes on their version numbers. But yes, there are users who require this sort of thing, which is why PEP 751 is in the works.
> It works on any Python version that all of the dependencies work on
No, it doesn’t. It’s not a lockfile: it’s a snapshot of the dependencies you have installed.
The dependencies you have installed depend on the Python version and your OS. The obvious case would be requiring a Linux-only dependency on… Linux, or a package only required on Python <=3.10 while you’re on 3.11.
> PyPI doesn't allow you to replace the package for the same version
Yes and no. You can continue to upload new wheels (or a sdist) long after a package version is initially released.
This is why libraries should not use lockfiles, they should be written to safely use as wide a range of dependencies' versions as possible.
It's the developers of an application who should use a lockfile to lock transitive dependencies.
The confusion might be that this used to be a problem with pip. It looks like this changed around 2020, but before then pip would happily install broken versions. Looking it up, this change of resolution happened in a minor release.
I think a lot of people probably have strong memories of all the nonsense that earlier pip versions resulted in, I know I do. I didn't realise this was a more solved problem now as not seeing an infrequent issue is hard to notice.
Nearly every other language solves this better than this. What your suggesting breaks down on large projects.
"Nearly every other language" determines the exact version of a library to use for you, when multiple versions would work, without you providing any input with which to make the decision?
If you mean "I have had a more pleasant UX with the equivalent tasks in several other programming languages", that's justifiable and common, but not at all the same.
>What your suggesting breaks down on large projects.
Pinned transitive dependencies are the only meaningful data in a lockfile, unless you have to explicitly protect against supply chain attacks (i.e. use a private package source and/or verify hashes).
> It works fine until you upgrade to a new version of python, in which case you install the package again.
Debian/Ubuntu doesn't want you to do either, and tell you you'll break your system if you force it (the override flag is literally named "--break-system-packages"). Hell, if you're doing it with `sudo`, they're probably right - messing with the default Python installation (such as trying to upgrade it) is the quickest way to brick your Debian/Ubuntu box.
Incredibly annoying when your large project happens to use pip to install both libraries for the Python part, and tools like CMake and Conan, meaning you can't just put it all in a venv.
No one with the most basic of sysad skills is “bricked” by having to uninstall a library. Again have not experienced a conflict in over 15 years.
Use the system package manager or buid yourself for tools like cmake.
So that's why pipx was required, or now, UV.
Or you install a second global tool that depends on an incompatible version of a library.
Why "should"? I think it's the other way around - Python culture has shied away from user-wide tools because it's known that they cause problems if you have more than a handful of them, and so e.g. Python profilers remain very underdeveloped.
I've spent most of the last two years making myself an expert on the topic of Python packaging. You can see this through the rest of the thread.
>No, it doesn’t. It’s not a lockfile: it’s a snapshot of the dependencies you have installed.
Yes, it does. It's a snapshot of the dependencies that you have installed. For each of those dependencies, there is some set of Python versions it supports. Collectively, the packages will work on the intersection of those sets of Python versions. (Because, for those Python versions, it will be possible to obtain working copies of each dependency at the specified version number.)
Which is what I said.
> The dependencies you have installed depend on the Python version and your OS. The obvious case would be requiring a Linux-only dependency on… Linux, or a package only required on Python <=3.10 while you’re on 3.11.
A huge amount of packages are pure Python and work on a wide range of Python versions and have no OS dependency. In general, packages may have such restrictions, but do not necessarily. I know this because I've seen my own code working on a wide range of Python versions without making any particular effort to ensure that. It's policy for many popular packages to ensure they support all Python versions currently supported by the core Python dev team.
Looking beyond pure Python - if I depend on `numpy==2.2.1` (the most recent version at time of writing), that supports Python 3.10 through 3.13. As long as my other dependencies (and the code itself) don't impose further restrictions, the package will install on any of those Python versions. If you install my project on a different operating system, you may get a different wheel for version 2.2.1 of NumPy (the one that's appropriate for your system), but the code will still work. Because I tested it with version 2.2.1 of NumPy on my machine, and version 2.2.1 of Numpy on your machine (compiled for your machine) provides the same interface to my Python code, with the same semantics.
I'm not providing you with the wheel, so it doesn't matter that the wheel I install wouldn't work for you. I'm providing you(r copy of Pip) with the package name and version number; Pip takes care of the rest.
>You can continue to upload new wheels (or a sdist) long after a package version is initially released.
Sure, but that doesn't harm compatibility. In fact, I would be doing it specifically to improve compatibility. It wouldn't change what Pip chooses for your system, unless it's a better match for your system than previously available.
It doesn’t handle environment markers nor is it reproducible. Given any non-trivial set of dependencies and/or more than 1 platform, it will lead to confusing issues.
Those confusing issues are the reason for lock files to exist, and the reason they are not just “the output of pip freeze”.
But you know this, given your two years of extensive expert study. Which I see very little evidence of.
I didn't say it was. I said that it solves the problems that many people mistakenly think they need a lockfile for.
(To be clear: did you notice that I am not the person who originally said "'pip freeze' generates a lockfile."?)
>It doesn’t handle environment markers nor is it reproducible.
You can write environment markers in it (of course you won't get them from `pip freeze`) and Pip will respect them. And there are plenty of cases where no environment markers are applicable anyway.
It's perfectly reproducible insofar as you get the exact specified version of every dependency, including transitive dependencies.
>Given any non-trivial set of dependencies and/or more than 1 platform, it will lead to confusing issues.
Given more than 1 platform, with differences that actually matter (i.e. not pure-Python dependencies), you cannot use a lockfile, unless you specify to build everything from source. Because otherwise a lockfile would specify wheels as exact files with their hashes that were pre-built for one platform and will not work on the others.
Anyway, feel free to show a minimal reproducible example of the confusion you have in mind.
So upgrade to new distro release, it has a new Python. Then pip install --user your user tools, twine, httpie, ruff, etc. Takes a few moments, perhaps once a year.
I do the same on Fedora, which I've been using more lately.
If you do have platform-specific dependency requirements, then you can't actually use the same versions of libraries, by definition. But you can e.g. specify those requirements abstractly, see what the installer produces on your platform, and produce a concrete requirement-set for others on platforms sufficiently similar to yours.
(I don't know offhand if any build backends out there will translate abstract dependencies from an sdist into concrete ones in a platform-specific wheel. Might be a nice feature for application devs.)
Of course there are people and organizations that have use cases for "real" lockfiles that list provenance and file hashes, and record metadata about the dependency graph, or whatever. But that's about more than just keeping a team in sync.
Lockfiles aren't helpful to me here because the entire point is not to be dependent upon specific versions. I actively want to go through the cycle of updating my development environment on a whim, finding that everything breaks, doing the research etc. - because that's how I find out what my version requirements actually are, so that I can properly record them in my own project metadata. And if it turns out that my requirements are narrow, that's a cue to rethink how I use the dependency, so that I can broaden them.
If I had a working environment and didn't want to risk breaking it right at the moment, I could just not upgrade it.
If my requirements were complex enough to motivate explicitly testing against a matrix of dependency versions, using one of those "other tools", I'd do that instead. But neither way do I see any real gain, as a library developer, from a lock file.
The point of a lockfile is to only upgrade when you want to upgrade. I hope you understand that.
What is more likely:
1. Using a lockfile means you cannot use wheels and have to build from source
2. You don’t know what you’re talking about
(When deciding, keep in mind that every single lockfile consuming and producing tool works fine with wheels)
All the other project-specific things go in venvs where they belong.
This is all a non-issue despite constant "end of the world" folks who never learned sysadmin and are terrified of an error.
If a libraries conflict, uninstall them, and put them in a venv. Why do all the work up front? I haven't had to do that in so long I forget how long it was. Early this century.
It's not a non-issue. Yes it's not a showstopper, but it's a niggling drag on productivity. As someone who's used to the JVM but currently having to work in Python, everything to do with package management is just harder and more awkward than it needs to be (and every so often you just get stuck and have to rebuild a venv or what have you) and the quality of tooling is significantly worse as a result. And uv looks like the first of the zillions of Python package management tools to actually do the obvious correct thing and not just keep shooting yourself in the foot.
Still I’m looking forward to uv because I’ve lost faith in pypa. They break things on purpose and then say they have no resources to fix it. Well they had the resources to break it.
But this doesn’t have much to do with installing tools into ~/.local.