While `uv` works amazingly well I think a lot of people don't realize that installing packages through conda (or let's say the conda-forge ecosystem) has technical advantages compared to wheels/pypi.

When you install the numpy wheel through `uv` you are likely installing a pre-compiled binary that bundles openblas inside of it. When you install numpy through conda-forge, it dynamically links against a dummy blas package that can be substituted for mkl, openblas, accelerate, whatever you prefer on your system. It's a much better solution to be able to rely on a separate package rather than having to bundle every dependency.

Then lets say you install scipy. Scipy also has to bundle openblas in their wheel, and now you have two copies of openblas sitting around. They don't conflict, but this quickly becomes an odd thing to have to do.

The problem conda solved that nothing had solved before was installing binary dependencies on MS Windows.

Before conda, getting a usable scipy install up and running on MS Windows was a harrowing experience. And having two independent installations was basically impossible. The real hard work that went into conda was reverse engineering all the nooks and crannies of the DLL loading heuristics, to allow it to ensure that you loaded what you intended.

If you are working on macOS and deploying to some *nix in the cloud, you are unlikely to find any value in this. But in ten years as lead on a large tool that was deployed to personal (Windows) laptops in a corporate environment, I did not find anything that beat conda.

As someone with admittedly no formal CS education, I've been using conda for all of my grad school and never managed to break it.

I create a virtual environment for every project. I install almost all packages with pip, except for any binaries or CUDA related things from conda. I always exported the conda yaml file and managed to reproduce the code/environment including the Python version. I've seen a lot of posts over time praising poetry and other tools and complaining about conda but I could never relate to any of them.

Am i doing something wrong? Or something right?

Can somebody please eli5 why it is so unanimously accepted that Python's package management is terrible? For personal projects venv + requirements.txt has never caused problems for me. For work projects we use poetry because of an assumption that we would need something better but I remain unconvinced (nothing was causing a problem for that decision to be made).

Really the issue is python itself, it shouldn't be treating it's installs and packages as something that's linked and intertwined to the base operating system.

People like to complain about node packages but never seen people have the trouble with them that they have with python.

I hope you read it while it was available, because the domain has expired

    Domain Name: pyherald.com
    Registry Domain ID: 2663190918_DOMAIN_COM-VRSN
    Registrar WHOIS Server: whois.namesilo.com
    Registrar URL: https://www.namesilo.com/
    Updated Date: 2024-12-21T07:00:00Z
    Creation Date: 2021-12-21T07:00:00Z
    Registrar Registration Expiration Date: 2024-12-21T07:00:00Z

https://web.archive.org/web/20241220211119/https://pyherald.... is the most recent snap

Conda used to be a life saver when years and years ago, compiled extensions were hard to install because you had to compile them yourself.

Nowadays, thanks to wheels being numerous and robust, the appeal of anaconda is disappearing for most users except for some exotic mixes.

conda itself now causes more trouble than it solves as it's slow, and lives in its own incompatible world.

But anaconda solves a different problem now that nobody else solves, and that's managing Python for big corporation. This is worth a lot of money to big structures that need to control packages origin, permissions, updates, and so on, at scale.

So it thrives there.

Nothing in the "article" seems to support the title. A lot of it is just about Python packaging in general, or about problems when mixing conda- and pip-installed packages.

In my experience conda is enormously superior to the standard Python packaging tools.

I think Pixi mostly solves the main issues of conda by forcing users to have project-specific environments. It also solves environments incredibly fast, so it’s really quick to create new projects/environments. https://pixi.sh/

Conda is the only package manager I've used on Ubuntu that intermittently and inexplicably gets stuck when installing or uninstalling. It will sometimes resolve itself if left alone for hours, but often won't.

I avoid it as much as possible.

I feel like a major selling point of Nix is "solving the Python dependency-hell problem" (as well as that of pretty much every other stack)

I've seen so many issues with different Python venvs from different Python project directories stepping on each others' dependencies somehow (probably because there are some global ones) that the fact that I can now just stick a basic and barely-modified-per-project Python flake.nix file in each one and be always guaranteed to have the entirely of the same dependencies available when I run it 6 months later is a win.

This seems to be an aggregation of some posts on python-list. Basically, extra-random opinions.

I'll offer mine: I won't say that Python packaging is generally excellent, but it's gotten much better over the years. The pyproject.toml is a godsend, there's the venv module built-in to Python, pip will by default no longer install package outside of a venv. Dependency groups are being added, meaning that the requirements.txt files can also be specified in the project.toml. Documentation is pretty good, especially if you avoid blog posts from 5+ years ago.

I tried Conda a number of time over the years, regretted it every time.

These days, when I absolutely have to use it because some obscure piece of software can't run unless Conda, I install it in a VM so that:

    - I protect my working system from the damage of installing Conda on it

    - I can throw the whole garbage fire away without long term brain damage to my system once I'm done

All thoughts and opinions about conda aside, it’s the only sane way (on several platforms) to install gdalbins + gdal-python-bindings.

I don’t mind conda. It has a lot of caveats and weird quirks

I prefer uv now. https://github.com/astral-sh/uv

People here focus on Python, but to me, a bioinformatician, conda is much more, it provides 99.99% of the tools I need. Like bwa, samtools, rsem, salmon, fastqc, R. And many, many obscure tools.

Besides the horrendous formatting, some stuff in this article seem incorrect or irrelevant. Like, is this even possible?

> A single Anaconda distribution may have multiple NumPy versions installed at the same time, although only one will be available to the Python process (note that this means that sub-processes created in this Python process won’t necessarily have the same version of NumPy!).

I’m pretty sure there’s not, but maybe there is some insane way to cause subprocesses to do this. Besides that, under the authors definition, different Python virtualenvs also install multiple copies of libraries in the same way conda does.

The comments about Jupyter also seem very confused. It’s hard to make heads or tails of exactly what the author is saying. There might be some misunderstandings of how Jupyter kernels select environments.

> Final warning: no matter how ridiculous this is: the current directory in Python is added to the module lookup path, and it precedes every other lookup location. If, accidentally, you placed a numpy.py in the current directory of your Python process – that is going to be the numpy module you import.

This has nothing to do with conda.

The domain got parked.

I think Python had a pretty good idea in standardizing a packaging protocol and then allowing competing implementations, but I would have preferred a single "blessed" solution. More than one package management option in an ecosystem always adds some kind of "can't get there from here" friction and an additional maintenance burden on package maintainers.

poetry has been working well enough for me as of late, but it'd be nice if I didn't have to pick.

I honestly have no idea why anyone still uses Conda, it's a right pain in the ass. Python package management in general is a nightmare, but whenever I run up a project that uses Conda I immediately disregard it and use uv / pyenv.

Conda: a package manager disaster that became paid license required for companies over 200 employees. It worked 5 years ago, we can no longer legally use it

The Five Demons of Python Packaging That Fuel Our Persistent Nightmare: https://youtu.be/qA7NVwmx3gw?si=QbchrYvCEp8aazvL

I strongly suspect that there is about to be a spike in Python packaging discussion over and above the high ambient baseline.

uv is here to kick ass and chew bubblegum. And it’s all out of gum.

Are most people having problems with python packages using Windows? It's been mentioned a couple of times in this thread, but not that often.

conda was for scientific python, but had to solve for everything below python to make that work. There was no generic binary solution before python for multiple architectures and operating systems.

My experience with conda is that its fine if you're the original author of whatever you're using it for and never share it with anyone else. But as a professional I usually have to pull in someone else's work and make it function on a completely different machine/environment. I've only had negative experiences with conda for that reason. IME the hard job of package management is not getting software to work in one location, but allowing that software to be moved somewhere else and used in the same way. Poetry solves that problem, conda doesn't.

Poetry isn't perfect, but it's working in an imperfect universe and at least gets the basics (lockfiles) correct to where packages can be semi-reproducible.

There's another rant to be had at the very existence of venvs as part of the solution, but that's neither poetry or anaconda's fault.

You are doing something right, author does some pretty unusual things:

- Setup custom kernels in Jupyter Notebook

- Hardlink the environments, then install same packages via pip in one and conda in others

- install conda inside conda (!!!) and enter nested environment

- Use tox within conda

I believe as long as you treat the environments as "cattle" (if it goes bad, remove it and re-create from yaml file), you should not have any problems. It's clearly not the case of for the post's author though.

As someone with a formal computer science, half of my friends who work in other sciences have asked me to help them fix their broken conda environments

This is exactly the kind of thing that causes python package nightmares. Pip is barely aware of packages it's installed itself, let alone packages from other package managers and especially other package repositories. Mixing conda and pip is 100% doing it wrong (not that there's an easy way to do it right, but stick to one or the other, I would generally recommend just using pip, the reasons for conda's existance are mostly irrelevant now)

I had the same experience. But you should try pixi, which is to conda what uv is to pip.

God forbid you should require conda-forge and more than three packages lest the dependency resolver take longer than the heat death of the planet to complete.

Same but I try to use conda to install everything first, and only use pip as a last resort. If pip only installs the package and no dependency it's fine

i think you got lucky and fell into best practices on your first go

> except for any binaries or CUDA related things from conda

doing the default thing with cuda related python packages used to often result in "fuck it, reinstall linux". admittedly, i dont know how it is now. i have one machine that runs python with a gpu and it runs only one python program.

1. You need to run export manual while other tools you mentioned would create it automatically (the lock file) 2. Distinguishes between direct dependencies (packages you added yourself) and indirect dependencies (packages of the packages)

In your requirements.txt, do you pin the concrete versions or leave some leeway?

If you aren't precise, you're gonna get different versions of your dependencies on different machines. Oops.

Pinning concrete versions is of course better, but then there isn't a clear and easy way to upgrade all dependencies and check whether ci still passes.

> For personal projects venv + requirements.txt has never caused problems for me.

That means you don't use Windows.

What is great. Keep not using it. But most people will have a different experience.

For using packages, venv + requirements.txt works, but is a bit clunky and confusing. Virtual environments are very easy to break by moving them or by updating your OS (and getting a new Python with it). Poetry is one alternative, but there are far too many options and choices to make. For building packages, there are similarly many competing options with different qualities and issues.

i think there might be merit to gdiamos's point that python is a popular language with a large number of users, and this might mean that python package management isn't unusually bad, but more users implies more complaints.

i think there was a significant step change improvement in python packaging around 2012, when the wheel format was introduced, which standardised distributing prebuilt platform-specific binary packages. for packages with gnarly native library dependencies / build toolchains (e.g. typical C/fortran numeric or scientific library wrapped in a layer of python bindings), once someone sets up a build server to bake wheels for target platforms, it becomes very easy to pip install them without dragging in that project's native build-from-source toolchain.

venv + pip (+ perhaps maintaining a stack of pre-built wheels for your target platform, for a commercial project where you want to be able to reproduce builds) gets most of the job done, and those ingredients have been in place for over 10 years.

around the time wheel was introduced, i was working at a company that shipped desktop software to windows machines, we used python for some of the application components. between venv + pip + wheels, it was OK.

where there were rough edges were things like: we have a dep on python wrapper library pywhatever, which requires a native library libwhatever.dll built from the c++ whatever project to be installed -- but libwhatever.dll has nothing to do with python, maybe its maintainers kindly provide an msi installer, so if you install it into a machine, it gets installed into the windows system folder, so venv isn't able to manage it & offer isolation if you need to install multiple versions for different projects / product lines, as venv only manages python packages, not arbitrary library dependencies from other ecosystems

but it's a bit much blame python for such difficulties: if you have a python library that has a native dependency on something that isnt a python package, you need to do something else to manage that dep. that's life. if you're trying to do it on windows, which doesn't have an O/S level package manager.. well, that's life.

Try building a package and you will get hundreds of little paper cuts. Need a different index for some packages? It will work with a cli "pip install -from-index", but pip will not let you add an index in a requirement.txt for... security reasons. That means, good luck trying to "enforce" the CUDA version of pytorch without using third party tooling. So you either hard code a direct link (breaks platform portability), as that will work, or give up trying to make your project installable with "Pip install " Or "python build". Remember, pytorch basically has no CUDA builds anymore in its pypi index and no way to get CUDA torch from there (but I think this might have changed recently? )

Oh, and if some package you are using has a bug or something that requires you to vendor it in your repo, well then good luck because again, PEP 508 does not support installing another package from a relative link. You either need to put all the code inside the same package, vendored dependency included, and do some weird stuff to make sure that the module you wanted to vendor is used first, or... you just have to use the broken package, again for some sort of security reasons apparently.

Again, all of that might even work when using pip from the cli, but good luck trying to make a requirements.txt or define dependencies in a standard way that is even slightly outside of a certain workflow.

venv + requirements.txt has worked for every single python project I made for the last 2 years (I'm new to python). Only issue I had was when using a newish python version and not having a specific library released yet for this new version, but downgrading python solved this.

Being new to the ecosystem I have no clue why people would use Conda and why it matters. I tried it, but was left bewildered, not understanding the benefits.

I can sort of the the argument, if you really really need to lock down your dependencies to very specific version, which I don't recommend you do.

For development I use venv and pip, sometimes pyenv if I need a specific Python version. For production, I install Python packages with apt. The operating system can deal with upgrading minor library versions.

I really hate most other package managers, they are all to confusing and to hard to use. You need to remember to pull in library update, rebuild and release. Poetry sucks too, it's way to complicated to use.

The technical arguments against Python packages managers are completely valid, but when people bring up Maven, NPM or even Go as role models I check out. The ergonomics of those tools are worse than venv and pip. I also think that's why we put up with pip and venv, they are so much easier to use than the alternative (maybe excluding uv). If a project uses Poetry, I just know that I'm going to be spending half a day upgrading dependencies, because someone locked them down a year ago and there's now 15 security holes that needs to be plugged.

No, what Python needs is to pull in requests and a web framework into the standard library and then we can start build 50% of our projects without any dependencies at all. They could pull in Django, it only has two or three dependencies anyway.

It's popular enough that it causes pain for a lot of people.

Coming from C++, IMO, it is vastly better.

What do you do though if you want to import code written in C++? Especially complex, dependency-heavy like CUDA/ML stuff?

You can just give up and say that "The proper way to do this is to use the Nvidia CUDA toolkit to write your cuda app in C++ and then invoke it as a separate process from node" [0]. That apparently works for node, but Python wants much more.

If you actually want to use high-performance native code in your slow compiled language, then no solution is going to be very good, that's because the problem is inherently hard.

You can rely on host OS as much as possible - if OS is known, provide binaries; if it's unknown, provide source code and hope user has C/C++/Rust/Fortran compilers to build it. That's what uv, pip, etc.. do.

You can create your own parallel OS, bringing your own copy of every math libray, as well as CUDA even if there are perfectly good versions installed on the system - that's what conda/minconda does.

You can implement as much as possible in your own language, so there is much less need to use "high-performance native language" - that's what Rust and Go do. Sadly, that's not an option for Python.

[0] https://stackoverflow.com/questions/20875456/how-can-i-use-c...

This is spot on. Running some python projects on nixos is a nightmare because of this model. Especially if it’s ML related.

In Python you need to deliberately mess with the system Python by running your package installer under sudo. That’s not something you do accidentally.

When dealing with the system Python you should always use the system package manager. That extends to Macs for both brew and MacPorts.

It's because of the SAT solver for dependencies. Unlike Pip, it keeps track of every package you installed and goes out of its way to avoid installing incompatible packages.

Why go through all this trouble? Because originally it was meant to be a basic "scientific Python" distribution, and needed to be strict around what's installed for reproducibility reasons.

It's IMO overkill for most users, and I suspect most scientific users don't care either - most of the time I see grads and researchers just say "fuck it" and use Pip whenever Conda refuses to get done in a timely fashion.

And the ones who do care about reproducibility are using R anyway, since there's a perception those libraries are "more correct" (read: more faithful to the original publication) than Pythonland. And TBH I can't blame them when the poster child of it is Sklearn's RandomForestRegressor not even being correctly named - it's bagged trees under the default settings, and you don't get any indication of this unless you look at that specific kwarg in the docs.

Personally, I use Conda not for reproducibility, but so all of my projects have independent environments without having to mess with containers

The situation has since changed as the solver is rewritten. (Upstreamed from the work done in mamba.) I encourage you to try again.

Try it again. It now uses libmamba for solving dependencies

I wish you luck with tracking down versions of software used when you're writing papers... especially if you're using multiple conda environments. This is pretty much the example used in the article -- version mismatches.

But, I think this illustrates the problem very well.

Conda isn't just used for Python. It's used for general tools and libraries that Python scripts depend on. They could be C/C++ that needs to be compiled. It could be a Cython library. It could be...

When you're trying to be a package manager that operates on-top of the operating system's package manager, you're always going to have issues. And that is why Conda is such a mess, it's trying to do too much. Installation issues are one of the reason why I stopped writing so many projects in Python. For now, I'm only doing smaller scripts in Python. Anything larger than a module gets written in something else.

People here have mentioned Rust as an example of a language with a solid dependency toolchain. I've used more Go, which similarly has had dependency management tooling from the begining. By and large, these languages aren't trying to bring in C libraries that need to be compiled and linked into Python accessible code (it's probably possible, but not the main use-case).

For Python code though, when I do need to import a package, I always start with a fresh venv virtual environment, install whatever libraries are needed in that venv, and then always run the python from that absolute path (ex: `venv/bin/python3 script.py`). This has solved 99% of my dependency issues. If you can separate yourself from the system python as much as possible, you're 90% of the way there.

Side rant: Which, is why I think there is a problem with Python to begin with -- *nix OSes all include a system level Python install. Dependencies only become a problem when you're installing libraries in a global path. If you can have separate dependency trees for individual projects, you're largely safe. It's not very storage efficient, but that's a different issue.

And then somebody tries to install mamba via conda and that house of cards reveals itself.