Package reproducibility in Python notebooks using uv isolated environments

Stuff like this is so convenient and intuitive that it makes you wonder why we didn't start here in the first place!

One thing I wonder is how many data scientists will use this feature given that it is not enabled by default (which is understandable, would be messy for every notebook to have a venv), and only via command-line arguments.

I guess this is easily remedied by helpful beginner tooltip UX ("This notebook has several dependencies. Would you like to build it in a sandbox?").

mscolnick 2 years ago | |

We would like to make this configurable via a setting so you don't need to pass it through the CLI every time. And I agree, maybe a tooltip/notification to nudge them in that direction.

We would also like to auto-detect a pyproject.toml, and use that when desired.

Onawa 2 years ago |

Seems like an admirable goal and the utilization of UV is great! I'm wondering if there's any possibility for integrating this sort of package management into a project like Quarto which already has a strong ecosystem of plugins and outputs from their scientific notebooks. Quarto notebooks are also markdown based and can utilize Jupyter as a back end for Notebook rendering?

WCSTombs 2 years ago |

Nice!

I haven't tried this yet, but I love that the functionality of Jupytext is also incorporated, so I guess you get to reproduce the whole end product and all dependencies just from a plain-text script that you can track in Git.

najork 2 years ago |

How've you found `uv` as a package manager? I've generally been a fan of Astral's tools in the Python ecosystem and I'm considering making the switch from `poetry`.

baggiponte 2 years ago | |

it's an absolute beast.

They need to add a couple minor painful paths that people usually use package managers with (private indexes, binding packages to third party indexes...) but if you don't need fancy corner cases, it's the best thing ever.

odie5533 2 years ago | | |

For any commercial company, those are not corner cases!

Onavo 2 years ago | | |

Also any machine learning use cases! There’s a big caveat with uv where they merge your Pipfile and pip installations. For example if you do uv add flask (this is added to pipfile) but then you need to add PyTorch with custom indexes (because most ML libraries distribute driver specific versions) and you do uv pip install PyTorch —-index-url what you end up with is a pipfile.lock that merges the two. uv won’t automatically add your dependency to the Pipfile.

https://pytorch.org/get-started/locally/

baggiponte 2 years ago | | |

There's an issue about pytorch (I think you might already have seen it since you encountered this). This is still due to lack of support for pinning a package to a third party index (see https://github.com/astral-sh/uv/issues/171).

I am not sure I totally understand what you mean: yes, the lockfile might be messed up, but is the environment working? I'm a bit blessed because I don't have CUDA on my machine so I save myself a lot of hassle.

I am trying to submit an improvement to the docs about this. https://github.com/astral-sh/uv/pull/6523 I'd love to hear your feedback.

Onavo 2 years ago | | |

I think the docs should be more clear that the uv pip command is an escape hatch, and the moment you use it you basically lose the ability to have the Pipfile as a source of truth. Most devs won't be looking too closely at the lockfile unless it's to debug issues, they usually only refer to the Pipfile. It would also be nice to have a pip to uv/rye command converter like those curl command generators.

baggiponte 2 years ago | | |

You are correct - I should've been more specific. Binding a package(s) to a third party index is currently unsupported (see issue: https://github.com/astral-sh/uv/issues/171) as it is something that pip itself does not support. PDM/Poetry implemented this on their own - though this it's an important feature and as far as I can tell it's coming.

Onawa 2 years ago | |

It is absolutely blazingly fast and surprisingly so. Few small pain points but the speed increase makes it absolutely worth it for many cases.