A plea not to use uv-build(blogs.gentoo.org) |
A plea not to use uv-build(blogs.gentoo.org) |
(I am one of uv’s maintainers.)
The nice thing about Python packaging is that it’s standardized: if uv’s build backend doesn’t fit into the hole you want, then you can use any number of other backends that do an admirable job of conforming to the packaging standards.
With that said, I don’t understand why Gentoo needs to rebuild every single release of uv-build. It doesn’t change as much as uv itself, and it’s intentionally compatible across uv versions. Why not set a policy of only updating once weekly, or even less frequently, with exceptions for important breaking changes or bug fixes?
(For context: the only reason uv-build releases as often as it does is because it’s easier to keep its internals in lockstep with the rest of uv. But you can do a normal version bound in your Python build backend metadata and uv will happily use whatever version of uv-build is already present.)
But it isn't. Once upon a time, it was: you used distutils. But of course distutils had a lot of problems. Instead of adopting a single standardized solution for those problems, the Python developers punted and let third parties do it. So now we do not have a single standardized solution for building Python packages; we have a bunch of third party solutions, none of which is in the Python standard library.
Yes, all those third party solutions have to be PEP 517 compatible, but that's not standardization on a tool; it's just standardization on an interface between the backend and the frontend. Which is nice, but doesn't solve the problem under discussion here.
> if uv’s build backend doesn’t fit into the hole you want, then you can use any number of other backends that do an admirable job of conforming to the packaging standards.
As the project developer, sure. But as a Linux distro building your python package, Gentoo has to use the build backend you specify. It can't say, well, this particular package is just pure python, so really all we need is a minimal tool to build a pure python none-any wheel. It has to use the backend you specified.
> I don’t understand why Gentoo needs to rebuild every single release of uv-build.
Because if you don't pin a version of uv-build in your project, that means "use the latest version available". Gentoo can't second guess you--even if it were able to verify somehow that the new version of uv-build that just got released will end up producing an identical wheel for your project as the previous version, without actually using the new version of uv-build.
> you can do a normal version bound in your Python build backend metadata
Sure, but how many users of uv-build actually do that? Seems like the answer to that is "not enough to matter".
At the end of the day, it’s hard for me to even pinpoint what is problematic for Gentoo here: I can understand that they need to support multiple build backends as a reality of Python packaging and that’s frustrating, but per below it seems extremely unlikely that they actually need to build every single patch version of uv. Nothing about uv’s defaults or the packaging template would require that, and users would have to go out of their way to configure their build backend in that way.
(And note: this is not unique to uv! You can grief anybody who does an sdist build by pinning to a random ancient version of setuptools or any other build backend, including other ones that use native code. This is why uv, like everybody else, encourages semver ranges for build backends.)
Admittedly rustification like that is a problem not only for gentoo, I have commercially deployed systems where one of the most annoyings elements of updating is presence of rust in python deps...
(This is the default when you initialize a project with uv, so I think someone would need to go out of their way to configure their build backend to request a specific patch version of uv-build.)
Has anybody done the analysis to determine if this is something that can, in practice, be relied on?
(I think so much of the confusion around Python packaging comes down the layer of abstraction/standardization not being the one that users intuitively expect. This is unfortunate and is made worse by lackluster user-facing documentation. But there are standards that guarantee interoperability, and they’re pretty close to Pareto-optimal in terms of making all involved parties happy.)
Happy user for over 23 years now!
https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-python/uv...
Looks like it's being built with fat LTO by default (and gentoo patched to thin LTO). Which is fine if you want to spend CI cycles once to ship executables to thousands of users. If you compile from source and run the tool maybe once then they should consider going further and run with maybe opt-level=1, lto=off; halfway between release and debug. This is how rustc itself builds some internal buildtools that get recompiled frequently relative to how often they run.
If only there were ways to skip this waste...
btw, I know of a guy who using extensive amounts of tokens to make his fork of mrustc usable via "claude, make this rustc compile 99% of crates.io", I think this also can yield some results.
> The other reason is performance and that argument makes even less sense. Even if it mattered, we’re realistically talking about a difference in the ballpark of 100 ms, for an action executed rarely; it is equally possible that any gain will be outweighed by the time needed to fetch the 1.5 MiB wheel for uv-build.
Modern CI architectures are building Docker stacks from scratch and using a readthrough cache of wheels. Downloading may still be the dominant factor, but the second-dominant factor does start to add up, especially when the downloading gets cached at various layers.
(At the end of the day, this is exactly what support tiers are for. CPython, at least to my eyes, has been very transparent about not guaranteeing support for every platform that has a C compiler.)
People who don't care about HPUX will never care about it but they would if e.g. we decided that only windows and Ubuntu were supported ... except for the windows and ubuntu python users who wouldn't care.
It's I'm-all-right Jack until it's your use case that gets bulleted.
> In the early days, there was much promotion of Gentoo as being “faster”, because you could compile everything based on your particular processor, etc. And you could increase the optimization level for a boost. Their web site still touts this as a reason to use Gentoo.
> In reality, the performance is more or less the same as on any other distribution. The folks who stick to Gentoo tend not to care about performance as much. Unfortunately, this perception of Gentoo remains, and I wish they would remove the verbiage from their site.
Linux wouldn't exist if people submitted to that argument.
There's not much of an argument for adopting obscure niche practices for their own sake
The bigger problem, to me, is that it’s not really clear why this is actually a problem for Gentoo. I am glad they perform user-facing source builds, however, because they see things that other distributors don’t.
Now though? It's way easier to have a herd of unicorn docker containers running that each handle one weird thing in isolation.
By just adding
preempt=full
sudo rpm-ostree kargs --editor
If every new patch version is still within the semver range specified in a project that uses uv-build, then yes, every Gentoo user will have to rebuild that new patch version of uv-build in order to rebuild that project. And since the build backend is a dependency of the project, pushing a new patch version will trigger a rebuild of the project.
I'm not a Gentoo user BTW but I do think building from source is attractive for the same reasons cargo (Isn't that build from source mostly?) and go do it.
This wouldn't necessarily solve Gentoo's problem. The only way to be sure to solve Gentoo's problem would be for every Python project that uses uv-build to specify one particular version. Then Gentoo could be sure that once that release of that project was built once, it would never change.
In other words, for a distro like Gentoo, the build backend is a dependency, and if it changes, that triggers a change in the build, even if the build backend change would not actually change the final wheel that got built.
It seems much easier to solve the problem of compatibility than the one of getting every open source maintainer to upgrade their build backend version (whether uv-build, or flit, or anything else) every Tuesday.
This doesn't make sense. Gentoo doesn't "pick" any build backend version. They build each Python project exactly the way the project owner tells them to, using whatever build backend is specified in pyproject.toml for that project. That's why it's such a burden for them (or more precisely their users, since each Gentoo user is having to do this) if build backend versions change so often and projects don't pin a specific version but just say "use the latest"--because then they have to use what the project owner says, and the project owner says, "Update to a new build backend every week because that's how often my chosen backend, uv-build, likes to push updates".
This coordination problem is already solved by PyPI. That's where Gentoo looks to find the build backend--just like everyone else that builds packages. The project owner had to be able to find whatever version they used there or they couldn't even build their project locally. So they should be able to depend on the fact that anyone else trying to build their project can also find that same version at the same place.
> getting every open source maintainer to upgrade their build backend version (whether uv-build, or flit, or anything else) every Tuesday.
You're missing my point. If I know that my project builds properly with build backend X version Y, and I pin that version in my pyproject.toml, I don't care how many newer versions you want to release. Unless I make a change to my project that merits updating my build backend to a newer version, I'm going to simply leave my project alone. I already know it will build fine with the version I have. Why should I push an update every Tuesday just because you decided to release a new version of the build backend, which I don't need anyway?
It's on Itanium so obviously we have to move away from that to Linux but the existence of open source tools like coreutils gives us time.
No such luck with python. We have to postpone some changes as a result.
As I say, there's some kind of effect where changes that are only negative on the margins are cheered until one finds oneself on one of those margins.
(But with all that said: there’s no ideological dispute here. If there was a reliable way to build and test for HP-UX, I assume a lot more people would declare support for it. The problem comes down to maintainers being asked to support platforms they ultimately have no ability to support, except in the weakest sense of “it compiles.”)
I'm still on Gentoo 22 years later.
Good timing! If memory serves, you missed the Big Libpng Upgrade that either got the concept of Package Slots into Portage or convinced everyone that they were a _really_ good feature to have and use where appropriate.
Based on what I can see after putting a minute or two into searching, that verbiage being referred to only appears on the About page. [0]
The distro's home page says
Welcome to Gentoo, a highly flexible, source-based Linux distribution.
And -speaking as a Gentoo user since 2002- neither the user community nor the maintainer community hold the opinion that Ultimate Maxxximum Performance is the distro's goal. Trouble reports that come from "ricers" pretty much always get told "Set your compiler flags to something sane (try '-O2 -pipe' or '-O2 -pipe -march=native') and rebuild. If you can still repro the problem, file a new report because this one is getting closed INVALID.".
And -while the old Gentoo Wiki burned down so I can't prove it- [4] the advice about GCC optimization in the wiki has always cautioned against using aggressive compiler flags and warned that doing so will -among other things- not substantially improve performance. The advice about reasonable compiler flags has pretty consistently been "Use -O2, -pipe, -march=$THE_CORRECT_SETTING_FOR_YOUR_CPU, and maybe -fomit-frame-pointer"... though the "-fomit-frame-pointer" advice has always come with a warning that doing that may make debugging either extremely difficult or impossible, and the "-march" advice the warning that this may make it impossible to run those binaries on another computer.
Anyway. There's only so much you can do to erase an -allegedly- pervasive myth. The fact that some folks have the belief that Gentoo Linux is all about Maxxximum Performance doesn't change the fact that nearly all of the people who use it and -almost certainly- all of the people who maintain it strongly disagree with that claim. The distro is and always has been about flexibility, user choice, and having fun. And it's a nice bonus if building for -say- -march=athlon-xp instead of -march=i586 happens to get you a bit of extra performance here and there.
[0] <https://www.gentoo.org/get-started/about/>
[1] <https://web.archive.org/web/20230101024843/https://www.gento...>
[2] <https://web.archive.org/web/20150406115846/https://www.gento...>
[3] That year's April 1 redesign was a bit cool, IMO.
[4] The oldest copy that I can easily find is from mid 2013 [5], so you'll just have to trust me that this has been the long-standing opinion of the overwhelming majority of users and maintainers. While you're there, take a minute or two to skim the prose and notice how it consistently warns against using aggressive compiler flags.
[5] <https://web.archive.org/web/20130829083155/https://wiki.gent...>
The Ubuntu users are only there because after they had told the rest of us how silly we were for using Linux they eventually had to use it themselves for work.
Now we're going to listen to that sort of stuff again without rolling our eyes? ....
I started using linux in 1994 with slackware. Nowadays I use ubuntu, I've spent too many hours of my life tweaking my linux installs.
So every Gentoo user that has a Python project that uses uv-build ends up having to rebuild uv-build, and then every Python project they have that uses it, every time uv-build pushes an update.
Other distros don't necessarily have quite the same problem because, if they include packages of Python projects, they don't normally track the build backend as a dependency. So if uv-build pushes an update, the distro would push an updated package for uv-build, but they wouldn't look at every Python project they package to see if it used uv-build as a build backend and rebuild it if it did. It's the "rebuild from source every time" aspect of Gentoo that forces that connection.
my take on linux distros being annoyed by their own policies is that perhaps they should change the policy. in this case, perhaps they should pretend uv-build updates once a month or something.
for example, they seem to be fine doing a quick sed on pyproject.toml (https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-python/pa...) or patching stuff for compat (https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-python/as...)
This is what i mean when it seems typical for a distro to monkey around with packages and/or their build system. These were just the first two packages i clicked on in dev-python. maybe others are "pure upstream", but there's definitely the capability here to just... not have the problem?
That's my understanding, yes.
I don't follow. This feels like Gentoo is mishandling build-tool dependencies.
Does Gentoo also recompile every C/C++ library in the dependency tree every time GCC pushes a minor version bump?
I would say it's doing what it believes is necessary for the Python ecosystem. The build backend is an explicit part of the specification of a Python project.
> Does Gentoo also recompile every C/C++ library in the dependency tree every time GCC pushes a minor version bump?
No, because C/C++ libraries don't have an explicit specification of the build tool to be used in their project, the way a Python project does.
I don't either. I used to run Gentoo for quite a few years in the past and build dependency updates would not trigger rebuilds of build-dependent packages.
Rebuilding the entire system with a new GCC was recommended by the documentation, but there was nothing in Portage forcing it. Neither would you rebuild the system if, e.g. make got an update.
This isn’t what’s happening: uv defaults to semver bounds, so a maintainer would have to go out of their way to introduce a troublesome backend constraint.
And note: there is no “use the latest” constraint in Python packaging. If you leave the build backend version unconstrained, the build frontend is allowed to resolve any version it sees fit, including the version it already has.
(I also know from first-hand experience that Gentoo and other distributions do not build Python packages as-is; they patch them in all kinds of ways. But that’s neither here nor there.)
But a new patch release will be within the same semver bound, so it amounts to the same thing.
> If you leave the build backend version unconstrained, the build frontend is allowed to resolve any version it sees fit, including the version it already has.
What do build frontends actually do, though?
Sorry, can you lay out the actual problem you’re envisioning for me? If uv-build goes from 0.13.1 to 0.13.2, then anybody with a semver bound of `uv-build>=0.12,<0.14` doesn’t need to update anything. They can keep using whatever version of the build backend they already have that satisfies that bound.
The main thing that would be troublesome is if a user pins their build backend to a specific range or version, e.g. `uv-build==0.13.1`. Then, upon bumping it, everyone who has already built 0.13.1 will need to invalidate and rebuild their build backend. But that’s why that isn’t the default.
> What do build frontends actually do, though?
uv will definitely use a compatible version. It won’t even download a copy of that that compatible version by default, since the uv binary itself is typically compatible and can invoke the build backend directly.
I’m not sure what pip and others do, but I would expect they do the same. It’s the obvious way to implement backend resolution.
If a compatible version is already present, yes, a frontend can just use it if it's within the semver range specified.
But in many cases, builds are done in a fresh virtual environment that has nothing installed. In such a case, the only option is to download the latest version that's in the semver range specified. That's how the "build" frontend works, for example, unless you specify a flag to keep it from creating a fresh virtual environment.