[dependencies]
foo_v1 = { package = "foo", version = "1" }
foo_v2 = { package = "foo", version = "2" }For example, the compiler error in this example:
note: perhaps two different versions of crate `smithay_client_toolkit` are being used?
(I've seen cases where that happens with C and C++ software, and things seem to compile and run... until everything explodes. Fun times.)
The author correctly contrasts Rust (and NPM's) behavior with that of Python/pip, where only one version per package name is allowed. The Python packaging ecosystem could in theory standardize a form of package name mangling wherein multiple versions could be imported simultaneously (akin to what's currently possible with multiple vendored versions), but that would likely be a significant undertaking given that a lot of applications probably - accidentally - break the indirect relationship and directly import their transitive dependencies.
(The more I work in Python, the more I think that Python's approach is actually a good one: preventing multiple versions of the same package prevents dependency graph spaghetti when every subdependency depends on a slightly different version, and provides a strong incentive to keep public API surfaces small and flexible. But I don't think that was the intention, more of an accidental perk of an otherwise informal approach to packaging.)
I've come to the opposite conclusion. I've "git cloned" several programs in both python and ruby (which has the same behaviour) only to discover that I can't actually install the project's dependencies. The larger your gemfile / requirements.txt is, the more likely this is to happen. All it takes is a couple packages in your tree to update their own dependencies out of sync with one another and you can run into this problem. A build that worked yesterday doesn't work today. Not because anyone made a mistake - but just because you got unlucky. Ugh.
Its a completely unnecessary landmine. Worse yet, new developers (or new teammembers) are very likely to run into this problem as it shows up when you're getting your dev environment setup.
This problem is entirely unnecessary. In (almost) every way, software should treat foo-1.x.x as a totally distinct package from foo-2.x.x. They're mutually incompatible anyway, and semantically the only thing they share is their name. There's no reason both packages can't be loaded into the package namespace at the same time. No reason but the mistakes of shortsighted package management systems.
RAM is cheap. My attention is expensive. Print a warning if you must, and I'll fix it when I feel like it.
(One of the ways I have seen this happen in this past is people attempting to use multiple requirements sources without synchronizing them or resolving them simultaneously. That's indeed a highway to pain city, and it's why modern Python packaging emphasizes either using a single standard metadata file like pyproject.toml or a fully locked environment specification like a frozen requirements file.)
For web dev and something like requests, it's just not as big of a deal to have a bunch of versions installed. You don't typically use/debug that kind of functionality in a way that would cause confusion. That said, it would be definitely be great sometimes to just be like "pip, I don't care, just make it work".
This! I have transitive conflicts almost every time I clone an exsting Python repo. It's one of the main reasons why Python == PITA^3 in my head.
Maybe it's the kind of Python repos I clone. Mostly ML/diffusion & computer graphics stuff.
As a Rustacean I had hoped that the Rhye + uv combo would 'fix' this but I now understand they won't.
The end result of this is that you end up with some random library in your stack (4 transitive layers deep because of course it is) holding back stuff like chokadir in a huge chunk of your dep tree for... no real good reason. So you now have several copies of a huge library.
Of course new major versions might break your usage! Minor versions might as well! Patch versions too sometimes! Upper bounds pre-emptively set help mainly in one thing, and that's reducing the number of people who would help "beta-test" new major versions because they don't care enough to pin their own dependencies.
The worst spaghetti comes from hard dependencies on minor versions and revisions.
I will die on the hill that you should only ever specify dependencies on “at least this major-minor (and optionally and rarely revision for a bugfix)” in whatever the syntax is for your preferred language. Excepting of course a known incompatibility with a specific version or range of versions, and/or developers who refuse to get on the semver bandwagon who should collectively be rounded up and yelled at.
In Rust, Cargo makes this super easy: “x.y.z” means “>= x.y.z, < (x+1).0.0”.
It’s fine to ship a generated lock file that locks everything to a fixed, known-good version of all your dependencies. But you should be able to trivially run an update that will bring everything to the latest minor and revision (and alert on newer major versions).
With the added special case of `0.x.y` meaning `>= 0.x.y, < 0.(x+1).0`, going beyond what semver specifies.
You would need:
A function v_tree_install(spec) which installs a versioned pypi package like “foo=3.2” and all its dependencies in its own tree, rather than in site-packages.
Another pair of functions v_import and v_from_import to wrap importlib with a name, version, and symbols. These functions know how to find the versioned package in its special tree and push that tree to sys.path before starting the import.
To cover the case for when the imported code has dynamic imports you could also wrap any callable code (functions, classes) with a wrapper that also does the sys.push/pop before/after each call.
You then replace third party imports in your code with calls assigning to symbols in your module:
# import foo
foo = v_import(“foo==3.2”)
# from foo import bar, baz as q
bar, q = v_from_import(
“foo>=3.3”,
“bar”,
“baz”,
)
Edit: …and you’d need to edit the sys.modules cache too, or purge it after each “clever” import?
It sucks when there is a vulnerability in a particular library, and you're trying to track all of the ways in which that vulnerable code is being pulled into your project.
My preference is to force the conflict up front by saying that you can't import conflicting versions. This creates a constant stream of small problems, but avoids really big ones later. However I absolutely understand why a lot of people prefer it the other way around.
Or if I depend transitively on two versions of a library (e.g. a matrix math lib) through A and B and try to read a value from A and send it into B. Then presumably due to type namespacing that will fail at compile time?
So the options when using incompatible dependencies are a) it compiles, but fails at runtime, b) it doesn't compile, or c) it compiles and works at runtime?
If the log endpoint is internal to your process, how did you end up with two independent mutexes guarding (or not guarding) access to the same resource? It should be wrapped in a shared mutex as soon as you create it, and before passing it to the different versions of the logging crate. And unless you use unsafe, Rust's ownership model forces you to do that, because it forbids having two overlapping mutable references at the same time.
It's an in memory counter doing an atomic increment that returns the next ID. Two of my projects in depend on it when they create new items. Both want to generate process wide unique IDs. But if they depend on two versions of the crate then there would be two memory locations, and thus two sequences of IDs generated, so two of the frogs in my game will risk having the same ID?
There is no sharing problem here, the problem is the opposite: that there are two memory locations instead of one?
I guess you could slap a `#[used]` attribute on your exported functions, and use their mangled name to call them with dlopen, but that would be unwieldy and guessing the disambiguator used by the compiler error prone to impossible.
Other than that, you cannot. What you can do is define the `#[no_mangle]` or `#[export_name]` function at the top-level of your shared library. It makes sense to have a single crate bear the responsibility of exporting the interface of your shared library.
I wish Rust would enforce that, but the shared library story in Rust is subpar. Fortunately it never actually comes into play, as the ecosystem relies on static linking
Yes, exactly.
> Other than that, you cannot.
so, to the question "Can a Rust binary use incompatible versions of the same library?", then the answer is definitely "no". It's not yes if it cannot cover one of the most basic use cases when making OS-native software.
To be clear: no language targeting OS-native dynamic libraries can solve this, the problem is in how PE and ELF works.
Rust/Cargo have been designed for it from the start.
You depend on two packages, each with a function that returns a “requests.Request” object. These packages depend on different versions of “requests”.
How would you implement “isinstance(return_value, requests.Request)” on each of these calls?
Or, the indirect case of this: catching a “requests.HttpException” from each of these calls?
Importing the right thing isn’t hard, but doing things with it is the hard bit.
from m1 import T1
from m2 import T2
from m3 import f
x = f()
assert isinstance(x, T1, T2)
Ts = v_from_import_all(
“foo”,
“T”,
)
assert isinstance(x, *Ts)
What you do with the return_value defines the behaviour you expect from it so to that extent you can rely on that instead of using isinstance:
x: Union[T1, T2] = f()
print(x.foo() ** 12.3)
If you’re importing a module to use it in some way you’re also declaring some kind of version dependency / compatability on it too, so that’s another thing your static analysis could check for you. That would actually be incredibly useful:
1/ Do your dependencies import an older version of requests than you do?
2/ Does it matter, and why? (eg x.foo() only exists in version 4.5 onwards, but m1 imports 4.4.)
But the main issue here is somewhat designed around a "scripts and folder of scripts from a package" design principle while such a loading system would fundamentally need to always work in terms of packages. E.g. you wouldn't execute `main.py` but `package:main`. (Through this is already the direction a bunch of tooling moved to, e.g. poetry scripts, some of the WSGI and especially more modern ASGI implementations etc.)
Another issue is that rust can reliable detect type collisions of the same type of two different versions and force you to fix them.
With a lot of struct type annotations on python and tooling like mypy this might be possible (with many limitations) but as of today it in practice likely will not be caught. Sometimes that is what you want (ducktyping happens to work). But for any of the reflection/inspection heavy python library this is a recipe for quite obscure errors somewhere in not so obvious inspection/auto generation/metaclass related magic code. Python can't, escept it can
Anyway technically it's possible, you can put a version into __qualname__, and mess with the import system enough to allow imports to be contextual based on the manifest of the module they come from. (Through you probably would not be fully standard conform python, but we are speaking about dynamic patching pythons import system, there is nothing standard about it)
cargo tree -i log@0.3.9
There are equivalents for must other package managers that take this approach, and I've never found this a problem in practice.
Of course, you still need to know that there's a vulnerability there in the first place, but that's why tools like NPM often integrate with vulnerability scanners so that they can check your dependencies as you install them.
Also forces people to actually take backwards compatibility seriously.
I import a@1 and b@1 a@1 transitively depends on c@1 b@1 transitively depends on c@2
Even with different import paths, I still have two different versions of c in my codebase. It'll just be that one of them is imported as "c" and the other will be imported as "c/v2" - but you don't need to worry about that, because that's happening in transitive dependencies that you're not writing.
You still have the same issue of needing to keep track of all the different versions that can exist in your codebase.
Having multiple copies of numpy installed isn't an emergency. Tidy it up at your leisure. Its pretty rare that you end up with multiple copies of the same library installed in your dependency tree anyway.
As for debugging - well, debugging should still work fine so long as your debugging tools treat foo-1.x.x as if it were a completely distinct package from foo-2.x.x. Nodejs and rust both handle this by installing both versions in separate directories. The stack trace names the full file path, and the debugger uses that to find the relevant file. Simple pimple. It works like a charm.
> For web dev and something like requests, it's just not as big of a deal to have a bunch of versions installed.
I think you've got it backwards. Its much more of a big deal on the web because bundle size matters. You don't want to bundle multiple 150kb timezone databases in your website.
I've also worked on multiple web projects which ran into problems from multiple versions of React being installed. Its a common problem to run into, because a lot of naively written web components directly depend on react. Then when the major version of react changes, you can end up with a webpage trying to render a component tree, where some components are created using foreign versions of react. That causes utter chaos. Thankfully, react detects this automatically and it yells at you in the console when it happens.
I haven't ever observed a problem of concurrent access to stdout/err though, I expect because the methods for accessing stdout/err lock them for the duration of their printing. If you Google for "Rust print console slow", you'll probably find advice to explicitly lock it, to avoid individual printlns from each acquiring the lock.
Nobody writing Rust needs to cover this "basic use case" you're referring to, so it is the same as people saying "unsafe exists so Rust is no safer than C++". In theory that's true, in practice in 18 months, 604 commits and 60,008 LoC I wrote `unsafe` exactly twice. Once for memory mapping something, once for skipping UTF-8 validation that I'd just done before (I guess I should have benchmarked that one as it is probably premature).
In practice when developing Rust software at a certain scale you will mix and match incompatible library versions in your project, and it will not be an issue. Our project has 44 dependencies with conflicting versions, one of which appears in 4 incompatible versions, and it compiles and runs perfectly fine. In other languages I used (C++, python), this exact same thing has been a problem, and it is not in Rust. This is what the article is referring to
This mechanism allows Cargo to prevent multiply linking to an external library.
The problem is they are different types, which has huge downstream impacts. Not least of all would be subclassing.
The static analysis you’ve described could work in simple and trivial cases, but the problem you’re now trying to solve is “what concrete type is this object”, and in a dynamic language like Python this can only be fully and 100% determined at runtime.
Mypy and the like do a good job, but often rely on protocols rather than concrete classes. There’s also no impact if the types are wrong or ignored, whereas for this a typing mismatch becomes a subtle runtime issue.
In the easy cases, digging through the PyPI version history to identify the latest version as of some date is enough to get a working install (as far as I can tell -- maybe it's half-broken and I only use the working half?). In the hard cases, it may take an entire day to locate a CI log or contemporary bug report or something that lists out all the installed package versions.
It doesn't help that every Python-based project seems to have its own bespoke packaging system. It's never just pip + requirements.txt, it'll have a Dockerfile with `apt update`, or some weird meta-packaging thing like Conda that adds it own layers of non-determinism. Overall the feeling is that it was only barely holding together on the author's original machine, and getting it to build anywhere else is pure luck.
For example: https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob... (with some discussion at https://github.com/AUTOMATIC1111/stable-diffusion-webui/disc...)
The program depended on fundamentally incompatible sub-dependencies, on different major versions.
The free() function frees the memory space pointed to by ptr, which must have been returned by a previous call to malloc(), calloc() or realloc(). Otherwise, or if free(ptr) has already been called before, undefined behavior occurs. If ptr is NULL, no operation is performed.In practice, usual rust projects that have not put a ton of work into their dependencies encode X.y.z in Cargo.toml matching the current release at the time they developed the system. So you get at worst an unnecessarily higher version requirement but never a lower one.
Moreover, rust semver would normally imply that new features should only be introduced in X.y releases, so this doesn’t really happen in practice!
Take a look at any random crates's cargo config, and you'll regularly see dependencies specified as "1" or "0.3" instead of "1.0.119" or "0.3.39". If another crate depends on this crate, and has a more precise version needed (say "=1.0.100" (perhaps due to a bug introduced in "1.0.101", but the included library relies on features introduced in some version after "1.0.100", then your library won't compile. Stating your dependency on "1" instead of "1.0.119" is what caused this problem.
This is not hypothetical - I've run into this quite a few times with crates that over-specify their dependencies interacting with crates that under-specify theirs.
Yes, the solution to this is pretty simple - check minimal versions in CI. That's something I do for most of my stuff, but it's not universal.
If the frog numbering code is using either counter, it gets unique IDs. The problem only arises if somehow it's using both of them. But why would we expect the "unique" IDs from two different pieces of software be guaranteed never to collide? Imagine instead of counter 1.2.3 and counter 4.5.6 we've got bob_counter and cathy_counter, are you still astonished that they might give out the same IDs? Neither mentions the other, perhaps they're the same code, copy-pasted by egomaniacs.
No I think two sequences is exactly what's asked for. What I'm wondering is whether this is a warning when cargo fails to restore the crates and must use incompatible versions. I'd only be surprised if the behavior was a clean compile + a runtime crash. Because that's usually not the design chosen in Rust.
> If the frog numbering code is using either counter, it gets unique IDs
I guess this is the issue: what I'm imagining is that the counter crate has a static mutable state. That's not a problem with atomics, but it's an antipattern (And especially so in Rust). The solution of course is to NOT use static mutable state at all. Whoever wants to create either a Frog or a Wizard must pass in a universe, from which it can grab the next ID, so it's a counter instance rather than get_next_static_id().
My question isn't "do we really get two memory locations" (of course we do) my question is: how afraid should one be about this, i.e. what is the behavior of the compiler/cargo or other linters when it comes to warnings etc? Are there any non-contrived scenarios where bumping a version of a dependency causes a problem at runtime (only)?
What does "fails to restore the crates" mean here?
I've seen this verb "restore" used in some .NET software, but I can't see how it relates to Rust.
So I'm still not convinced that Go's approach is materially different here - certainly in terms of the practical output, NPM does a good job of ensuring that the fewest number of different versions will get installed for each dependency.
Coming back to a midsized JavaScript codebase after a few months and trying to upgrade to new major versions of things have always been a shitshow.
And I believe Go only using one version is the best solution. It avoids many problems.
And btw: you can force any transitive version in gradle/maven/npm/cargo as well, that’s not a feature unique to Go.
That means you can't use library A and unrelated library B together in the same project, even though you can use A alone and can use B alone. That's lack of orthogonality.
The cargo dependency resolver runs and the results can be viewed by cargo tree. I saw in the docs now that cargo tree -d can be used to help find incompatible packages.
This lets the type's owner decide what affordances to give it (probably adding two UniqueIDs together is nonsense, but comparing them for equality definitely makes sense - should they be Ordered... chronologically? Or not?)
But importantly here now that it's a type, Rust knows counter v 1.2.3 UniqueID and counter v 4.5.6 UniqueID are different types, even though they have the same spelling. So this code now won't compile unless everybody is consistent e.g. the Frogs and Wizards all use v 1.2.3 but the unrelated Space Combat module works with v 4.5.6 UniqueID. Code that looks like it could work, where we try to use the unique ID of a Wizard as the identify of a sub-space laser blaster won't compile.
A pair of crates (one which defines the struct UniqueNumber(u32), and one which produces sequences of them) would still be suffering from this, because the sequencing crate could use v1 of the unique number struct crate, in both its v1 and v2.
This of course is even more contrived (now we have a specific split of state and types instead). So the real question is: does this happen in non-contrived scenarios in the wild?
One of my favourite decisions about Rust which could have gone the other way was their choice to define &str (and String) to be actual text. &str is basically &[u8] but with an extra requirement that this is valid UTF-8 text, likewise then String is basically Vec<u8> but requiring UTF-8.
I feel like UniqueNumber is the same, if you call yourself UniqueNumber don't just be a thin wrapper around an integer, you need to own that uniqueness problem.
But yeah I get your point.