uv: Deduplicate all files in the wheel cache(github.com) |
uv: Deduplicate all files in the wheel cache(github.com) |
Funny to list OpenAI as an example at the end of the blog post ...
Or (nasty) bugs hard to debug
Disk isn't free, especially now.
Blake3 is really a wonderfully fast cryptographic hash. I use it for my own "deduplication / integrity / berzerker" utility (which I made before LLMs were a thing).
If I've got a file named:
DSC98731-b3-7b39197a22.JPG
- if that file doesn't checksum back to 7b39197a22 there's a file integrity problem (amazing and it already helped me troubleshoot issues)
- if any other file has the same Blake3 7b39197a22 hash, it's a duplicate
- if that 7b39197a22 checksum is in my database, "things can happen".
It's really great (and I know several here independently made similar schemes) and Blake3 is an amazing hash for those kind of use.
I wasn't very clear.
Do you have it in a public repo you could share?
But I know others did similar thing so maybe there are public repos out there. But in any case: it should now be the kind of thing relatively easy to vibe-code if it's for your own use.
Even formats which should know better, like SQLite, delegate that to the filesystem, most of which are also not checksumed and which delegate that further to the storage.
PostgreSQL, which prides itself by it's quality and reliability, only turned on checksums by default in the last version, 18.
This is one great benefit of using .zip files as file formats, you get this for free.
Though fair enough, it could offer it as an opt-in thing.
I agree.
> Even formats which should know better, like SQLite, delegate that to the filesystem, most of which are also not checksumed and which delegate that further to the storage.
I mostly run ext4 (desktop, laptops, etc.) but for my main server at home, it's a ZFS (mirrored) tank on an old server with ECC RAM.
And backups. So much backups.
pip install git+https://github.com/some-org/repoCompared to what? Fast is relative. Compared to pip it's miles ahead.
Sounds like it's worth another look at your settings to make sure they are right.
Astral people are already in the comments making pro AI coding statements. By using uv, you are literally supporting the people who want to make you unemployed for stock options.
By using any FOSS or software produced by software developers and programmers, you are supporting the idea that computation can and should be automated, instead of having humans doing it. This is the origin of computing, and what we've been doing so far, and it continues to "eat the world" via automation, just like the past decades of it.
The AI cartel is the new Microsoft.
It is amazing how software developers have been brainwashed. Ted Ts'o, Google employee and pro-AI shill at Debian and LWN, recently argued that supporting AI is similar to supporting the Internet in 2000.
That is another fallacy. The Internet is a common carrier and was supposed to lead to freedom in 2000. It has nothing to do with outsourcing your thinking to a bunch of wannabe trillionaires.
I can't see any other comment on this post mentioning AI, did you just make this up?
Whereas your sqlite data are your data, and if they're corrupted they can be lost forever or propagate the issue to backups.
I had to give up on things like Lean theorem prover projects and VM projects because the disk space wasn't there.
The causes of disk waste are manifold and usually can't just be disabled and even when they can the result is constant rebuilding, redownloading, etc.
Too many developer products act like disk and memory is free to waste and the consequences have gotten ridiculous.
Software engineers are the socially dumbest creatures on the planet and deserve every bit of scorn that was heaped on them at high school. The industry used them for multi level marketing since about 2010 and now has instructed them to direct the MLM against themselves.
And they comply!
"Programming" as a concept started with humans being "tired" of manually calculating things, then it became faster than humans, and now our sand can talk human language.
And speaking about false equivalency, there are plenty of models you can run locally yourself. I agree that big tech sucks, and many places in the world need less surveillance and less corporate control, but I don't think the entire LLM-space (besides very obviously the biggest companies) are responsible for that kind of usage.