Compressing Images with Neural Networks(mlumiste.com) |
Compressing Images with Neural Networks(mlumiste.com) |
The contextual information surrounding intentional data loss needs to be preserved. Without that context, we become ignorant of the missing data. Worst case, you get replaced numbers. Average case, you get lossy->lossy transcodes, which is why we end up with degraded content.
There are only two places to put that contextual information: metadata and watermarks. Metadata can be written to a file, but there is no guarantee it will be copied with that data. Watermarks fundamentally degrade the content once, and may not be preserved in derivative works.
I wish that the generative model explosion would result in a better culture of metadata preservation. Unfortunately, it looks like the focus is on metadata instead.
One key item with emerging 'AI compression' techniques is the information loss is not deterministic which somewhat complicates assessing suitability.
It is technically possible to make it deterministic.
The main reason you don't deterministic outputs today is that Cuda/GPU optimizations make the calculations run much faster if you let them be undeterministic.
The internal GPU scheduler will then process things in the order it thinks is fastest.
Since floating point is not associative, you can get different results for (a + (b + c)) and ((a + b) + c).
2. "Lower DPI" is extremely common if your definition for that is 300dpi. At my company, all the text document are scanned at 200dpi by default. And 150dpi or even lower is perfectly readable if you don't use ridiculous compression ratios.
> Other artifacts at lower resolution can exhibit similar mangling as well (specifics would of course vary)
Majority of traditional compressions would make text unreadable when compression is too high or the source material is too low-resolution. They don't substitute one number for another in an "unambiguous" way (i.e. it clearly shows a wrong number instead of just a blurry blob that could be both).
The "specifics" here is exactly what the whole topic is focus on, so you can't really gloss over it.
https://pub.towardsai.net/stable-diffusion-based-image-compr...
HN discussion: https://news.ycombinator.com/item?id=32907494
Even when going down to 4-6 bits per latent space pixel the results are surprisingly good.
It's also interesting what happens if you ablate individual channels; ablating channel 0 results in faithful color but shitty edges, ablating channel 2 results in shitty color but good edges, etc.
The one thing it fails catastrophically on though is small text in images. The Stable Diffusion VAE is not designed to represent text faithfully. (It's possible to train a VAE that does slightly better at this, though.)
This is intuitive, as the competition organisers say: compression is prediction.
Best overview you can probably get from “JPEG AI Overview Slides”
The interesting diagram to me is the last one, for computational cost, which shows the 10x penalty of the ML-based codecs.
One problem is that without broad adoption, support even in niche cases is precarious; the ecosystem is smaller. That makes the codec not safe for archiving, only for distribution.
The strongest use case I see for this is streaming video, where the demand for compression is highest.
Plus the entire model, which comes with incorrect cache headers and must be redownloaded all the time.
I looked at the bear example above and I could see how either the AI thought that there was an animal face embedded in the fur or we just see the face in the fur. We see all kinds of faces on toast even though neither the bread slicers nor the toasters intend to create them.
However, this weekend someone released an open-source version which has a similar output. (https://replicate.com/philipp1337x/clarity-upscaler)
I'd recommend trying it. It takes a few tries to get the correct input parameters, and I've noticed anything approaching 4× scale tends to add unwanted hallucinations.
For example, I had a picture of a bear I made with Midjourney. At a scale of 2×, it looked great. At a scale of 4×, it adds bear faces into the fur. It also tends to turn human faces into completely different people if they start too small.
When it works, though, it really works. The detail it adds can be incredibly realistic.
Example bear images:
1. The original from Midjourney: https://i.imgur.com/HNlofCw.jpeg
2. Upscaled 2×: https://i.imgur.com/wvcG6j3.jpeg
3. Upscaled 4×: https://i.imgur.com/Et9Gfgj.jpeg
----------
The same person also released a lower-level version with more parameters to tinker with. (https://replicate.com/philipp1337x/multidiffusion-upscaler)
Their example with the cake is the most obvious. To me, the original image shows a delicious cake, and the modified one shows a cake that I would rather not eat...
Make sure to test the models before you deploy. Nothing will be lossless doing superresolution but flows can get you lossless in compression.
[0] https://huggingface.co/docs/diffusers/api/pipelines/stable_d...
It is relevant only when you assume that lossy compression has no way to control or even know of such critical changes. In reality most lossy compression algorithms use a rate-distortion optimization, which is only possible when you have some idea about "distortion" in the first place. Given that the error rarely occurred in higher dpis, its cause should have been either a miscalculation of distortion or a misconfiguration of distortion thresholds for patching.
In any case, a correct implementation should be able to do the correct thing. It would have been much problematic if similar cases were repeated, since it would mean that it is much harder to write a correct implementation than expected, but that didn't happen.
> Majority of traditional compressions would make text unreadable when compression is too high or the source material is too low-resolution. They don't substitute one number for another in an "unambiguous" way (i.e. it clearly shows a wrong number instead of just a blurry blob that could be both).
Traditional compressions simply didn't have much computational power to do so. The "blurry blob" is something with lower-frequency components only by definition, and you have only a small number of them, so they were easier to preserve even with limited resources. But if you have and recognize a similar enough pattern, it should be exploited for further compression. Motion compensation in video codecs were already doing a similar thing, and either a filtering or intelligent quantization that preserves higher-frequency components would be able to do so too.
----
> 2. "Lower DPI" is extremely common if your definition for that is 300dpi. At my company, all the text document are scanned at 200dpi by default. And 150dpi or even lower is perfectly readable if you don't use ridiculous compression ratios.
I admit I have generalized too much, but the choice of scan resolution is highly specific to contents, font sizes and even writing systems. If you and your company can cope with lower DPIs, that's good for you, but I believe 300 dpi is indeed the safe minimum.
Many core codecs are pretty good at adhering to reference implementations, but are still open to similar issues so may not be bit exact.
With a DCT or wavelet transform, quantisation, chroma subsampling, entropy coding, motion prediction and the suite of other techniques that go into modern media squishing it’s possible to mostly reason about what type of error will come out the other end of the system for a yet to be seen input.
When that system is replaced by a non-linear box of mystery, this ability is lost.
The cartoons & illustrations lose all of their gradations in feeling & tone with every outline a harsh edge. The landscapes lose any sense of lushness and atmosphere, instead taking a high-clarity HDR look. Faces have blemishes inserted the original actor never had. Fruit is replaced with wax imitation.
As an artist, I would never run any of my art through anything like this.
https://replicate.com/philz1337x/clarity-upscaler https://replicate.com/philz1337x/multidiffusion-upscaler
Learn More: https://www.dkriesel.com/start?do=search&id=en%3Aperson&q=Xe...
Brief: Xerox machines used template matching to recycle the scanned images of individual digits that recur in the document. In 2013, Kriesel discovered this procedure was faulty.
Rationale: This method can create smaller PDFs, advantageous for customers that scan and archive numerical documents.
Prior art: https://link.springer.com/chapter/10.1007/3-540-19036-8_22
Tech Problem: Xerox's template matching procedure was not reliable, sometimes "papering over" a digit with the wrong digit!
PR Problem: Xerox press releases initially claimed this issue did not happen in the factory default mode. Kriesel demonstrated this was not true, by replicating the issue in all of the factory default compression modes including the "normal" mode. He gave a 2015 FrOSCon talk, "Lies, damned lies and scans".
Interesting work!
To see what a lossy generator hallucinating patterns means in practice, I recommend viewing HiFiC vs original here: https://hific.github.io/
Could you explain what you mean by "not safe for archiving"? The standard is published and there are multiple implementations, some of which are open-source. There is no danger of it being a proprietary format with no publicly available specification.
As a counter, J2K has been well established by the professional market even if your mom doesn't know anything about what it is. It has been standardized by the ISO, so it's not something that will be forgotten about. It's a good tool for the right job. It's also true that not all jobs will be the right ones for that tool
Truly standard ANSI C along with a number of other implementation strategies (LLVM IR seems unlikely to be going anywhere) seem just as durable as WASM if not more, but there are applications where you might not want to need a C toolchain and WASM can be a fit there.
One example is IIUC some of the blockchain folks use WASM to do simultaneous rollout of iterations to consensus logic in distributed systems: everyone has to upgrade at the same time to stay part of the network.
LLVM isn't as simple: there's not really such thing as target-independent LLVM IR, there are lots of very specific keywords with subtle behavioural effects on the code, and it's hard to read. I think LLVM is the only full implementation of LLVM. (PNaCl was a partial reimplementation, but it's dead now.)
ANSI C is a very complicated language and very hard to implement correctly. Once Linux switches to another language or we stop using Linux, C will go the way of Fortran.
Part of archiving information has always been format shifting. Never think you can store information, forget about it for a thousand years (or even five), and have it available later.
ANSI C is among the simpler languages to have serious adoption. It’s a bit tricky to use correctly because much of its simplicity derives from leaving a lot of the complexity burden on the author or maintainer, but the language specification is small enough in bytes to fit on a 3.5” floppy disk, and I think there are conforming implementations smaller than that!
You seem to be alluding to C getting replaced by Rust as that’s the only other language with so much as a device driver to its name in the Linux kernel. Linus is on the record recently saying that it will be decades before Rust has a serious share of the core: not being an active kernel code tribute I’m inclined to trust his forecast more than anyone else’s.
But Rust started at a complexity level comparable to where the C/C++ ecosystem ended up after 40 years of maintaining substantial backwards compatibility, and shows no signs of getting simpler. The few bright spots (like syntax for the Either monad) seem to be getting less rather than more popular, the bad habits it learned from C++ (forcing too much into the trait system and the macro mechanism) seem to have all the same appeal that template madness does to C++ hackers who don’t know e.g. Haskell well. And in spite of the fact that like 80% of my user land is written in Rust, I’m unaware of even a single project that folks can’t live without that’s married to Rust.
Rust is very cool, does some things very well, and it wouldn’t be hard to do a version of it without net-negative levels of opinionated on memory management, but speaking for myself I’m still watching Nim and V and Zig and Jai and a bunch of other things, because Rust takes after its C++ heritage more than its Haskell heritage, and it’s not entrenched enough in real industry to justify its swagger in places like HN.
The game is still on for what comes after C: Rust is in the lead, but it’s not the successor C deserves.
