Reverse engineering Gemini's SynthID detection(github.com) |
Reverse engineering Gemini's SynthID detection(github.com) |
If you assume competence from Google, they probably have two different watermarks. A sloppy one they offer an online oracle for and one they keep in reserve for themselves (and law enforcement requests).
Also given that it's Google we are dealing with here, they probably save every single image generated (or at least its neural hash) and tie it to your account in their database.
How is the model relevant? The models are proprietary and you never see any of its outputs that haven't been watermarked.
I also read a lot of comments on HN that start by attacking the source of the information, such as saying it was AI assisted, instead of the actual merits of the work.
The HN community is becoming curmudgeonly and using AI tooling as the justification.
As for AI specifically.. life is too short to read all the interesting pages already, and AI just makes is so much worse.
- AI is verbose in general, so you are spending a lot of time reading and not getting much new facts out of that.
- Heavy AI use often means that author has little idea about the topic themselves, and thus cannot engage in comments. Since discussion with authors are often most interesting part of HN, that makes submission less interesting.
And yes, it is possible to use AI assistance to create nice and concise report on the topics you can happily talk about, but then this would not be labeled as "AI".
The README itself reads like unedited AI output with several layers of history baked in.
- V1 and V2 appear in tables and diagrams but are never explained. V3 gets a pipeline diagram that hand-waves its fallback path.
- The same information is restated three times across Overview, Architecture, and Technical Deep Dive. ~1600 words padded to feel like a paper without the rigor.
- Five badges, 4 made up, for a project with 88 test images, no CI, and no test suite. "Detection Rate: 90%" has no methodology behind it. "License: Research" links nowhere and isn't a license.
- No before/after images, anywhere, for a project whose core claim is imperceptible modification.
- Code examples use two different import styles. One will throw an ImportError.
- No versioning. If Google changes SynthID tomorrow, nothing tells you the codebook is stale.
The underlying observations about resolution-dependent carriers and cross-image phase consistency are interesting. The packaging undermines them.
There are already ten million AI image generators, the overwhelming majority of which do not watermark their outputs. Google auto-inserting them is nice, but ultimately this kind of tool to remove them will inevitably be widespread.
If a system depends on every person on the planet not doing one particular thing or the system breaks, expect the system to break quickly.
This is an especially common trope in software. If someone can make software that does something you consider bad, it will happen. Also it's software. There is no difference between it being available to one person or a million. The moment the software exists and can be copied an unbound number of times.
So it's a "no" by default.
Also something about how AI is not special and we haven't added or needed invisible watermarks for other ways media can be manipulated deceptively since time immemorial, but that's less of a practical argument and more of a philosophical one.
[0]: if it does what it claims to do. I didn't verify. Given how much AI writing in the README my hunch is that this doesn't work better than simple denoising.
This one is such a gigantic clusterfuck... They're mimicking ASCII tables using Unicode chars of varying length and, at times, there's also an off-by-one error. But the model (not Claude, but the model underneath it) is capable of generating ASCII tables.
P.S: I saw the future... The year is 2037 and we've got Unicode tables still not properly aligned.
This project proves what red teaming was in place wasn't good enough.
Oh hey, neat. I mentioned this specific method of extracting SynthID a while back.[1]
Glad to see someone take it up.
Meta: your comment was marked [dead], like a few other constructive comments I saw in recent days. Not sure why.
I appreciate you pointing it out, but this account is banned. Thank you for vouching though!
Because it’s meaningless for what it’s being marketed for. It’s conceptually inverted. It’s a detector that will detect 100% of the stuff that doesn’t mind being detected, and only the dumbest fraction of stuff that doesn’t want to be detected.
No fault of the extremely smart and capable people who built it. It’s the underlying notion that an imperceptible watermark could survive contact with mass distribution… it gives the futile cat-and-mouse vibes of the DRM era.
Good guys register their guns or whatever, bad guys file off the serial numbers or make their own. Sometimes poorly, but still.
All of which would be fine as one imperfect layer of trust among many (good on Google for doing what they can today). The frustrating/dangerous part is that it seems to be holding itself out as reliable to laypeople (including regulators). Which is how we end up responding to real problems with stupid policy.
People really want to trust “detectors,” even when they know they’re flawed. Already credulous journalists report stuff like “according to LLMDetector.biz, 80% of the student essays were AI-generated.” Jerry Springer built an empire on lie detector tests. British defense contractor ATSC sold literal dowsing rods as “bomb detectors,” and got away with it for a while [2].
It’s backward to “assume it’s not AI-origin unless the detector detects a serial number, since we made the serial number hard to remove.” Instead, if we’re going to “detector” anything, normalize detecting provenance/attestation [e.g. 0]: “maybe it’s an original @alwa work, but she always signs her work, and I don’t see her signature on this one.”
Something without a provable source should be taken with a grain of salt. Make it easy for anyone to sign their work, and get audiences used to looking for that signature as their signal. Then they can decide how much they trust the author.
Do it through an open standards process that preserves room for anyone to play, and you don’t depend on Big Goog’s secret sauce as the arbiter of authenticity.
I hear that sort of thinking is pretty far along, with buy-in from pretty major names in media/photography/etc. The C2PA and CAI are places to look if you’re interested [1].
…and that is why I am “con.”
[0] https://contentcredentials.org/
One workflow that some artists use is that they draw with ink on paper, scan, and then digitally color. Nothing prevents someone from generating line art using generative AI, printing it, scanning it, and coloring it.
And what if someone just copy pastes something into Photoshop or imports layers? That's what you'd do for composites that mix multiple images together. Can one copy paste screenshots into a multi layer composition or is that verboten and taints the final image?
And what about multi program workflows? Let's say I import a photo, denoise it in DxO, retouch in affinity photo, resize programmatically using image magick, and use pngcrush to optimize it, what metadata is left at the end?
If only everyone just agrees with me.