Small time players like GE routinely fail to correctly sign industrial control software, the odds of people recording video paying enough attention to get it right and the meme crowd bothering to check even if they did seems vanishingly small without a lot of educational effort.
We are starting to see adoption of software supply-chains with SBOMS, albeit imperfectly. We are starting to see increased adoption of things like DMARC in the email space to better authentic the originator of an email. Both are highly imperfect systems ... but you can start kludging something together ... and if the incentives are there I think you can build out more of a workable system.
It's not the cryptography which is the problem. It's, who do you trust with the signing keys? The list inherently has to include every camera maker, despite that industry generally not having a great security culture, as well as every camera's country of origin, and every country with a security service capable of infiltrating some other country's camera maker. Which is probably all of them.
Worse, the keys have to be in the camera. Every camera. Break one of any model and you can forge images with it. Break one of any model and publish the break and you call into question every image from every camera of that type.
Then, even if a camera hasn't given up its keys, someone can use it to take a picture of a picture.
None of this requires a cryptographic break of public key signatures.
I've wanted to build a product in this space ever since I heard about deepfakes. Mix of keybase and appropriate file hash, and hash gen for subsets of sections of video. Maybe it needs to be a protocol, maybe a product, not sure, but the need seems apparent to me.
For example, HDCP is a DRM scheme where Intel convinces (or legally requires) every manufacturer of HDMI output devices (e.g. set-top boxes, Blu-ray players) in the world to encrypt certain video streams.
Then, Intel requires manufacturers of HDMI input devices (e.g. TVs) to purchase a license key that can decrypt those video streams. This license agreement also requires the manufacturer to design their device such that the device key cannot be easily discovered and the video content cannot be easily copied.
Then, Intel gets media companies to include some extra metadata in video media like Blu-ray discs. This metadata can contain revoked device keys, so that if a TV manufacturer violates the terms of the license agreement (e.g. leaks their key, or sells a device that makes copies of video content), that manufacturer's TVs won't be able to play new content that starts including their key in the revocation list.
Of course, Intel's HDCP master key was either leaked or reverse-engineered, so anyone can generate their own valid device keys. Intel will probably sue you if you do this, I guess.
(though if you have lots of time/effort/money you could still extract the key)
1) Authentic and only lightly edited with image manipulation software (e.g., cropped, color balanced, or text placed over top of the image) 2) Produced on a phone that has had to go through hardware hacks
Note that the guarantee in (1) wouldn't prevent someone from taking a photo of a TV screen. When I asked that original question, I had quite a few more details about how the certification might be done, how the credentials would be hosted, and how the results would be shown on a website.
Anyway, just asking this question was met with a storm of negative responses. I counted two dozen messages that were either neutral (asking for clarification) or else outright hostile before the first hesitantly positive message. My favorite hostile response was that allowing people to certify images as real would steal peoples' rights. I didn't follow the logic, but the guy who made the argument was really into it.
There were lots of comments about how using AI would be a better solution, some commenting on how Cannon already did it (and messed up gloriously), others stating they didn't have faith in hardware... it makes a fella never want to ask a question again.
In the end, I got an expert to speculate that the technology currently exists, and has existed for 5-10 years, to do this with a modern smartphone. However, unless a high-level engineer or executive argues that providing this feature will somehow be a competitive advantage, there is no appetite to provide this kind of feature.
My guess is likely because it seems like this would be impossible to implement without adding DRM to the smartphone and/or locking down Open Source image editors out of the attestation process. You would need to prevent access to the software, firmware, etc... otherwise the device could be virtualized or the program recompiled to circumvent the signature.
And for obvious reasons there's going to be pushback to adding that kind of DRM to smartphones. The tech does likely exist; this sounds to me like just normal attestation? It would likely hook into something like the Play Integrity API. A lot of people already hate the Play Integrity API though.
It's not the tech that's the problem, it is as you say, that people are hesitant to do it because it would require locking down the phone's software stack in a way that is widely understood by many developers and user advocates to be anti-user and in contrast to user rights to control their own devices and load their own software and/or firmware onto their devices.
I could maybe see an argument introducing some kind of signature to a raw camera input in firmware before it ever reached the user at all -- mostly just because devs seem to have given up the fight about custom firmware on a phone in general. But if you're talking about the phone signing the image after light editing like a crop has happened, at that point you're talking about moving this signature into user-space code, and while I'm sure that problem could have been explained better to you by the devs, it's not surprising to me at all that you'd get a hostile response to that suggestion because I don't see how it would be possible to do that without locking down user-space code.
Perhaps something like this is what your hostile responder was thinking of.
If you're talking someone on Twitter or Facebook is putting a photo in your feed claiming a human photographed BLM throwing bricks through a window, don't trust shit being posted on Facebook or Twitter no matter what, probably, but even there, unless the profile was hacked, you either trust the person who owns it or you don't. Nothing would prevent them from signing a forgery of reality that they legitimately forged themselves. Even with device-level keys, what are you trying to prove? You can pay actors to throw bricks through windows.
I guess the concern is this doesn't scale as well as asking Midjourney to do it, but I wonder to what extent that is even true. With 8 billion people on the planet and counting and a whole lot of them doing this shit, given the limited input bandwidth of human sense organs, there is seemingly some maximum saturation of bullshit a person can be exposed to that a lot of people have already hit, and having the Internet host even more of it doesn't mean they'll grow bigger eyes and a faster brain that can actually ingest more bullshit than it already does.
The Washington Post might not trust its photographers completely either (journalists making stuff up happens[0]), so they too might want proof the photos they're getting are real.
[0]: https://www.nytimes.com/2003/05/11/us/correcting-the-record-...
If I see a photo on twitter claiming to come from the Washington Post, it might not be.
If I see a photo in my facebook feed of a rioter, did it come from poster, or are they just reposting something else? Did that repost come from a newsource I trust, like the WP in this case, or from some reddit post, maybe edited or synthetically generated?
> Maybe they're showing you something made by AI that isn't real, but as the owner of their own signing key, nothing would prevent them from signing an AI-generated image
That's right. This only helps narrow the source down, then you still need to decide if you trust the originator. But I think a lot of the problems we've seen with social media disinformation is the wide dispersion of content claiming trustworthiness from a reputable source, falsely.
Unless you have reason to trust Bill himself you can't trust that he actually took the photo, or that it isn't ai generated. Although knowing that Bill isn't tech savvy enough to do those things might be enough.
There are industry initiatives around this already such as CAI https://en.m.wikipedia.org/wiki/Content_Authenticity_Initiat...
My take is that proving authenticity might not be something we can do with any degree of accuracy in a general sense. So if that is infeasible, then we need _some_ kind of mitigation. Something like CAI allows us to make the an assessment about the how much trust to give an informational source, probably taking into account multiple factors (known exploits in the source device, reputation of originator and what claims are be attached in the metadata). This might allow me to accept that a given video originated from a local tv station, rather than tiktoker edit, but I still need to asses if that station used genAI or has been compromised or whatever else. But that seems a much narrower reputational problem, that also will be contextual.
It’s a direct (and open source) implementation of public key cryptography into the LLM logit distribution.
The paraphrasing model/beam search needs work - feel free to pitch in :)
Would signing content with a cryptographically consistent encoding of this field be workable?
Similarly, I suspect watermarking LLM output is probably unworkable. The output of a smart model could be de-watermarked by fine tuning a dumb open source model on the initial output, and then regenerating the original output token by token, selecting alternate words whenever multiple completions have close probabilities and semantically equivalent. It would be a bit tedious to perfectly dial in, but I suspect it could be done.
And then ultimately, short text selections can have a lot of meaning with very little entropy to uniquely tag (e.g., covfefe).
[1] https://dl.acm.org/doi/abs/10.1145/2382448.2382450
Curious if Scott Aaronson solved this challenge...
Current LLMs have stylistic quirks imprinted on them by RLHF (ChatGPT's endless "it should be noted" and "it is important to remember that" verbiage is a good example), but they learned those from human writing.
That will be much harder to evade, but also pretty hard to implement.
I guess we will end up in the middle ground, where any non-signed image could be ai generate, but for most day to day use it’s ok.
If you want something to be deemed legit (gov press release, newspaper photo, etc) then just sign it. Very similar to what we do for web traffic (https)
If you create a digital record, then sign it, then that signature is only an attestation of may claim you make, not evidence of that claim. That is the problem with relying on technology to establish trust - the moment you attach an economic benefit to a technology you incentivize people to circumvent it, or to leverage it to commit fraud.
If AI will eventually generate say 10k by 10k images, I can resize to 2.001k by 1.999k or similar, and I just don't get how any subtle signal in the pixels can persist through that.
Maybe you could do something at the compositional level, but that seems restrictive to the output. Maybe something about like larger regions average color balance or something? But you wouldn't be able to fit many bits in there, especially when you need to avoid triggering accidentally.
Also: here are some play money markets for whether this will work:
https://manifold.markets/Ernie/midjourney-images-can-be-effe...
https://manifold.markets/Ernie/openai-images-have-a-useful-a...
It needs to publicly fail first to manufacture consent for full surveillance of every human interaction with any computer. Nobody would ever want that otherwise.
At the moment the internet is a wash with bullshit images. Its imperative that news outlets are at a high enough standard to actually prove the provenance of them.
You don't trust some bloke off facebook asserting that something is true, its the same for images.
If you want the camera sensor in your phone to certify that the user hasn't altered the image, the device can't be in the control of the user.
Also bear in mind that Photoshop does exist today, and even without AI it would be well within the budget of the Washington Post to do edits to photographs that would be good enough to convince most non-experts. I don't run into many people saying their photos are doctored, but the conspiracy theories around photos today don't seem to be swayed by saying, "experts say it's not doctored."
I think if a chain of trust requires ordinary people to check signatures, it's probably not going to matter very much? I've seen people comparing this to HTTPS; people don't check HTTPS certificates either. There's a reason why PGP and signed messages haven't taken off. And the biggest criticism of Matrix that I see today when I introduce new users is that identity verification is too hard and they don't care about it. And these are the reasonable people.
I'm trying to imagine talking to someone who believes the mainstream media is lying about everything and telling them that, "no, it's OK, Google the tech company that you hate and that you think is trying to swing the election checked some math from its hardware and it says the photo is real." I don't think that's going to persuade those people of anything.
Signing only helps if we have a reliable way to verify identities out of band, and we don't, that's one of the reasons the other problems you mention still exist today. Today, how do you determine that an article is actually from the Washington Post? You check the website. It's not like verification of whether an article exists is hard today -- it's everything else around it that's hard.
If checking if an article is actually on a website is too hard for people, importing a signing key is also going to be too hard. If people are confused remembering the Washington Post's URL, they won't learn how to use a signing key to check identity. Now maybe a website could automate that and put some kind of verification badge next to trustworthy verified identities, but I'm skeptical because Facebook did try to do that with news sources and it was a disaster and a bunch of politicians accused them of censorship.
We have a lot of mechanisms for verifying identity and sources of information that aren't leveraged today, and I think the immediate question to ask about a chain of trust is "what's going to make this different from all of the other chains of trust that people are already ignoring today?"
Narrowing down the problem to "Do we trust Bill" is at least something we can attempt to address. This also eliminates some sources where we can more easily discount things. If the source is 4chan, maybe I don't need much more information to make my repetitional assessment.
And I suspect in most cases that is enough - the world runs on trust, so if a reliable source (eg, the NY Times) attests that a photograph is authentic, then we can reasonably trust that. And as you noted, this chain of provenance can go all the way back to the device itself.
Fraud, on the other hand, is much harder to prove without a doubt. It is still a problem, but probably less so in the general case [1]. The concerning thing seems to be extremely targeted attacks, eg, hacking a CCTV to implicate someone in a crime.
[1] Notwithstanding folks who slurp up unfiltered content on, eg, TikTok, but the most outrageous stuff usually doesn't "make the news" on its own without more vetting. If anything, this will reinforce the importance of actual journalism, fact-checking and corroborating evidence.
What happens if the original was taken with any existing device that doesn't make signatures, or is a model that subsequently had its keys revoked?
This is more like prioritization. If you can't even install your own apps, you focus on the gorilla holding a knife to your throat.
> But if you're talking about the phone signing the image after light editing like a crop has happened, at that point you're talking about moving this signature into user-space code, and while I'm sure that problem could have been explained better to you by the devs, it's not surprising to me at all that you'd get a hostile response to that suggestion because I don't see how it would be possible to do that without locking down user-space code.
That's the part that isn't a problem. If you had an existing image with an existing signature, you could modify it and store the changes as a diff against the original. You don't need or even want to sign it again, you just keep the original and its signature intact. Compressing two images that are nearly identical against each other shouldn't even have particularly high overhead.
Doing it this way would also be more secure because you wouldn't have to trust the device doing the modifications in any way.
The problem continues to be how to create such a signature to begin with, without depriving the user of control over their own property or leaving the keys inside of devices that are in the physical possession of every attacker in the world.
I think this glosses over things a little bit. Are you going to transmit the original and the diff to every image viewer? People are talking about doing these checks on clientside devices, not just having an attestation check somewhere else.
Ultimately the only way you can check this is to give someone the original and the signature to compare. Want to blur or censor a face? Tough. Want to crop? Tough. And the person doing that verification would want to be able to look at the photograph to tell how extensive your edits were.
Technically what you're saying is true in that you could do diffs this way, but in practice you'd have to commit to publishing the pre-edit photo. We're also suddenly no longer talking about a behind-the-scenes process that just puts a little green check on the photo or something; because edits can be anything and only the original photo would be signed; so the "verification" in your image editor would now be a software stack that shows you the original photo alongside the edits I guess?
----
> The problem continues to be how to create such a signature to begin with, without depriving the user of control over their own property or leaving the keys inside of devices that are in the physical possession of every attacker in the world.
I'm quibbling though, I think we're mostly in agreement. This is the DRM aspect that people seem to be forgetting. Commentary about attestation is not making the obvious and direct comparison that controlling device behavior is already something companies are trying to do and failing at.
Well of course they would. Otherwise what are you even trying to attest? Otherwise someone could take an image from a camera, replace literally every pixel with whatever they want and then claim it's the same image.
If this is some cloud provider, what are you getting from a blockchain? Just have the cloud provider do the certification. If they betray you or go out of business you've lost your hashes anyway.
If it's stored on the endpoint device, you can't prove it anymore if the device gets lost or damaged. In theory people could back them up, but we all know perfectly well that ordinary people are not going to do that unless it's automated.
So then you're back to storing them in a distributed system, i.e. making them a necessary part of the blockchain. And then it gets too big.
Of an older version of HDCP. New media can require a higher HDCP version where that bypass isn't possible.
Yes, see 4k blurays for requiring HDCP 2.2 requiring people to get a new bluray player.
What stops the blu-ray reader from just ignoring the revocation list on the disk?
Now that I think of it, I wonder if compliant Blu-ray players actually save the new revocation entries and then continue refusing to negotiate with revoked TVs even for old Blu-ray discs.
What stops them from being sold that way would probably be the licensing agreement and honest players. I'd imagine in China, there are lots of these types of devices available.
if you want your blu-ray player to be able to read blu-ray disks you sign a contract that says you will respect the revocation list
if you change your mind later: your player key will be revoked and new blu-rays won't play on your device
(it's actually more sophisticated than this... they can block specific players too)
Obscuring faces for privacy, cropping would no longer work. And even minor touchups like lighting would be of questionable value since you're expecting users not to look at the edited photo or at least to primarily look at the edited photo next to the original.
I suspect in practice that doing edits on top of a signed photo would be basically the same as not having editing capability at all; and even that's assuming users would compare the edited and non-edited versions at all, which is not a safe assumption in my mind given how hard it is to even get people to click into a full article past the headline.
You could also handle cropping and omissions by having the original device sign the picture as a grid of individual tiles. Then you could omit some and still prove that the others are original.
> You could also handle cropping and omissions by having the original device sign the picture as a grid of individual tiles.
is a pretty good idea, I like that quite a lot. Not saying it means I'm on board with signatures overall (I mean, we're still in agreement that this would require locking down devices to at least some degree) but I do think that's an elegant solution for the cropping/censoring part of it.
For the actual hashes, you can back them up in multiple places literally anywhere, including the image metadata itself. The advantage is nobody needs to trust a cloud provider to keep that S3 object intact and online in perpetuity, or the owner to keep paying for that, because the original date is on a public blockchain synced by 10,000s of nodes.
you can obviously think whatever you want, but you'd be completely wrong
DVD supported this 20 years ago, blu-ray's system is far more sophisticated and can even block individual players
The approach of AACS provisions each individual player with a unique set of decryption keys which are used in a broadcast encryption scheme. This approach allows licensors to "revoke" individual players, or more specifically, the decryption keys associated with the player. Thus, if a given player's keys are compromised and published, the AACS LA can simply revoke those keys in future content, making the keys/player useless for decrypting new titles.
the spec also supports a persistent CRL so a new disk can also stop your old disks from working
Then every time another player's keys are published it allows anyone to use the older player to read discs using the newer player's leaked keys. And some players are cracked but the keys aren't published, instead they use them to extract the disc key for every new disc and then publish all the disc keys, which can be used in the same way without revealing which player was cracked.