I find it amusing that they probably ran this tool against a set of millions or even billions of images and this is the best they could come up with. They are practically praising Apple here lmao
It won't catch anything but the dumbest of dumb criminals, because those who care about CSAM can surely figure out a better way to share images, or find a way to obfuscate their images enough to bypass the system (the lower the false positive rate, the easier it must be to trick the system).
So what's left when all the criminals this is supposed to catch have figured it out?
False positives. Only false positives.
Is it really worth turning personal devices into snitches that don't even do a good job of protecting children?
Also, numbers about false positives must be taken with a grain of salt because of the non-uniform distribution of perceptual hashes. It might be that your random vacation photos and kitty pics have a 1-in-a-million chance of a fapo, but someone who happens to (say) live in an apartment that has been laid out very similarly to a scene in pictures appearing in the CSAM database may have a massively higher chance of fapos for photos taken in their home.
Dumb is a pretty accurate description of a large fraction of criminals. For the most part you only get smart criminals when you are talking about crimes where you have to be smart to even plan and carry out the crime.
Decompress and downsample. Drop the least significant bit or two, maybe do it in the dct domain instead. SHA256. It'll preserve matching for at least some cases of recompression and downsampling. But finding an unrelated image that matches is as hard as attacking SHA256, the only false positives that could be found would be from erroneous database entries.
Is there any reading on that? I'd love it to be true.
Yes, because the point is not to protect children. It's to get everyone used to the idea that their content is being monitored. Once that is accomplished, other forms of monitoring can and will be added.
I'd also appreciate if Apple let me know if my false positives were reviewed and found to not be CASM.
How can we be sure they won’t cut costs by increasing worker load? I could see them giving each reviewer less time to review individual pictures before passing it on to law enforcement.
Trolls will be able to easily use tools slightly modify ambiguous adult porn to collide with a "known CP hash".
A human reviewer will see a blurry grayscale derivative of adult pornographic content and hit "report" every time.
False. The Apple proposed system leaks the cryptographic keys needed to decode the images conditional on the match (threshold of matches) of the faulty neuralhash perceptual hash.
Matching these hashes results in otherwise encrypted highly confidential data being decodable by apple, accessable on their servers to the relevant staff along with anyone who compromises them or coerces them.
It's true that in any arms race, a given advance gets adapted to. This will surely catch a bunch of people up front and then a pretty small number thereafter as the remainder learn to avoid iPhones. But that's how arms races work. You could say that about almost any advance in fighting CSAM.
Source: I've met a few white collar criminals.
Apparently that better way is by using Facebook. Facebook made 20.3 million reports to NCMEC in 2020.
https://www.missingkids.org/content/dam/missingkids/gethelp/...
"We found that more than 90% of this content was the same as or visually similar to previously reported content. And copies of just six videos were responsible for more than half of the child exploitative content we reported in that time period."
"we evaluated 150 accounts that we reported to NCMEC for uploading child exploitative content in July and August of 2020 and January 2021, and we estimate that more than 75% of these people did not exhibit malicious intent (i.e. did not intend to harm a child). Instead, they appeared to share for other reasons, such as outrage or in poor humor (i.e. a child’s genitals being bitten by an animal)."
Based on this, I wouldn't conclude that FB is the platform where people pedos go share their stash of child porn.
Their numbers also include Instagram, which I believe is quite popular among teenagers? I wonder how likely it is for teens' own selfies and group pics get flagged and reported to NCMEC.
(https://about.fb.com/news/2021/02/preventing-child-exploitat...)
Which appears to have resulted in what... 5 prosecutions?
Given the reported numbers of illegal images detected by similar systems within Facebook and Google, I think it is very clear that this will catch a lot of illegal content.
So closer to 1/10M. The reporting threshold is made artificially higher by requiring more than one positive.
But anyway, that's beside the point.
A perceptual hash is not uniformly distributed; it's not a random number. Likewise for photos taken in a specific setting; they do not approach the randomness of a set of random images.
So someone snapping a photos in a setting that has features similar to a set of photos in the CSAM database may risk a massively higher false positive rate. It's no longer a million sided dice, it could be a thousand sided dice when your outputs happen to be clustered around similar values due to similar setting.
But I can't say I care about false positives. To me the system is bad either way.
I really doubt this. In the long term, a few people Apple wants to frame will surely slip into the mix. If Apple didn't want Trump to win, a CASM flag a week before the election might do it.
This includes the vast majority of pedophiles.
According to Apple only images that will be uploaded to iCloud will be scanned.
If this is the case there is zero reason to scan locally and you can just scan the uploaded image once it is on the server.
Apple has not implemented E2E nor has it released a statement indicating this will be implemented in the future.
The system is specifically designed so that colliding images does not pose a threat to the user.
NeuralHash and the CSAM scanning is grotesque, but please, criticize it for what it is, not some bullshit that is easily dismissed as technical ignorance.
This algorithm doesn't even give exact matches for the same image on different hardware.
https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX
Note: Neural hash generated here might be a few bits off from one generated on an iOS device. This is expected since different iOS devices generate slightly different hashes anyway. The reason is that neural networks are based on floating-point calculations. The accuracy is highly dependent on the hardware. For smaller networks it won't make any difference. But NeuralHash has 200+ layers, resulting in significant cumulative errors.
According to the U.S. law, key snippets of which are quoted on the Stratechery blog (by Ben Thompson), Apple isn’t obligated to scan for CSAM. It’s only obligated to act on CSAM if it finds them.
While it’s good for Apple to scan on its systems (iCloud) like Facebook, Google and other companies do on their servers, it’s inappropriate to do it on individual devices, which starts with the assumption that anyone who has iCloud photos enabled is a potential CSAM hoarder and needs to pay with their device’s battery life and time for the scanning to happen and report back. It’s a sort of micro-robbery that Apple is doing on the devices when there is no legal compulsion to do so.
Everything else on trusting Apple’s NeuralHash or the sanctity of the NCMEC hashes come later, IMO.
I sincerely hope Apple realizes that it’s got a dud solution on hand, eats humble pie (which it’s usually not capable of) and ditches this whole thing. I know a lot of egos at Apple are at stake here. But doing the right thing matters for a company that claims that “privacy is a fundamental human right” and has a CEO who’s a member of a marginalized/discriminated community and understands the risks of these efforts.
That is not bad. As a tool to filter down what apple human reviewers need to look at this is pretty good.
Ultimately these images will make it to a human reviewer who can make a call as they would in any flagging system.
Could a backend server side system do a more precise hash (96 bits is not a ton) prior to human review?
This system is unwanted because it puts a spy literally in your house and in your hands. It's bad enough that cloud everything blurs the line between what's yours and what's mine. Placing any law enforcement tech on a user's own device takes that line between "public" and "private" and completely erases it.
This alone should be bad enough, but some people are rather trusting. Showing that the spy is also tripping balls both exposes additional risks and emphasizes that Apple neither has their best interest at heart nor is putting adequate care into their actions. The latter gives people reason to question apple's claims of additional protection mechanisms that are non-falsifiable.
It's already a public knowledge that Apple has 2 more systems (some server-side verification and a manual check later) to prevent false-positives. So what's the point of researching collisions in NeuralHash?
Can two collisions really be called a catalog?
Am I correct in that the primary reason folks are so upset is that the system could (probably) be easily modified such that -any- content could invoke legal action? That the main problem is really the scanning at all, and not the chances that it could be attacked by an individual actor but instead by a government?
That said, Nextcloud is my backend and I do not upload anything to iCloud (except for MS authenticator 2fa backups), so I'm safe right?
Yeah two sticks (ski and nail) are visually similar on a white background. Why is this news to anyone?
EDIT: if you are going to downvote please leave a comment unless you are just downvoting for wrong think.
Besides, you could probably "naturally" obtain such type of colliding images by photographing similar-looking objects against a white (or generally featureless) background. Furthermore, it suggests/demonstrates that similar-looking images with similar backgrounds can lead to unexpected collisions in practice (i.e. "naturally"), even if you do not assume an adversarial scenario.
Are you sure that, if you take a picture of a naked body part, it won't collide with anything that looks similar in their database?
It is unlikely that there is a collision of benign image with the database and even if that happens it is not some automatic process that just sends cops to your house to raid it.
Of course we can get bunch of collitions with essentially same images, I don't get why this is so magical just squint your eyes and I'm sure you have two objects with in your reach that could be made to collide, but that isn't a gotcha on any level
At any rate, IANAL, but I'm pretty sure you can't be convicted based on a hash alone. If you get busted for possession of a picture of a nematode and you can show the jury it's just a picture of an axe that has the same value when run through this algorithm, you'll be fine. And there's a decent chance prosecutors won't chase down individuals who will just have a single collision in their photo library with this tech in the first place - people who have dozens or hundreds will be much more interesting.
The more interesting technical question for me is: do collisions transfer across models? or how to find collisions that transfer across models?
But un-natural image collisions or bad images in the database and similar are a different matter and had been the main critique point from the get to go as far as I can tell.
I wouldn't be surprised if some flat, small height fully adult (e.g. 30) woman does some sexting and goes from 0 to >40 collisions in a month. Not because of arbitrary collisions but because the similarity some of here sexting pictures might have with the ones from a 14y old but older looking girl (which e.g. where forced and ended up in the database).
E: Better yet, only run the second hash if you have a collision, which should be very rare.
Whereas if the CSAM scanning was performed exclusively in the cloud, protection under the 4th Amendment does not exist as it would likely fall under the third party doctrine.
Now I'm not saying the US Government would let mere unconstitutionality get in the way of any surveillance program. But Apple would. You don't think Apple wouldn't be itching for another opportunity to flex in public? Especially now, with their reputation on the line? Apple would love nothing more than to have more opportunities like they got with the San Bernardino iPhone.
Scanning makes phones a greater threat, and also erodes the expectation of privacy that is a legal barrier to surveillance.
To all of a sudden introduce this scanner doesn't negate the expectation of privacy as that is how it was sold and marketed. There is an implied warranty of merchantability of how this service functions.
The government can't compel warrantless searches of Apple. 3rd party doctrine means Apple can search your iCloud, and can give it away if they choose. Same as how Apple can search your phone if you run their software, and can give away whatever they find if they choose.
There's no reason not to assume this isn't already happening, being closed source and proprietary. The question to ask is, what are we going to do about it?
In summary, I’m guessing they tried to invent a way where their server software never has to decrypt and analyze original photos, so they stay encrypted at rest.
https://www.apple.com/legal/privacy/law-enforcement-guidelin...
(Note: I have worked with law enforcement in the past specifically on a case involving Apple and two iCloud accounts. You submit a PDF of the valid warrant to Apple. Apple sends two emails one with the iCloud data encrypted. A second email with the decryption key.)
Surely that's just the data, but resized?
I suppose folks who don’t like privacy implications can downgrade to an iPhone 4 and maybe it will not support the feature.
And for the suspicious, it's of course much easier to notice if Apple would change their algorithms if they happen on device.
You’re having a house party. Because of the pandemic, you’d rather people who have COVID not attend. You can’t trust everyone to get vaccinated or get tested beforehand. So, you decided to set up a rapid-test system, just to be sure.
Would you rather test in your kitchen or your driveway?
If contagion wasn't a factor, I'd rather test in the kitchen, it's cozier.
Are you suggesting CSAM will infect more unwilling victims if it gets into a private iCloud account?
If it's a critical part of the system, then it should be inspected thoroughly. If Apple claims a minuscule chance of a hash collision, and the reality is that collisions are relatively common, that significantly changes the requirements for the backend system, which Apple keeps secret. We have every right to believe, bbased oon ppublic info, that Apple was expecting that NeuralHash would be almost fool-proof, leaving the backend system to be a rubber stamp. This would be tragic.
Now, how well this NeuralHash does preserve privacy is a different question, and /not/ one that is being answered by the original post here. In fact, I've not seen anybody look at the hash distribution over natural images, which would be an actual argument against the system.
Are they?
0. Most importantly: the existence of a preimage attack makes Apple's system completely useless for its original purpose. The NeuralHash collider allows the producers and distributors of CSAM material to ensure that nearly all of the next generation of CSAM will suffer from hash collisions with perfectly innocent images. Two weeks after it was deployed, Apple's CSAM scanning is now _only_ an attack vector and a privacy risk. Thanks to the preimage attack, it's now completely useless for its nominal function! Apple put a lot of effort into a system that reduced the privacy and security of all their customers, and made the company itself more exposed to the whims of governments. And for no gain whatsoever.
1. There are no known perceptual hash functions on which preimage attacks are difficult. Barring a major "secret cryptographic breakthrough", Apple's second hash function is not resistant to preimage attacks either. In fact, the second algorithm is almost certainly easier to attack than NeuralHash itself, since it has to work on the "visual derivative", a fixed-size low-resolution thumbnail of the original image.
2. But isn't Apple's second algorithm kept secret, making it difficult to perform preimage attacks against it? No.
First of all, the second algorithm cannot be kept secret. Apple doesn't have its own CSAM database (the whole point is that they don't want to deal with CSAM on their servers!), so the algorithm has to be shared with multiple organizations which do have such databases, so that they can pre-compute the hashes that Apple will match against. Due to Apple's policy, some of these organizations will be located outside the US [1]. Chances are, the hash function will leak: Apple won't know if and when that happens.
Secondly, this _is_ security by obscurity. Some people argue that keeping the hash algorithm secret is similar to keeping a cryptographic key secret. This is not the case. Of course, any security system relies on keeping _something_ secret, but these secret somethings are not created equal. The secret keys of cryptographic algorithms are designed to satisfy Kerckhoffs's assumption. This means that the key, as long as it remains secret, should be sufficient to protect the confidentiality and integrity of your system, even if your adversary knows everything else apart from the key, including the details of the algorithm you use, the hardware you have, and even all your previous plaintexts and ciphertexts (inputs and outputs).
The second hash does not have this property at all. Keeping the algorithm secret does not ensure the confidentiality or integrity of Apple's system. E.g. if somebody gets access to a reasonable number of inputs-output examples, that allows them to train their own model which behaves similarly enough to let them find perceptual hash collisions, even if they don't know the exact details of the original algorithm. This is incredibly hard for cryptographic hashes, but very easy for perceptual hashes, since a small change in the input should cause only a small change in the output of the perceptual hash algorithm. So, to maintain security, Apple doesn't have to keep just the hash algorithm (or its configuration parameters) secret, but all the inputs and outputs as well. This is bad: the fewer and simpler the secrets that one must keep to ensure system security, the easier it is to maintain system security.
Finally, the second hash algorithm is unlikely to be original (NeuralHash was original, and by all accounts it was a massive effort). If an attacker successfully guesses that Apple's secret algorithm H is closely related to a known algorithm, say PhotoDNA, they will probably be able to make a transfer attack against it. By engineering a PhotoDNA collision on the resized thumbnail (e.g. via a resizing attack, extensively discussed in a previous thread [3]), they have a reasonable chance of generating a H-collision as well. How good is fairly good? Well, something like 5% is more than enough! The attacker needs to produce a certain number of NeuralHash collisions (say 30 images) to get through the first threshold of Apple's algorithm. But after that, Apple will decode all the thumbnails in the user's safety voucher: the attacker only needs one of those 30 to get through the second hash. Given a sufficiently high probability of hash collisions, this can be achieved "blindly".
3. It's incredibly easy to come up with these kinds of attacks. Even the HN audience could come up with several reasonable plans, and could point out several reasonable issues, in two weeks. People who do malice for a living will have a much easier time with it. Even if somehow all the plans presented on HN turned out to be unviable, it will not take long for someone to stumble upon something practical. Any reassurance that Apple could provide at this point is fake. Cf. the timelines for real security: it took 17 years to come up with an analogous attack against SHA-1 [4], and two years after that to turn it into something that can be exploited in practice [5]. The existence of a preimage attack made Apple's system completely useless for its original purpose in two weeks. It's now just a security and privacy hole, with no other function. Keeping it around would be a travesty, even if it was difficult to exploit. But it's not.
[1] https://www.itnews.com.au/news/apple-to-only-seek-abuse-imag...
[2] https://en.wikipedia.org/wiki/Kerckhoffs%27s_principle
[3] https://news.ycombinator.com/item?id=28236102
[4] https://security.googleblog.com/2017/02/announcing-first-sha...
[5] https://www.zdnet.com/article/sha-1-collision-attacks-are-no...
Let's set aside the questions of where you got all these hashes to generate collisions with, how you got 30 of these mangled images into your victim's camera roll without them noticing. And let's also set aside whether your victim's device is an iPhone with iCloud Photo Library enabled (and has sufficient storage). I still don't get what these mangled images have achieved, other than giving the manual review team something other than child porn to look at.
Seems to me like it'd be easier to just find actual child porn, print it out, place it somewhere in the victim's house and then report it to the police.
That’s a really interesting attack vector I hadn’t seen mentioned previously.
Most people are talking about the potential for adversarial images to be sent to users. If they were instead injected into the database itself (either by poisoning real CSAM or social engineering) that would have far wider ramifications.
I wonder what the most widely-saved pornographic images are across iCloud users.
If actual CSAM were perturbed to match the hash of, say, images from the celebrity nude leak a few years back and added to the database then thousands of users could be sent to “human review”. Since the images are actually explicit how would the human reviewers know not to flag them to authorities?
(1) Review by apple staff (2) Access and leaking by other apple staff (3) Access by hackers who have compromised their system (4) Access by parties coercing apple/staff, including via national security letters.
All of which compromise the privacy of the user. This matters or the neuralhash comparison wouldn't exist in the first place.
Totally agree that the whole system is grotesque-- but that doesn't stop it also being grotesque in every detail as well. The fact that there are false positives when they easily could have designed a system that had none (at the expense of increased false negatives) shows that Apple doesn't especially value customer privacy even if you accept their vigilante privacy invasion. The fact that it's possible to construct adversarial false positives and that their reports didn't disclose this fact shows they either don't know what they're doing or they're not being honest about the risks (or both).
Apple explained in their technical summary [0] that they'll only consider this an offence if a certain number of hashes match. They estimated the likelihood of false positives there (they don't explain which dataset was used, but it was non-CSAM naturally) is 1 out of a trillion [1]
In the very unlikely event where that 1 in a trillion occurrence happens, they have manual operators to check each of these photos. They also have a private model (unavailable to the public) to double-check these perceptual hashes which also used before alerting authorities.
[0] https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni... [1] https://www.zdnet.com/article/apple-to-tune-csam-system-to-k...
The main advantage of using exact collision is that you can then blind the perceptual hash with a cryptographic hash and avoid any leak of information. (Taking for example sha256 of this perceptual hash won't allow any attacker to get any information on the features from the hash, but if the perceptual hash are the same then the input of the sha256 is the same and therefore the output of the sha256 is the same).
This is important because it alleviates the risks of an eventual leaking the database as Apple never touched and compared sensitive content but only cryptographic hashes of the perceptual hashes.
Some other system like PhotoDNA, rely on a euclidean distance between features being less than a threshold to register a match, which allows to quantify how far the image is from CSAM, but mean that the hash leak some information about the original content.
Accidental Tech Podcast - A Storm of Asterisks https://atp.fm/443
I haven't listened to the follow up episode yet.
I still have zero opinion on this photo scanning kerfuffle. I just don't know enough. Of all the "hot takes" on this issue, ATP's has been the most comprehensive. So appreciated.
https://daringfireball.net/2021/08/apple_child_safety_initia...
1. What does “exact” mean to you in this context?
2. What else is more interesting about a hashing algorithm used to identify things, other than its collision rate?
Ad.2 If hashes are to be matched approximately, not exactly, for example will be considered a match if they differ in less than 3 bits out of 96 then the most interesting thing should be how many collisions you can find if you compare them like that.
Apple's private set intersection which leaks the keys to decrypt the images coniditional a neuralhash match requires an exact match.
They probably didn't realize they got different results on different toolchains/devices, since they target a mono-culture and the whole subsystem shows fairly little careful thought went into it. They could easily make an exact integerized version which would be consistent.
It would still be broken. :)
Not sympathetic to a rival gangster? Ok lets find an innocent victim: not a rival criminal, but an innocent witness who our protag wants to intimidate. Gangster wants to intimidate the witness, but can't get at them, so cooks up a scheme to convince the witness that the police are in his pocket. Exactly as above, causing the police to investigate the witnesses phone.
Another one might be, a certain government wants to identify opposition groups using images associated with them . Apple is not keen to be associated with that, but the government can simply generate fake child-porn (remember, programmatically generated CP is just as illegal) for each image of interest.
I would think surreptitiously placing actual child porn on a rival's phone/computer would be much, much more effective.
Cybercriminals could likely do all this remotely. Phish for apple account login, upload images. Done.
they can either just send the police an anonymous message or set up a child porn web site and have it ‘accidentally’ leak its password database, and make sure your email address is in it.
I would even challenge the justification to do this on servers, unless the data is public. If it's behind a personal login, you might as well consider it personal property/data. I find the distinction of where data is stored not very meaningful.
Allowing things to be searched for criminal content just because it's not in your immediate physical sphere makes no sense. It doesn't work like that in the physical world either. When I send a letter, and it leaves my house, no authority has the right to check its contents without a legitimate reason. Likewise, if I put stuff in a storage box in some warehouse, no authority can search it without a warrant.
Note that I'm talking about personal storage (iCloud, Gmail), not public social networks like Facebook.
Our major privacy blunder was accepting scanning of private data in any context. The fight should be for the absolute privacy of personal data. Where the scanning happens is mostly irrelevant.
It's unclear if their claimed threshold of 30 is before or after the false positives they intentionally introduce. I'm going to guess it's before.
Imagine that you play a game of craps against an online casino. The casino throws a virtual six-sided die, secretly generated using Microsoft Excel's random number generator. Your job is to predict the result. If you manage to predict the result 100 times in a row, you win and the casino will pay you $1000000000000 (one trillion dollars). If you ever fail to predict the result of a throw, the game is over, you lose and you pay the casino $1 (one dollar).
A casino that makes no adversarial assumptions about the clientele could argue as follows: the probability that you accidentally win the game is much less than one in one trillion, so this game is very safe, and the House Edge is excellent [3]. But this number is very misleading: it's based on naive assumptions that are completely meaningless in an adversarial context. Some of the clientele will cheat. If your adversary has a decent knowledge of mathematics at the high school level, the serial correlation in Excel's generator comes into play [4], and the relevant probability is no longer less than 1/1000000000000. In fact, the probability that the client will win is closer to 1/216 instead! When faced with a class of adversarial math majors, a casino that offers this game will promptly go bankrupt. With Apple's CSAM detection, you get to be that casino.
(reposted based on my comment on last week's thread [1])
[1] https://news.ycombinator.com/item?id=28236102
[2] https://blog.roboflow.com/neuralhash-collision/
[3] https://wizardofodds.com/gambling/house-edge/
[4] How to crack a linear congruential generator? http://www.reteam.org/papers/e59.pdf
But I really wonder why you think this is an important objection, do you think a lot of people want to go to the "get flagged for child porn" casino?
[1] https://pseudorandom.resistant.tech/obfuscated_apples.html
Now, I can't really call something I voluntarily uploaded to Apple's servers a "private picture". But that's just a matter of perspective, and I understand that many people would disagree with me on this.
On the natural hash collisions (of which there are two), we have objects of similar shape against a solid background. It seems that a natural hash collision of a CSAM image would be unlikely (or if it does occur, it would be something that perhaps is also an infringing image).
As for the synthetic hash collisions, there are visible artifacts in the picture that, if you compare with the original picture, make the overlay of the original picture on the synthetically generated hash collision obvious. Could people get tricked into downloading memes¹ with synthetically generated hash collisions? Sure, people are idiots. But I'm guessing the majority of folks will look at the picture and say, this is a sh*t picture in this meme and download something else.
1. And that, of course, assumes that meme hosters don't apply similar scanning techniques to what they serve up.
NO. Adversarial preimages can be created that look like perfectly normal images. Please stop repeating this falsehood.
Here are some examples I generated (with a link to more):
https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issue...
Try to find a SHA256 collision.
Anywhere, ever, in the history of mankind.
This isn't for lack of looking. A lot of very smart people have looked for them. If you find one, I bet you'll be eligible for a tenured faculty slot at a good university, if not more. A whole world of secure systems would need to be re-engineered.
Hypothetical collisions of course exist, by the pigeonhole principle, just not in the real world.
You do know that they don't see the whole photo at full megapixel resolution? They're just given "a visual derivative" of the photo for checking.
Also, you really think that the persons tasked with this process are just randos off the street and not vetted specifically?
And where do you get the "visual derivative" information? Apple sure didn't communicate that to me. All I know is some person may look at my pics at some point.
We are not speaking about a situation where not a "arbitrary" picture is miss-classifieds.
We are speaking about a situation where a innocent picture involving a naked or not fully clothed child is deemed similar to a non innocent picture of a naked or not fully clothed child.
Now you might argue that there should not be a picture a a naked or not fully clothed child of any form ever on any phone, but IMHO that is short sighed, discriminating and at best shows you don't know to much about the world and other cultures.
Let's list some simple reasons such a think could happen first:
- Photos meant for a doctor, or living partner to ask if something is normal or a problem. In many different ways.
- Photos of little children bathing or similar a e.g. dad sends to their mom who is currently on a business trip.
- etc.
Reasons people are less aware of is that not all countries are as stuck up about nakedness especially in the family. So it's totally normal for families that e.g. before or after taking a shower family member independent of age and gender walk through the apartment naked. Similar if you didn't got any shame about the naked body indoctrinated you might totally do thinks like visiting a "naked-beach" with your family (meeting other families and taking advantage of it often being less crowded) and in turn normal innocent beach family pictures contain naked children. And on itself that's not a problem. But with Apples approach stuff like this is like to trigger both systems Apple announced and wrongly label your while family as pedophiles...
Baby in the sink? No. But a bunch of the aforementioned? Yeah.
There are two basic reasons for this first it’s a backup service which makes end to end encryption risky, but second they also let users share access to their baked up photos. iCloud > photos > shared album.
How many entries do you think there were when the first live version was announced only a few hours later (http://www.nupedia.com/pipermail/nupedia-l/2001-January/0006... )?
As a different metaphor than your "Octopus", this is "first light".
"First light" in astronomy is the first time a telescope is used. It doesn't need to start with an amazing or ground-breaking image.
To me it's pretty clear they are doing the absolute minimum possible to keep congress from regulating them into a corner, where they lose decision making control around their own privacy standards. The system they came up with is their answer for doing it in the most privacy conscious way (e.g. not decrypting user data in icloud) while balancing a lot of other threat model details, like what if CSAM-hash-providing organizations provide img hashes for a burning American flag, and lots of other scenarios outlined in the white paper.
All I'm saying is that the implementation Apple has described would be constitutionally blocked from being co-opted by US law enforcement. Obviously if there's no end-to-end encryption, any cloud operator could still be coerced into searching for material server side, as that falls under the so-called third party doctrine.
Darknet might sound bit complex, but as darknet user, you literally just install different browser.
We're just in an echo chamber of people who know what JavaScript is, and that distorts our perception of the world.
Reading your comment, I realize how these… ‘criminals’ could use phone number networks to share illegal sexual content peer to peer.
In other words, Apple doesn’t need to analyze your images to find these criminals. They only need to analyze the frequency or quantity of flagged images.
In other words not one image correctly/falsely tagged, but individuals and networks of individuals who are *collecting* and *storing* mass quantities of these images. And, they’re using Apple privacy and security to hide from law enforcement.
Racketeering?
It's a dauntins realisation that sinks in once you have to do support for a web site or app catering to the general population instead of a niche.
They don't read, don't know the diff between an app and a web site, don't know right click or drag and drop, think google or chrome is the internet and overall their startegy to solve any problem is as follow:
- look for something obvious that seems like the answer but is not scary
- click
- wait for it
- repeat 3 times until ok or give up and call someone or get angry or both
Working on a streaming video site really opened my eyes on this one. Most tickets we received were insults, some were incomprehensible garbage, a few were actionable request from someone not understanding anything about their computer.
This is nothing like your github ticket. Your parent number are being generous IMO.
No amount of data center optimization will beat running computations on hundreds of millions of devices other people have to pay for.
I find it quite shocking that a foundational element of criminal justice, innocent until proven guilty and needing a reason to search individual property, is tossed aside like it's nothing.
1. One example of this would be that theoretically, LaTeX's cross-reference mechanism can get caught in a cyclic state. This can only happen with page references and the most likely scenario is a reference to a roman-numeraled page number where if the page reference is output as ix the referenced location moves to page x and when the page reference is updated to x the referenced location moves to page ix (in practice, functioning examples required a shift between xcix and c, but either way, the probability of this happening in a real document is vanishingly small).
“ Only when the threshold is exceeded does the cryptographic technology allow Apple to interpret the contents of the safety vouchers associated with the matching CSAM images. Apple then manually reviews each report to confirm there is a match”
The design goal was no human review for individual matches.
> The only interesting metric is the false positive rate for normal images.
Wrong. The only interesting metric is the false positive rate _in practice_: i.e. how likely are false positives to affect innocents. Indeed, Apple is presenting their "one in a trillion" number as reassurance, as if it was the probability that an account that doesn't distribute CSAM gets flagged by their system. But that probability depends strongly on adversarial questions, and cannot be calculated using optimistic assumptions about all images being "normal".
Timeline:
1. Leaked documents show Pegasus software exploits all iphones using an exploit in iMessage 2. Apple releases security update (doesn’t patch imessage. exploit) 3. Apple announces CSAM client scanning coming soon 4. Apple releases another security update (still leaves iMessage exploit unpatched and used by Pegasus)
….
Perhaps Apple is under pressure to provide a back door prior to patching a tool that may be widely used by governments around the world.
Every piece of data is CSAM encrypted with a one-time pad. It's just that nobody knows the one-time pad.
But I'm unsure that the thumbnail is included with every CSAM "voucher" -- it's likely only included when you pass the 30 image limit. Need to read that section more clearly.
A secret sharing scheme is used to drip-feed Apple the key: each time a positive match occurs, Apple learns a bit more about your key. Once the threshold is reached, Apple will have learned enough to recover your encryption key, and will be able to use it to decrypt all your matching thumbnails at once.
I seem to recall that the white paper speaks of a "visual derivative" without specifying it further.
>The decrypted vouchers allow Apple servers to access a visual derivative – such as a low-resolution version – of each matching image.
https://www.apple.com/child-safety/pdf/Security_Threat_Model...
It's just your friendly trillion dollar tech company putting on a mask and cape and engaging in a bit of vigilante fun. You know? Like batman! ( https://www.youtube.com/watch?v=Kr7AONv3FSg )
The hashes are supplied by NCMEC, a non-profit which is auditable, not a secret government agency.
In any case, even if a non-CSAM hash were somehow in the database, Apple reviews the images before making reports, and those reports are used in normal criminal prosecutions.
Just look at political memes. What is the chance that both sides will have a few people try to create memes for their opponents which are adversarial. They'll take care to make sure those images aren't on their own apple devices before spreading it to areas where the other side likes to share memes.
Another example are the very people trying to be caught. Someone who is against current laws about the subject might seek to create as many false positives as possible to overwhelm the system. They might even specifically target otherwise legal baby photos to manipulate in this way as it would make it more likely they get past any sort of second tier manual review and result in law enforcement wasting resources.
Lastly, this is nothing new. Planting such material and flooding websites with it has long been a tactic used by some. Up until now it has been limited because doing so requires violating the law yourself and few people hate others enough to go through that level risk to self. But this is mostly risk free because creating adversarial images like this isn't outright illegal and even in cases where there are laws against it violating those laws is extremely different than violating actual laws against CSAM in every way that factors into a person's willingness to break laws.
If these new images never make it to the NCMEC database, then new CSAM content will be completely NeuralHash-proof. However, if these images eventually make their way into the NCMEC database, then everybody who has the perfectly innocent originals will be dealing with an adversarial environment.
The remark that an image might not be in the database applies equally to all CSAM pictures, and so is not actually dependent on any technical aspect of the hash at all.
The fact that you can modify an image to the point that it gets a different hash isn't important compared to the (separate) issue of how you keep your database up to date. And detecting old CSAM is in any case not as important as tracking and interdicting production of new images.
The second remark is good, but mainly because it reminds us to demand Apple ensures that additions are scanned for malicious images before being added to the central database. If the operator determines that a certain images collides with some known public image they can notify Apple. Apple in turn modifies the hasher to dissolve the collision, roll out a patch, and can then update the database.
Even then, for that to affect _John Doe_, they would have to make 30+ images whose hash matches that of images in _John Doe’s_ iCloud account.
I think that means they could target individuals, but only if they knew or could guess what photos they have in their account.
They also might be able to target groups of individuals, say people who went on holiday to Paris. It would be interesting to see whether such people have enough overlap in the sets of Neuralhashes of photos they took there.
Perhaps I could even do that without revealing my motives to you.
And human reviewers are in the process. If you've got 30 matches and they are all pictures of bridges or whatever do you think the FBI is going to show up at your house?
If you send actual CP, then that's a wholly different matter.
At least their security analysis relies on that.
From their whitepaper: "The threshold is selected to provide an extremely low (1 in 1 trillion) probability of incorrectly flagging a given account"
If your claim is that their hash algorithm isn't cryptographic, their security analysis is incorrect.
"equivalent to a cryptographic hash" "change... only if the image is substantially different"
Both are not true, cannot be true.
If you downsample and quantize an image before sha256ing it you get a bit of robustness to accidental false-negatives. While both schemes are trivially bypassable.
Dwyer calculated 1431168 NeuralHashes and found two collisions. Humanity collectively calculates over 120000000000000000000 SHA-256 hashes every second. Still, we're reasonably sure that this immense brute-force search will not lead to any collisions in any reasonable amount of time.
They don't say what kind/distribution of non-CSAM images. Landscapes? Parent pix of kids in the bathtub? Cat memes? Porn of young adults? Photos from real estate listings?
I suspect some pools of image types would have a much higher hit rate.
Edit: And, well "hot dog / not hot dog" is impressive on a set of random landscapes too.
Anyway, I suspect that the algo is more likely to pick defining features of the scene and overall composition (furniture, horizon, lighting, position & shape of subject and other objects) more than the subject matter itself.
Understand that matching a file in the NCMEC database is not itself a crime. The whole CSAM-detecting ecosystem is just a tool for surfacing potential crimes. Having a few pics of your own child naked is not illegal and it’s pretty easy for law enforcement to figure out if that’s the case.
Is this a legal statute or simply convention due to the ways things have historically worked (i.e. pre-hash matching at scale)? If warrants are granted based on probable cause, it seems easy to convince a judge that a hash match is sufficiently unlikely that it would exceed the threshold for probable cause. In the context of cryptographic hashes, this is accurate. But if law enforcement doesn't distinguish between cryptographic and perceptual hashes, then there is the real possibility for cases opened and warrants issued unjustifiably.
Sure, matching a hash isn't a crime and you will eventually be exonerated. But as they say, you can beat the rap but you can't beat the ride.
s/will/might/
For now. But what will happen when there are thousands of false positives per day? Will they increase the staff? Or will they add another algorithmic layer? Or just up the threshold a bit? There's no guarantee. The only thing that's certain is that the NeuralHash doesn't inspire confidence.
At most you could maybe temporarily lock someone’s iCloud account. But again, the collisions would need to be multiple and all look like CSAM at reduced resolution.
In general, it seems not correct to think about NeuralHash like SHA or RSA. It’s not a cryptographic system and collisions are not a one-step endgame.
A quick search found four clear cases where law enforcement has favored technological false positives over evidence:
Ousmane Bah: https://www.businessinsider.com/teen-sues-apple-1-billion-fa...
Robert Williams: https://www.cbsnews.com/news/facial-recognition-60-minutes-2...
Nijeer Parks: https://www.cnn.com/2021/04/29/tech/nijeer-parks-facial-reco...
Michael Oliver: https://www.dailydot.com/debug/detroit-facial-recognition-wr...
The one on Ousmane Bah really frustrates me- not only was he on a date at prom during the theft, he was in another state! "Nothing to hide" does not mean "nothing to fear" and allegations (even false ones) of possessing CSAM will ruin lives.
At the end of the day, what it really comes down to is trust; personally, I do not have enough faith in due process to not ruin innocent people. But I'm just some guy online, I will readily admit I don't know anything about anything.
Images from my phone can be stolen and reviewed with no due process, based on proprietary Apple technology.
Not saying it will stay that way, but there are three distinct realms of objection to this system, and it's probably useful to separate them:
1. Objections that in the future, something different will happen with the technology, system, or companies; so that even if the system is unobjectionable now, we should object because of what it might be used for in the future; or how it might change. 2. Objections that Apple can't be trusted to do what they say they are doing, so that even if they say they will only refer cases after careful manual review, or that they will submit images for review that were not uploaded to iCloud, we can't believe them, so we should object. 3. Objections that hold for the system as designed and promised; in other words, even if all the actors do what they say they are doing in good faith and this monitoring never expands, it's still bad.
People who have the third kind of objection need to deal with the fact that Apple is basically putting in a system with more careful safeguards than are already in place in many Internet services, even for their "private" media storage or exchange. You likely don't know how the services you use are scanning for CSAM but if the service is at all sizeable (chat, mail, cloud storage) it's likely using PhotoDNA or something similar.
I think there are valid objections on all three bases. But there's a difference in saying "this is bad because of something that might happen" and "this is bad because of what is actually happening".
For many years, it happened in the cloud. Soon it will happen on device and send a message about which item in the cloud is an issue.
I think it’s all about apple moving ML jobs (like Siri) to device to lighten the load on their datacenters.
They are communicating numbers. For example, they tested with 100 million photos and got 3 false positives. They also tested with 100 k normal porn photos and got 0 false positives.
Basically, that 1 in a trillion number has an implicit "assuming people aren't cheating", as most mathematical models do. But it's already evident people can cheat this system.
I don't know what the odds will end up being, 1 in a trillion or 1 in 100, but they will not be based on statistical analysis. The odds will be based on cultural and social factors... how quickly do Apple reviewers get overwhelmed? How easily can script kiddies use the tools to fake hashes? Are there consequences for false reports? How many people want to get you in trouble?
Or what if a kid gets her/his hands on a phone and takes some pics (doesn't even require unlocking the phone) by accident?
The US may be one of the most up-tight countries about nakedness in general if you ask me. At the same time it's hyper sexualized and produces people like Nicky Minaj, but I guess there is still some fabric over their most "special" parts so it's ok. But oh god, what if Justin Timberlake rips it off... Pandemonium.
No I'm definitely not arguing that. I'm not American, where I live you'll sometimes see nude bathers in the city centre, and most definitely nude children on the beach.
1 in a trillion is derived from a dataset of 100 million photos, presumably a representable proportion of these were "similar images" like bathing kids.
I.e. there can be a massive difference between a probability "over all humans" and a probability "over people of a given culture" as long as either the given culture is in a minority or underrepresented in given data.
Given that people normally don't (knowingly) give out their private family photos when they know they culture is seen as "bad" by some people and this picture might be abused I think we can at least assume such culture(s) are underrepresented.
Through we can't say how much that changes the probability.
People don't seem to grasp what kind of images end up in the CSAM databases. They are most definitely not "leaked celebrity nude selfie" level stuff.
Think of the most vile sexual thing you could do to a child and then times that by two and halve the child's age in your mind. That's the shit that gets in there.
It's not something even 4chan weebaboos share. It's stuff that makes Liveleak regulars go "ewwwww, gross".
I think they can tell the difference between a naked celebrity and a molested child.
How is a human supposed to distinguish that a visual derivative (a low res sobel filtered image, presumably) of ordinary, lawful, adult pornography isn't child porn when the system has already identified it as such?
I agree that using real child porn is an attack too, but at least in that case you could say the system was doing as designed (even though what its doing shouldn't be something that we want) ... but it's not even guaranteed to do as designed.
The way I see it working is Apple scans a ton of shit, some of it shows up as possible child porn, human intervenes and looks at the source images, if there is indeed child porn they report to the police.
"Amazon has people transcribing audio in Costa Rica, India and Romania." according to Engadget. So it would be safe to assume Apple does something along those lines, I can't remember where Siri transcripts have been sent when there was the Alexa controversy.
From there it's an entry with law enforcement and you need to find a way to convince them that the images are your children.
And if data isn't cleared correctly and you have another run in with law enforcement, there will always be this picture-stuff.
It's inconceivable that anyone could desire possession of NCMEC-catalogued CSAM images without being aware that they're risking serious consequences if they're caught. Who wants their deepest, darkest, potentially life-ruining secrets just milling about with photos of the dog and last night's dinner?
[1] ...which is all but impossible for an average user to prove was effective; it's not like the Photos app has a "Not Child Porn!" checkmark.
But if nobody would import CSAM into their icloud library why do all the pictures need to be scanned in the first place? I would imagine anybody doing major illegal stuff being informed about important measures in order to not be caught.
1. Identify some innocuous pictures that many many people have (memes, Beyoncé, whatever).
2. Produce CSAM.
3. Mangle it such that it is still CSAM visually, but NeuralHash-collides with the innocuous pictures from step 1.
4. Distribute.
5. Wait until they are (via some other mechanism) a) identified as CSAM, b) added to the NCMEC database, c) added to the Apple on-device database of blinded hashes in some iOS update.
6. Millions of people are suddenly incorrectly flagged for exceeding the threshold by NeuralHash (since they have the innocuous pictures in their library), and the review teams are flooded and can't pick out the small number of actual CSAM holders.
That is not without a certain elegance. However, it seems to me that
A) it is predicated on the assumption that you can easily mangle pictures to NeuralHash-collide with a desired target picture (out of a set of widely circulating innocuous pictures) without deteriorating the visual content too much.
B) it would be quickly defeated by amending the 2nd tier algorithm (between NeuralHash and human review), though, as you highlight, that might be tricky given that the team working on this presumably only has access to the innocuous false positive collision image, not the (purposefully mangled) CSAM.
Note that this requires no single "desired" target picture. There are millions of popular, innocuous pictures. As long as you can make your CSAM match any one of them without significant mangling, you're good to go. Not having to choose one specific target makes this much easier to accomplish.
You can. Here is an example I created (with links to more): https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issue...
I'm so tired of people suggesting that you can't. Please explain to me why you posted suggesting otherwise.
I've contemplated making some that are also photodna matches, I expect that it's possible. But access to photodna is only through some awful windows tools, and AFAICT people would just keep posting denials even after an example was posted-- so it's not worth the effort at least not worth it just to further the public discussion.
Perhaps it's a prerequisite for deploying end-to-end encryption of iCloud Photo Library and/or iCloud Backups. The latter in particular has remained decryptable by Apple supposedly due to pressure from the FBI. Perhaps CSAM is what the FBI are using to justify their pressure.
Perhaps it's because Apple's team of lobbyists are seeing ahead to future anti-privacy, anti-encryption legislation being justified under the guise of CSAM. If Apple can show that the CSAM problem is already "solved" then such justifications disintegrate.
To the extent that you can say that they're not exactly a government agency, they absolutely have been deputized by the government.
Yes there is. It’s called legal liability. They are not immune to being held accountable for their actions just like any other non-profit. They may be immune from prosecution for possessing CSAM, but they don’t have any kind of immunity for damages they cause through their own actions.
Issues in any other Apple software will not send the police on you. Why would you install a software on your desktop/laptop that is designed to snitch on you, you would need to get some advantage or be forced by some law.
For now I see only disadvantages but please let me know of any real advantage and not speculation
Disadvantages:
- closed software with hidden db can't be trusted, so as a user you will always have a doubt that some non CP images are in the db(Apple always collaborates with governments)
- bugs in this stuff will cause you big problems(we seen in the past how false accusation destroyed peoples life) and we also seen bad actors abusing this kind of stuff.
- this is also clearly a beginning, now that Apple has the capability then even if they were saints a judge could force them to add new hashes, change the configs etc.
2. It’s one thing to rely on proprietary services like Find My or Siri. It’s another thing to rely on a secret server-side app that has the power to destroy your life.
Now, I don’t trust them anymore.
Now apple is getting a local client side scanning tool ready. Interesting timing.
It could also happen that you lose your phone and "a human" finds it and randomly puts in the correct passcode on the first try and visually inspects your personal images. In fact, that seems vastly more likely [1].
[1] About 4% of smartphones are lost or stolen every year [https://www.mcafee.com/blogs/consumer/family-safety/almost-5... ], but make it just 1/1000, so 1e-3. Then a 6 digit passcode, 1e-6, so we're at 1e-9 per year, or 1000x as likely as being falsely flagged, assuming Apple's numbers (which can easily be achieved by calibrating the threshold).
Apple now has the ability to encrypt the images before sending them to icloud, with a private key you own. Except that some percentage of images that match the CSAM fingerprint with their neural feature extractor will be sent to a CSAM filter on the server side (whose workings we don't have many details about)
This whole thing backfired on Apple entirely due to psychological effects, not because they are really doing anything more "panopticon" that they would already able to do now on their icloud storage (after all people are ready sending their photos to apple)
Therefore, why trust any of their other claims?
Just because someone has found an image of a nearly featureless diagonal thing which collides with another image of a nearly featureless diagonal thing, that doesn't disprove Apple's claims.
Given people can now generate images that collide, it seems like the statistical likelihood has drastically changed since it was originally announced.
So again the actual argument becomes: what is that distribution like?
I doubt that's what they did. I think they ran tests on huge numbers of pictures, got an estimate, put in a safety factor, and determined the threshold to hit their target (and put in another safety buffer then).
Naturally occurring collisions are not going to be an issue, and adversarial ones neither, I predict. Just as with current cloud providers.
Apple never made a false claim. They have never anywhere stated that neuralhash makes false positives at 1 in a trillion. Only that that is the rate for the system as a whole to flag accounts for review. The explicitly mention that they will vary the number of matches needed to maintain this if it turns out to be higher or lower based on images in the wild.
There are good arguments against this system but most of the technical debate seems to have devolved into amplifying lies now.
Still. Why trust them after that?
If a company can make my own smartphone report me to the police, and they want my business, they better prove I can trust them. Apple has plainly done the opposite.
The whole ordeal is just utterly 1984.
In contrast, a NH system believed to have a collision chance of 1 in a trillion trillion may well be considered infallible, and any detection be directly reported as CSAM, with the 'backend verification' amounting to nothing more than a rubber stamp.
Of course, if you implicitly trust Apple not to do the second, than you're right, the NH collision rate doesn't matter too much.
a) I didn't suggest otherwise, I said that it is predicated on that assumption, about which I was undecided, largely because b) I didn't know better.
I read that thread 7 days ago, when the collisions were a gray blob or a clearly modified dog (to Lena) or clearly modified Lena (to dog). I hadn't re-read the thread in the last 4 to 5 days, when you demonstrated the natural looking collisions (second-preimage images).
Very impressive work I wasn't aware of.
Apparently they have not - They say facebook reported something like 20m images in a year, and Apple reported 250.
Eroded trust, sure, but that is mostly because they did a terrible job communicating it.
The whole point for this is to be a probabilistic filter, so that they need to run the real CSAM scanner on a subset of files.
You can fall into two camps:
a) apple should never ever scan my private images I upload on their cloud. b) apple can scan the images once they reach their servers.
If you pick (a), then clearly neuralhash shouldn't exist and you can argue against that on the ground that you want utter privacy. But you have to be consistent:every other cloud service that does scan the images server side should receive the same critique.
If you pick (b), then you must recognize that this additional machinery doesn't increase their reach to your private data, but quite the opposite, it allows them to implement e2e encryption for 99.9% of your content. You may argue that it's unnecessary and confusing and spooky and be afraid of the slippery slope precendent for other uses.
FB and Google will exhaustively analyse every single facet of your online presence and use your pictures to train their ML models for face detection and object detection.
Apple, on the other hand, even explicitly splits Map directions to segments so that they can't know where you left from and where you are going to.
Anyway how is your assumption make sense , Apple cares about children and about your privacy so scanning your images in iCloud was wrong until 2021 when something changed, what changed? does Apple cares more about children starting from now or they care less about privacy? or are they forced to do it?