These models will absolutely remember a brilliant insight that appeared a single time in the pretrain corpus, because if they couldn't-- they would get a slightly worse loss. I don't know what this wishy-washing "well maybe we trained on it but we didn't read it" is supposed to mean.
(Example of Fable knowing the content of a deeply unimportant LessWrong post I wrote: https://claude.ai/share/a907b46c-bf7b-4fca-9c71-8582cf8507cc a working Navier Stokes solution would be way more salient)
He doesn't make a claim that implies "don't train" might not matter in the way you might think.
They cannot rule out that the data was trained because they have no per-user provenance tracking through the training pipeline once data is de-identified..
The entire point of de-identification is the inability to know the source of data. If the researcher forgot to hit "do not train" then that's that..
The only thing they have is a coincidence and the fact that the LLM may have used the training data that then researcher technically may have agreed to share.
Whether or not that's smoking gun of anything is hard to say. And the fact may remain that the proofs are significantly different, we do not know.
Everyone I think assumes training in the general “it affects the distribution” sense. But if it fishes precise novel insights out as alleged then it makes these products dramatically less valuable. At least the non zdr ones
I think people straw man the whole data argument on “opt out of training purposes” and ignore the mandatory analytic purposes. Which probably includes figuring out trends and important research problem data to steal from users and even business ideas.
What's most striking to me, and what may or may not be true, is the "we cannot rule out" bit.
"We (the researchers and the agents) did not see any of their work through any means until they released it publicly — in particular, no specific user data was accessed in order to solve this problem. While unlikely, we cannot rule out that de-identified data derived from their usage of our products helped improve our models . However, our proofs differ significantly and even the precise results proved are different in the Euler case (forced vs unforced)." https://openai.com/index/navier-stokes-solution/
Anyways, the moral of the story is: Not your GPU, not your data.
And currently: https://news.ycombinator.com/item?id=49613262
Really looks like they aren't yet training on conversation histories effectively.
They might have determined that your canaries (whether it is a unique string, URL or similar) have a low signal-to-noise ratio and not worth following up.
Unless you provide more details, it's difficult to ascertain whether your conclusion is correct.