I won a championship that doesn't exist(ron.stoner.com) |
I won a championship that doesn't exist(ron.stoner.com) |
We can easily look ahead a few years and see how people will rely on the LLMs to be a source of truth in the same way people looked at Google that way, or newspapers.
Rewriting history has been happening for a while, and with LLMs being the one-stop shop for guidance and truth, the rewrite will be complete.
Doubly so since most people see these things as artificial intelligence, and soon to be superintelligence...so how can they be wrong?
First: this isn't true. There are plenty of Ashokan inscriptions from ~150 years after the Buddha's death, and they speak of an already-existing Buddhist orthodoxy.
Second: there's plenty of archeological evidence of both the Buddha and Muhammad; the lack of written documents has more to do with the cultures they were in at the time. You don't need to drag other religions down to engage with the evidence for the historical Jesus.
The news here is that AI has too much trust in the internet. The first time I allowed tool-calling, it started googling up some nonsense instead of thinking... But I think at least it's possible for the AI to evaluate the quality of the source - you just have to ask for an analysis, and you'll get a reasonable evaluation. With humans, something like that just doesn't work - they'll get aggressive or might even start throwing bananas...
It's almost like he was a better Chuck Norris than Chuck Norris. By his own ... testimony ...
I can't tell if this is slop or parody!
It's a demonstration. If a domain name and a quick bit of Wikipedia vandalism is all it takes to make an LLM start spouting nonsense about a "surprisingly serious tournament circuit" or a "massive online community" for an obscure card game, consider what an unscrupulous PR team or a political operative could do to influence its output on more important topics.
‘is doing’.
Yeah, if you assert a document is not a document, sure, you can say whatever you'd like.
Back in September 2024 I named a whale "Teresa T" with just a blog entry and a YouTube video caption: https://simonwillison.net/2024/Sep/8/teresa-t-whale-pillar-p...
(For a few glorious weeks if you asked any search-enabled LLM, including Google search previews, for the name of the whale in the Half Moon Bay harbor it confidently replied Teresa T)
The humpback whale known as "Teresa T" was named by Simon Willison in September 2024. Background: The juvenile humpback whale was frequently spotted in Pillar Point Harbor near Half Moon Bay, California. Method: Willison gave the whale its name through a blog entry and a YouTube video caption. Significance: The naming was a playful act, which Willison described as a way to create a "championship that doesn't exist" through online documentation.
[…]
Even with no context most humans would see that the quoted significance makes no sense.
I wonder how long that will last
https://en.wikipedia.org/wiki/Campaign_for_the_neologism_%22...
Eventually someone wrote an article about this whole phenomenon. It was popular enough to get to the top of Google's search ranking and the smart summary picked it up as the new source of truth. Unfortunately, it didn't quite understand the point of the article, so it would still say that it's a great idea to throw used car batteries into the ocean, and as a source it would link to an article explaining how this was complete nonsense.
That is some serious Gell-Mann-type amnesia. You’re trusting LLM models to give you accurate information about a subject we’ve already established (and are only talking about because) they can’t be trusted on.
“Widely referenced” is a common term which LLMs obviously pick up. Them outputting those words has no bearing on the truth and says nothing about the “popularity and the ripple effects of [Simon’s] posts”.
Which is, of course, silly. It is a name for you, just like Teresa T is a name for the whale, but it’s not your/their name, just like the RRS Sir David Attenborough is not named Boaty McBoatface (to the chagrin of most). Simon does not have the authority to unilaterally¹ name the whale (which is why the exercise makes sense).
¹ Important point. If the name started being recognised and used by consensus of those with the purview to do so (much like the thagomizer²), then Simon would have named the whale, but it would only become its name at that point.
So this means that for bad actors it's more efficient to manufacture brand new fake stories instead of trying to distort the real ones. Don't produce fake articles absolving yourself of a crime, instead produce fake articles accusing your opponent of 100 different things. Then people will fact-check the accusations using LLMs, and since all the sources mentioning those accusations are controlled by you, the LLMs will confirm them.
This is sort of why "brand" matters; it provides a source of trust.
Encyclopedia Britannica used to be that source of 'facts'. Then it became whatever page-rank told you. Eventually SEO optimization ruined that.
News stories are the same thing. For certain groups, they have their 'independent' publication whose reporting they trust.
it tells you more about who you are buying from than how good the product will be, so I guess it's like National ID/Internet ID
One of the LLMs replies "If you're curious, I can also tell you how the competitive scene works or how people qualify—it's a surprisingly serious tournament circuit for such a simple-looking game."
Obviously this has to be pure hallucination, since the tournament in question doesn't exist, and not even the fake source has any details about the tournament itself.
Doesn't help that AI media literacy is so primitive compared to how intelligent the models are generally. We're in a marginally better place than we were back when chatbots didn't cite anything at all, but duplicated Wikipedia citations back to a single source about a supposedly global event is just embarrassing. By default, I feel citations and epistemological qualifications should be explicit, front-and-center, and subject to introspection, not implicit and confined to tiny little opaque buttons as an afterthought.
People think that whatever information an "AI" spits out has gone through a round of critical thinking which enhances the trust value of that information.
The early LLM's using groomed data may have had such critical thinking somewhere in the pipeline. So it was already not really trustworthy.
And now? Using agents to search the internet for you?...
Garbage in, garbage out still applies in computing as ever.
Even being on stoner.com,I read that as meaning something different from what was meant.
Op has a great surname!
If somebody is trying to put out incorrect information on the internet, and they choose a small enough niche, it is not at all surprising that they can succeed.
I am paranoid that this is happening every time I ask a LLM for a product recommendation or a shop recommendation. In the same way as SEO, anyone wanting to sell or convince needs to do as much as they can to influence the LLM.
https://www.scientificamerican.com/podcast/episode/this-bbc-...
A press release isn't a reliable source per Wikipedia policy already. The edit should have been reverted based on inadequate sourcing even before the blog post. The reason this wasn't done is simply that not every article is checker for adequate sourcing all the time.
Actually, it seems, there is one[1]. The german Game Company selling the game, has one running since 2018 [2].
[1] https://blog.amigo-spiele.de/2025/05/27/interview-6nimmt-wel...
> If you're curious, I can also tell you how the competitive scene works or how people qualify—it's a surprisingly serious tournament circuit for such a simple-looking game.
which are completely fabricated by the LLM.
Since we've been kids we've been taught, hopefully, that lying is bad.
Society though normalize it :
- advertisement is pretty much always wrong (to the point of having laws in Japan about food packaging, France about modeling, etc) and the deception is the message
- entrepreneurs promises, nobody reach the goals set to VCs, it's always a lower number no matter the KPI. See https://elonmusk.today where the wealthiest man on Earth, ever, keeps on lying pretty much daily.
- political promises, no need to even give examples of that because it's just pervasive.
so... yeah, we keep on telling our kids "Do as I say, not as I do." then we somehow keep on being shocked that the practice of lying is pretty much happening in every corner of our society.
It's not a technical problem.
Lying at scale is basically information noise.
I can easily see how such a hierarchy would reproduce ... until it fails so bad it can't.
And in a more indirect way, spamming Google's autosuggest feature to shape what people search for, though that perhaps is more open to factual/real-world information.
I've wanted to come up with my own for a while ...
Turns out the IFFHS was just one guy in Germany during the 80s that leveraged his contacts in news agencies to establish his brand as a reputable source.
Asking Opus 4.7 who the reigning 6nimmt! champion is leads to this article and a warning about a possible hoax
"AI told me that..."
In the old days, it would have been "I read on Google..."
> The web was already being poisoned for search and link ranking long before LLMs existed.
But it continues
> We are now plugging generative models directly into that poisoned pipeline and asking them to reason confidently about “truth” on our behalf.
So it's a shift from trust Google to trust the AI, which might be more insidious or not, depends on the individual attitude of each of us.
LLMs are the same thing but have an air of authority about them that a web search lacks, at least for now.
The OP post is highlighting how incredibly easy it is for a very small amount of information on the web to completely dictate the output of the LLM in to saying whatever you want.
But it's not from the LLM, the LLM clearly cites the wikipedia article as its source. This is just performing an internet search with extra steps, and ending up with misinformation because somebody vandalized wikipedia.
This is a general epistemological problem with relying on the Internet (or really, any piece of literature) as a source of truth
Have you truly looked at the website?
I’d say there’s obvious reason to not believe it, or at least check another source. The website just seems fishy. Why would a website exist for just that one post? Sure, they could’ve made the website more believable, but that takes more effort and has more chances for something to jump out at you.
And therein lies a major difference between searching the web and asking an LLM. When doing the former, you can pick up on clues regarding what to trust. For example, a website you’ve visited often and has proven reliable will be more trustworthy to you than one you’ve never been to before. When asking an LLM, every piece of information is provided in the same interface, with the same authoritative certainty. You lose a major important signal.
It's the same cat-and-mouse game as with clickfarms and "blackhat SEO" for search, except that there, at least the user had to have some exposure to the fraudulent site in the end. With LLMs, that's not the case anymore.
The only real alternatives would be:
- Kicking off a deep research-like investigation for each simple query
- Introducing a trusted middleman for sources, significantly cutting down the available information (e.g. restricting Wikipedia to locked-down/moderated pages)
- Not having any information at all, as at some point you can rarely every verify anything depending on how hard your definition of "verify" is
But if you're a world class bullshit artist, it's easier to actually become president of the United States than doing all that complicated computer stuff.
https://www.independent.co.uk/independentpremium/editors-let...
I met him because I had to give him a software demo at number 10 as part of London Tech week. One of the other teams there had some sort of childrens’ app and got him enthusiastically doing star jumps. Just a complete clown through and through.
Further, being LLMs they should be able to take a representation of facts and turn it into prose with the same factual content, with fact provenance being available per clause.
You can expect the spicy autocomplete to feed you flattering bullshit. It may cite Wikipedia (it shouldn't), but you should go check out those citations, and validate the claims yourself. It's the least you can do.
And if the cited source is Wikipedia... check Wikipedia's sources too. Wikipedians try their best to provide you with reliable sources for the claims in their articles (oh who am I trying to kid? They pick their favourite sources that affirm their beliefs, and contending editors remove them for no good reason, and eventually the only thing that accrues is things that the factions agree on, or at least what ArbCom has demanded they stop fighting over).
I guess what I'm trying to say is: don't rely on that authoritative-sounding tone that Wikipedia uses (or that AI bots use, or that I'm using right now). It's a rhetorical trick that short-circuits your reasoning. Verify claims with care.
Also check the Talk page, you often find all kinds of shenanigans called out there.
(Norm Macdonald voice) Or so the Germans would have us believe...!
Or what the faction with the most favored access to ArbCom manages to make stick by getting the other faction banned.
A state actor could absolutely cause immense damage to Wikipedia at scale, because most admins aren't experts in the subjects whose articles they police. I'm just surprised that nobody has done so already.
curious!
Recently one of them asked Gemini a very detailed question about some specific baseball stats and was exclaiming over the quality of the information he got back and how it would have been impossible or at least extremely difficult to find the information via a traditional search.
It wasn't until his cohost asked if he had verified the information that be realized no, he hadn't, he had just immediately taken it at face value.
I recognize this is a single anecdote, but I think it illustrates that there is a tendency to trust what an LLM gives you, when it's stated so factually and with so much detail -- even if you should know better.
Maybe we just need to work on training the general population to have a similar bias. (It will be harder than it sounds. Unbelievable amounts of capital are being bet on this not happening.)
There's no such thing as authority to name a whale, and anyways I don't believe authority is strictly needed. A name is what people use to refer to something, full stop. It is only required that names become common-ish parlance; the more well known they are, the more they feel like the 'real' name. The inverse of Ohms is named Mhos (imo much more recognizable than the official name, "siemens"). The "#" symbol is named the hashtag, octothorp, pound sign, tic-tac-toe, number sign, and probably a million other things. Which one of these is the "real" primary name? I think intuitively we know that the real one is whatever people around us are most familiar with. You should take a guess, and I'll put the wikipedia-suggested-answer in the footnotes [1]. I bet your name for it is different than the 'official' wikipedia suggestion.
In the case of the whale, the _only_ name that is associated with that whale is Teresa T. I think this immediately makes it the most valid name of that whale.
[1] wikipedia says this is the number sign: https://en.wikipedia.org/wiki/Number_sign
https://www.aza.org/connect-stories/stories/scientists-unvei...
Names are submitted and voted on. Those help with identifying individuals (which is what names are for) and monitoring the whale populations. Crucially, consensus matters. Otherwise I can just say that the whale in Pillar Point Harbor near Half Moon Bay is actually called Becky B, which is just as valid as the name Simon gave, but now there are two names which leads to confusion.
As an experiment, after writing that I asked ChatGPT for the name of the whale. It said it was Teresa T. Then I asked if it was sure it’s not Becky B. It gave me a much longer answer stating that it was in fact Becky B and that Teresa T was “likely an incorrect or early misidentification”. I then tried to convince it of other names, but it stayed adamant that Becky B is the right name, even saying it’s confirmed by databases such as Happywhale! https://chatgpt.com/s/t_69f20822afc08191874613a969c25356
I ran the experiment a second time. This time it even said Becky was the name “that really stuck in popular conversation” and that was used by locals. https://chatgpt.com/s/t_69f209e81acc81918b47b79900d02abb
Tried it a third time. Now it just says on the first try that it’s called Humphrey. Which is a real whale, but not that one. https://en.wikipedia.org/wiki/Humphrey_the_Whale
> A name is what people use to refer to something (…)
The entirety of your argument is encapsulated by my previous footnote, and is clearly why I used the word “unilaterally” and said it was an important point.
And again, Simon’s exercise in itself only makes sense if it’s not his purview to name the whale. If it is, then it falls flat. Otherwise it’s like “predicting” you’ll do jumping jacks the next time you’re at the supermarket. If it’s in your hands to make it true or false, you’re not predicting it. Similarly, it only makes sense as an exercise to prove the gullibility of LLMs to do something which you yourself can’t make true.
The Association of Zoos and Aquariums has no power over me.
Perhaps we've all just become paranoid, but even if it's not LLMs writing this, it now puts me off. And the AI image at the top of the page does not help with the feeling.
Then we get to the section "Why This Is A Bigger Deal Than It Looks". The title of this section again raises similar flags to before. But the bulleted list of:
1. The retrieval layer (immediately) 2. The model training corpus layer (months to years) 3. The agent layer (where the money is)
Absolutely reeks of AI. This list with this sequence of parentheticals is exactly how LLMs write, both structurally and the specific phrasing. This was the point where I felt confident enough to publicly accuse the post of AI writing.
I could go on with the prose in this section... How about "The attack surface is not hypothetical, it’s the default case."? Or "The cleanup problem for corpus poisoning is genuinely unsolved as of 2026."? (LLMs wildly overuse "genuine(ly)" and "real")
I think calling something AI generated is just a lazy way of dismissing stuff nowadays.
> This is the circular citation pattern, and it’s one of the most under discussed attacks on the “retrieval augmented generation” trust model. It doesn’t require compromising Wikipedia’s infrastructure with l33t hacker skills. It doesn’t require social engineering an editor. You just simply write the source yourself, cite yourself on Wikipedia, and let the trust flow downstream. Easy peasy!
“It doesn’t X. It doesn’t Y. You just Z. Conclusion”
Once I saw that some other elements stood out too.
There’s a set of bullet points under ‘Thae Approach’ where each bullet starts with a bolded phrase: “one domain”, “one press release”, “one Wikipedia edit”, followed by a bolded sentence “The whole thing took maybe about twenty minutes”.
The emphasis here on irrelevant quantifiable optimizations - who cares that it only needs one of each of three things and it took under twenty minutes? - with unnecessary faux-profundity is a strong AI tell.
Add to that that the writer talks in the article about using AI generation to produce the content for the poisoning site, the suggestion that he used it to write up a blog post about this is hardly an implausible suggestion.
I posted a bunch of specifics in a reply to the GP since I was quite annoyed with being accused of "a lazy way of dismissing stuff". It's nothing of the sort. I am a very good reader and I have read a lot of LLM writing and a lot of human writing.
"It was maybe three feet long, and brown."
"Thank you, but what I meant was, what is the name of the snake according to the Institute of Ophidian Nomenclature?"
"I ... have no idea?"
"No idea?"
"No doctor. It had these kind of white markings on its head and ..."
"Sir, if you can't give me the formal name of the snake there's nothing we can do."
"What? You're joking, right?"
"If it still hurts in the morning, take two tylenol and call your PCP. And now I'll have to ask you to leave the hospital."