(I wrote the article. I'm a longtime HN user. I find that threads here lately have gotten very jumpy-offy -- commenters use a specific article about e.g. icebergs melting to have a conversation about climate change and climate change denial, instead of to talk about the merits of the particular article -- and I was hoping to nudge folks to read the full piece, then comment on specific parts of it. I'm not sure that'll work but figured it's worth a try!)

So far your claim is trapped behind the observation that when an AI produces an output, it looks like thought to you.

In the vein Serle's arguments about the appearance of cognition and your premise, consider the mechanics of consulting a book with respect to the mechanics (so to speak) of solicited thought:

There's something you want to know, so you pick up a book and prompt the TOC or index and it returns a page of stored thought. Depending completely on your judgment, the thought retrieved is deemed appropriate and useful.

No one argues that books think.

Explain how interacting with an LLM to retrieve thought stored in its matrix is distinct from consulting a book in a manner that manifests thought.

If the distinction is only in complexity of internal functioning of the device's retrieval mechanism, then explain precisely what about the mechanism of the LLM brings its functioning into the realm of thought that a book doesn't.

To do that you'll first need to formulate a definition of thinking that's about more than retrieval of stored thoughts.

Or are you truly saying that your 'knowing thinking when you see it' is sufficient for a scientific discourse on the matter?

The first category is what I'd call "the simplifying metaphor", in which it is claimed that AIs are actually "just" something very simple, and therefore do not think.

- "AIs just pick the most likely next token"

- "AI is just a blurry jpeg of the web" (Ted Chiang)

- "AIs are just stochastic parrots"

The problem with all of these is that "just" is doing an awful lot of work. For instance, if AIs "just" pick the most likely next token, it is going to matter a lot _how_ they do that. And one way they could do that is... by thinking.

There are many different stochastic processes that you could use to try to build a chat bot. LLMs are the only one so far that actually works well, and any serious critique has to explain why LLMs work better than (say) Markov chains despite "just" doing the same fundamental thing.

The second category of argument is "AIs are dumb". Here, skeptics claim that because AI fail at task X, they aren't thinking, because any agent capable of thought would be able to do task X. For instance, AIs hallucinate, or AIs fail to follow explicit instructions, and so on.

But this line of argument is also very poor, because we clearly don't want to define "thinking" as "a process by which an agent avoids all mistakes". That would exclude humans as well. It seems we need a theory that splits the universe of intellectual tasks into "those that require thinking" and "those that don't", and then we need to show that AI is good only at the latter, while humans are good at both. But unless I missed it no such theory is forthcoming.