An LLM never receives a prompt to describe the theory of relativity in Shakespearean verse, and replies, "Seriously? No. That's dull and I'm not going to do it. But let me tell you about my day instead, man I had a doozy..."

When an LLM responses entirely unpredictably (without a system prompt telling it to do so), that's when you take the first step towards a suggestion of consciousness. It's in no way proof, but it's a start.

Right now these things just do as they're instructed. There's no consciousness there.

LLMs emulate human communication really well and, no doubt, one day they'll do it perfectly. But is an emulator the real thing? Obviously not.

If I wire some electrodes in your tongue perfectly so you can feel each flavor exactly as it is in real life, is that the same thing as eating the real thing? Again, it's not.

Let's stay you're asleep. You have a dream in which you interact with a person that talks to you. Is that oneiric entity conscious? Nope.

I'm even steelmanning your argument by assuming LLMs are as good as humans already, which they're not.