What Are Tokens in LLMs?(bearisland.dev) |
What Are Tokens in LLMs?(bearisland.dev) |
/s
And, more obviously, the fact that GPT-4 is being directly named even though that model is over 3 years old by now: "Ask GPT-4, Claude, or Gemini today and they will usually answer three.".
Sorry, I just think that the article wasn't produced by a human at all.
> The chunks aren’t characters and they aren’t words. They’re something more specific, and the specificity matters more than most people realize.
> Those numbers are real, but they hide what a token actually is.
> GPT-4’s vocabulary isn’t Claude’s. Claude’s isn’t Llama’s.
> The model never sees text. It sees a sequence of integer indices into its own private alphabet.
> So tokens aren’t “roughly like words” or “kind of like characters”. They’re the atoms of perception for one specific model, and they’re the only language that model speaks.
> The same sentence is nine tokens to GPT-4 and seven tokens to Llama 3. Not because Llama is smarter or the sentence changed, but because the two models have different vocabularies.
> That’s it. No clever scoring, no neural network.
Could people who use LLM to write articles at least prompt them to have a better style? I'm really tired of the default Claude style (a lot of Chinese models also reuse the same style)
What did you think about the more visual elements?
Simon
The underlying BPE algorithm, which is the main focus of this article, is the one used modern tokenizers today.
> The fact that GPT-4 is being directly named even though that model is over 3 years old by now
That is fair. Will be updated
> Sorry, I just think that the article wasn't produced by a human at all.
While I have used LLM to help me write and explain my content, my hopes is that most readers does not share this opinion of yours. Everything touched by AI is not slop, and I wanted to share the notes I created for myself.