The real prices of frontier models(playcode.io) |
The real prices of frontier models(playcode.io) |
Chattiness remains an open issue for some of the SoTA open weights & (to a lesser extent) Claude.
- A ~2000-2002 legacy C++ game codebase at about ~90kloc: GPT 1.12M, Claude 2.2M
- A ~30kloc TypeScript codebase: GPT 260K, Claude 437K
In the end, GPT's current tokenizer is ~1.6x-2x better than Claude's current one, depending on your data. And you can check for free for both, for OpenAI just use the open-source libraries, for Anthropic - you have to use their count_tokens endpoint as they don't publish the tokenizer, but the endpoint is free (and allows requests over 1M tokens as well).
That doesn't really appear to be the case as GPT and Anthropic models appear evenly matched despite Anthropic encoding the same text into almost ~2x the tokens...
I'd also - naively - assume this would make training their models more expensive. Though inference now dominates, and they'd probably rather have more tokens than less (to charge you for them at future 80% margins).
the Anthropic tokenizer is not worse, its more expensive/verbose
Today, I tested Sol 5.6 on various tasks. It performs similarly to Opus 4.8 but is still noticeably more expensive than Sonnet 5. Although Sonnet 5 isn't the top model, it's quite effective for creating typical websites for small and medium businesses. However, they will increase the price starting September 1, as their free offer is ending.
I'm also actively testing Grok 4.5. There's something promising about it. The design is mediocre, in my opinion, but it operates quickly and reliably without any deadloops. Usually, Grok models would fail or loop, but this one is stable.
Overall, I really want a benchmark based on real tasks.
In practice though as a result of cache reads over multiple turns you will end up paying quadratic pricing anyway.
Most people here probably don't know what it was like to work a contract job and being paid based on actual deliverables.
The incentive of AI companies is to create as many tokens as possible to solve any given problem. Just like your incentive as a software engineer is to create as much complexity as possible in order to use up as many hours as possible.
This is why big tech companies have millions of lines of code... They've got thousands of engineers rapidly churning out tokens.
The difference in number of tokens I use in my day job vs side projects is massive. You can see the inefficiency quantified.
Show me the incentive, I show you the result.
So yes, "big tech companies" often paid hourly, even if that pay was indirect, to contractors and job shoppers and people who were not direct hires.
I wouldn't say I'm doing anything groundbreaking but definitely at times obscure and that's when Fable has been able to dig me out of the rut. (the alternative I was actually following was reading textbooks myself to understand the domain better)
Opus's verbosity is actually a boon sometimes for catching false starts early.
On one hand, the price is just astronomical for Fable, well, not exactly astronomical, but I would say unaffordable. That is to say, so expensive that it is impossible to use.
But on the other hand, Fable is simply incomparable to anything else. I mean, it is just amazing. There is nothing even close to being equal to it.
> You will see people claim Claude uses 2x to 4x the tokens of GPT. Our measurements do not support that, and overstating it would undercut the real point.
It's not because a single prompt represents only 1.7x the number of tokens that a model doesn't use 4x as many tokens as another, when running as an agent. This doesn't take at all the number of tokens of the output into account, and the number of tokens of the potential tool calls from this output, which directly feeds back to input tokens.
The article also has a very small test set (16 documents), all of very small length (15K tokens at most, when models go up to 1M in context and agents routinely exceed this and have to summarize).
Complete garbage article.
My guess is something to this effect was in the prompt and the LLM made a point of correcting it
Other traits where models differ that have an even greater impact on your total spend:
* How much context do they load in to solve a given task?
* How long do they spend thinking to get equivalent results?
* How many times do they stop and ask you for input, and are you there to respond to them before the cache runs out?
* Etc.
Incorporating the tokenizer just makes a very imprecise measurement of cost a little bit more precise, but in my own experience I have not found that the token cost is a significant driver of task cost whether or not you incorporate the tokenizer. Everything else about the model's behavior has a much larger impact.
Honestly, haha
Of course, these are my guesses, but did anyone feel the difference in the transition from Opus 4.5 to 4.6? In my opinion, no. And it's unlikely to be a matter of the tokenizer.
Providers change tokenizers all the time with model updates, and it's often not even possible to query/figure out how text is tokenized without actually just sending the LLM a request.
Just switch to charging for bytes of intelligence. Please. Claude Shannon figured this out decades ago.
The best way to measure is really the end-2-end cost, price per task.
The reason being is that the only tokens I feel I really control are the input tokens, but the whole program seems to just run itself and they just charge you what they want to charge you and it’s more of a black box.
Very interesting article though.
This space can be increasingly avoided by becoming, and remaining, efficient and effective with prompts.
Regardless, it is cool to be able to contextualize the actual spend in terms of physical energy utilization. It even has a little co2 number (though again, kind of a "trust me bro" metric).
How we drive AI will cause mileage variances, but over time the improved practices can measurably change.
I find my brain disengages once I suspect something of being written by an LLM. If the author didn't put much effort into writing it, should I expect them to have put much effort into fact-checking it?
Edit: this specific title has been deleted from the article. That was not my point! Please put in more effort into writing things that you want others to read! Rather than putting in low effort but being better at hiding it.
"Authoritative: it is the same count Anthropic bills against."
"This reframes a headline that looked like good news."
It’s also using a bazillion words to make a point that could be summed up in a single paragraph: there’s a huge variance in the number of tokens required to encode the same content, with code leading the charts.
To be fair, most of this was already known, and Anthropic communicated very clearly about the different tokenizer they started using.
Their compute is also mostly 1:1 correlated to the number of tokens, so I don’t believe in the conspiracy that this is just to inflate prices.
It’s a shame because it’s making an excellent point! It just takes so long to get to the point that the reader loses the will to live.
Yes, I could probably ask an LLM to summarise it for me. No, I’m not going to. I would prefer the author just take care of that for me.
I could live with ai content if it was short and to the point. But it's always so lengthy. Hope that will change.
A tl;dr section at the top and then the long read from ai could also be OK if they marked it.
My general vibe hearing about AI.
The nudge to think about both "tokenization as variable" as well as actual tokens consumed per task is still good.
The issue is that it’s not just code - they suck at writing. Really bad. Unreadable, incoherent, messy.
Humans are also bad at judging the quality of things they themselves aren’t very good at. So a senior swe sees what claude spits out and says “This is trash.” And spends x amount of time getting it to not be trash. And Jr dev thinks “this is magic!” And pushes it to a PR.
So my theory is the people “writing” this AI slop think its great! But actually just aren’t very good at writing copy and don’t have the skill to recognize it and prompt their way out of it.
Or they don’t care. That’s an option as well.
PS for anyone reading, next time AI does something that you aren’t super familiar with that looks pretty good… maybe find an expert to review it.
It is actually a big result of work, a lot of research and attempts. And to just say that "oh, this is AI-slop," I consider unfair, but that is your choice.
There is a difference: - There are people who do, - And there are those who criticize.
LLM speak is like the new corporate speak. Enterprise writing is fulll of fluff and nothings and they all read the same. That sameness is what most readers here are sick of.
(Your comment here that I replied to is also written by AI which is even more sad :| )
Weird shield to hide behind, considering there are also people who can do both.
One way or another, I want to note that yes, this text was made in collaboration with AI. My English is non-native. It helps me translate, helps me structure better. Yes, there is a downside, it can bloat the text with unnecessary words. But that, unfortunately, is the price.
But the key thing is that I tried very hard to share my many years of experience, or rather a part of it, which I acquired, with all of you. And I am very glad that this information turned out to be useful to you.
The key here is: * The information that is written in the article. * Not how it is written, but what I was trying to convey to you.
Thank you very much for reading and responding.