Kimi K3, and what we can still learn from the pelican benchmark(simonwillison.net) |
Kimi K3, and what we can still learn from the pelican benchmark(simonwillison.net) |
Did you read either the post or the comment it was referencing?
On the note of training on SVGs, I have seen some labs models outperform when prompted for SVGs of certain animal and action combinations (pelican on bike, panda eating burger, etc.) compared to other similarly outlandish prompts for SVG output that are not part of widely reported benchmarks, even shared evidence one of the last times this came up on here.
[0] ... incredible Simon still believes ...
[1] I’m still not convinced that labs ....
"That artist saw a pelican at the beach once!" [cue the outrage] "He's not a real artist, he's a cheater and produces nothing original!"
I'll leave the contrasts between LLMs and people to the well-written sibling comments.
Plus obviously humans can still overfit to a specific style of test.
This is quite possibly reasoning-effort prompt which is injected before the opening <think> token whenever you set a custom reasoning effort, see e.g. DeepSeek-V4 max mode prompt: https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main...
So I put together a quick comparison of the last couple iterations of Opus, Fable and now Kimi.
Kimi is cheapest by 5x but also slowest by 2x
GLM is half the size of DeepSeek but costs four times as much, and beats it on every benchmark.
I'm not an expert on this stuff but it seems to be the attention mechanism. DeepSeek were bragging about how cheap they made it. But if you cut costs on attention you get worse results with way more parameters.
If I had to guess it seems to be the difference between memory (params) and intelligence (attention density). I think you need both.
Why does Kimi not use a "Double Cheese Whammy" branding for "their" butchered and stolen IP?
There seems to be more to producing a better model than brute forcing parameter count after all.
Perhaps more importantly can they do that during reinforcement training. Learning how to critically analyse the appearance of what it generates would be quite useful.
Manually feeding images back to models has been hilariously bad in the past which suggests that relating something it sees to something it wrote is not an ability it is very good at.
I'm starting to not trust any "benchmarks" when it comes to frontier models at least. As an example Sol feels the most "gets stuff done" but has zero taste, or any capability to surprise.
And for frontier models I go one step ahead and try to recreate a complex animation video, with the ability for the model to review its own work. And at this Fable is still the top one. Ex: https://www.youtube.com/watch?v=uDAeAuYyl0E (recreation of Claude announcement video) and https://www.youtube.com/watch?v=cSsVNtGPOIg (recreation of a fireship video). Sol did something similar but you can instantly tell its AI slop from very small things, and it just has no narrative or thought put into the writing.
https://mesmer.tools/benchmarks/ai-video-generation , I usually put basic ones here.
We may be boiling the oceans but at least we are finally getting some good SVGs of pelicans on bicycles.
I can't help but wonder where is the trend going? What will we have in five years? Maybe it will all have puttered out, and we will have moved to the next thing? Or maybe the prompt then will be "make a pelican ride a bicycle", and out will come the genetic code for a giant pelican with extremities suitable for a handle bar and pedals, and an inborn affinity to ride bicycles?
> What will we have in five years? Maybe it will all have puttered out, and we will have moved to the next thing?
We will just have more of the same.
I take exception to that! It's a performative joke for attention that works far more widely than just Hacker News.
You still need an OpenRouter API Key and be careful this can burn quite a bit of money.
You’re reading a personal blog and complaining about an open source personal project he runs and distributes for free. He’s allowed to talk about his personal work on his personal blog. Especially considering the cli utility he talks about is directly related to the post.
Imagine complaining about someone generating valuable content for free and not packaging it to your personal tastes.
Sorry, how again is this the end of the frontier labs?
Competition is always good.
Engineers get unbelievably silly about evaluating costs of things.
"The tokens are so expensive!" Oh my sweet child, how much would even the least capable human effort cost? This is what the executives properly understand that the programmers don't.
25 cents is 10x the cost of 2.5 cents, but it's still extremely cheap for the product. It's very much the wrong comparison for a world where the primary competition is still humans who need to eat, and it treats percentage differences as more important than absolute differences when the opposite is true.
Secondly, humans vs LLMs are apples vs oranges. It makes no more sense to compare human costs vs LLM costs as it would have to compare human costs vs calculator costs. LLMs are faster and cheaper but extremely different beasts with different limitations. Humans do not one-shot SVGs of pelicans riding bicycles, and they do not charge in tokens.
Comparing LLM cost efficiency is not something that should need to be defended. It's quite straightforward and reasonable...
I have shared examples of certain models by certain labs doing far better on the pelican cycling vs other, similar prompts. Just operating on a feeling that labs don't optimise for this (as mentioned, even if they don't training data is filled with these) is not solid enough that criticism shouldn't be leveraged when it comes up.
Please share those again!
One of the things I'm most looking forward to is a lab producing a model that creates a really great pelican riding a bicycle and then a terrible sloth riding a skateboard (or whatever).
I've not seen that myself yet.