GPT-6 Astra makes major gains in the Artificial Analysis Coding Agent Index(artificialanalysis.ai) |
GPT-6 Astra makes major gains in the Artificial Analysis Coding Agent Index(artificialanalysis.ai) |
They really don't seem to match my real-world experience, and based on the comments I see I don't think that match most other people's either.
For example, Opus 5 was at the top for some time. My experience is that it's not noticeably better than Opus 4.8, and it definitely seems worse than Fable 5, which AA benchmarks put behind Opus 5. GPT 5.6-sol and Opus 5 seem pretty interchangeable, although Sol is noticeably better at finding problems in code, particularly edge cases.
[0]: https://x.com/EpochAIResearch/status/2095602754282783108
First chart: from score 61 (GPT-5.6 Sol) to drumroll 61 (GPT-6 Astra)
Are they just scaling more? getting more data at the same rate? training against the same benchmarks? making the same breakthroughs?
How can this be explained?
I believe the development of new models has been like an s-curve: There were enormous incremental improvements earlier on, but now we're reaching the right-hand side of the s-curve and all the new models are clustering together.
Even the lighter and smaller and cheaper models going to end up near that limit, and that will likely evaporate the perceived economic value of both OpenAI and Anthropic unless they manage to lock it in/offset it with platform effects and branding (they might well be able to do that).
My impression is that we might well be able to move past that limit, but we would need another radical invention like the transformer architecture that the whole current generation of models is built on.
Am I missing something or is this not looking too... stellar?
Fable was better but I don't have 1000$/day to spend on it.