"Model X performed great, but we can't possibly tell you anything about the code it was looking at apart from it was a large code base from an unknown company".
So basically pinky-promise benchmarking ?
I'm not sure I follow the value here ?
But then if we take that argument to its natural extreme, surely it means people should take the marketing bullshit published in the 100-page system cards published by Anthropic & co as "valuable" too ?
I tried gemini on simple code base and it invoked 210 tool calls just to update 3 lines of code.
There's only two or three sane options here - you can easily try them all and pick yourself.
I'm not sure what others are doing that they're getting such different results, but I'll take Codex every day of the week.
Try and use gemini 3.8 yourself for any real world work and you'll see it's terrible. It'll just go in circles reading the same file 20 times for no reason making hundreds of tool calls for a simple change.
EDIT: I was using gemini cli... it's not a harness issue lol
PS: Just Googled it to confirm: Gemini CLI was deprecated on May 19th, 2026. The correct harness is called agy or antigravity for Gemini 3.8 Flash.
https://developers.googleblog.com/an-important-update-transi...
And it just BURNS tokens like crazy.
So I totally believe that Gemini is just bad. Which is surprising because Gemma is very good for some tasks, but I never ever had any success with Gemini, be it in cli or chat thing or anything else that has gemini branding.
But even so, pretty much yes: companies that actually have reliable and accurate info in their releases get trusted more. It takes time because the default is to disbelieve info from biased sources, but it is possible to trust some of them more than others.
The thing that takes the most time is finding the examples. In my real dev flow it’s rarely ticket -> PR -> merge, things bounce around a lot more. So, even though the stated goal is to get away from one shots, that is basically the environment you have to set or else test for specific other outcomes (e.g. agent stopped and raised a question when it realized x).
It takes time to do, but I would really recommend it. Now I can push new open models through the batteries and see how they line up to past ones in a few days (I run them locally, it’s slow). It moves my sense of x model is good at y and bad at z to from vibes to a better heuristic (these still run at temp 1, heuristic is the correct way to think about outcomes IMO). It grounds it in your actual code and problem space.
My takeaway from my testing: in Rails or front end codebases, most models I test are competent and with a human in the loop they would accomplish their goal of getting to a mergeable PR. They are not as good as Claude and since I pay subsidized rates via subscription Claude still gets first pass. They are very worthwhile to layer in as reviewers and catch many issues. My anxiety about a rug pull by the frontiers has been turned way down. I would have to adapt to a local only flow, but it wouldn’t be much adaptation and the opens can deliver in their current state.
Once we send code & prompts to the providers, it can no longer be considered private.
[0] And came to the same conclusion as you did: Claudes were better but not by much: https://news.ycombinator.com/item?id=48654635
I rebuilt my entire workflow around agents so the unease that the frontiers would change something (access, pricing, availability) and lock me out of that were high. Also why I spent way too much on hardware (at least that can be deducted). Now the whole stack could run in my house and I feel much better about the situation.
Once I got the testing going though it is worth it for its own pursuit. Building processes around the dev process and trying to get the best outcomes is at least as fun to me as actually delivering client code. For the first time in my tech career I feel like I’m in a place with no maps. No one has done my experiments yet. I have a custom quant of K3 at Q5 that lets me get 10 tok/s on a CPU inference box (admittedly you need a 72GB Blackwell also). As far as I can tell no one else has done this. It’s such an exciting time!
I run Fable as my main model with Sol as advisor that watches every turn. Fable likes to throw around assumptions that it didn't check that are simply false, and Sol always goes to actually verify them and then alert Fable it's assuming things.
I've tried reversing this pairing with Fable as advisor. It'll just sit there going "sounds good"
But I agree - Fable makes some spectacular assumptions (which are poor assumptions).
GPT models are extremely proactive and gredy. So when Fable mentions something that may affect some obscure component of the system, GPT will start digging the codebase, execute web searches, re-read AGENTS.md and hit fable on the head.
Fable never does that, it just reads the turns and acknowledges it read them. This also explains why GPT models tend to overengineer things and why they're amazing reviewers if you triage their findings.
I was using my own homegrown (mega slop) harness for a while, but it distracted me from working on my actual projects, and I realized oh-my-pi was doing the same things I've been doing, including advisor, native server-side compaction, etc. It's a really good harness.
One lesson of running botsbench.com, in a slightly different domain, is to measure for model contamination every time.
Even posted an article about it:
https://www.matheusmoreira.com/articles/code-reviewing-lone-...
Unlike TFA, the lone lisp code is public. I suppose the models could have been trained on my codebase. Still, I think it produced some interesting results.
Took months and loads and loads of tokens to do this, so I'm not gonna repeat this study as new models come out. It did anchor all of my future expectations, though. OpenAI is winning as far as I'm concerned, and their cybersecurity program is the only remaining pain point.
IMO Fable 5.1 ~ Astra > GPT 5.6 Sol > Opus.
For example, I found Kimi K3 to use more tokens than some other models, which caused it to cost twice as much purely because of the token volume. This experience lines up with ArtificialAnalysis’s benchmarks, but not these.
There’s a number of other comparisons here that don’t match up with my experience or other benchmarks. By many accounts, this is the outlier.
I could attribute the differences to harnesses used or something like reasoning levels, but none of those details are published.
While this seems interesting, I can’t take this seriously.
Correction: The harnesses are listed as a column, I missed that. My other concerns and questions still remain, it’s unclear why some of their results are the outlier that does not match my experience, ArtificalAnalysis’s benchmarks, or some of the experiences of others commenting.
You need to explicitly tell the LLM what to take into account when generating code or it will miss things. And that's with months of saved memories.
Where it fails is on plugging business logic in. Tell it to generate a new piece of UI and it will do that fine.
My guess is the 10x success stories are for from scratch applications where 80% of the code is boilerplate and self contained modules. And no one comes back and tells you maintenance dropped the productivity improvement to 2x.
Btw since it's mentioned a couple times in the comments: the Claude Team and Enterprise subscriptions do not train on or retain your code by default. Apparently the promises were good enough for my employer and their customers.
[1] By the way, real-world smells of LLM generation. Normal people write "real world".
I strongly recommend trying this approach out yourself. The recipe is not rocket science. Get your coding agent to take a first cut at building the atlas itself, and then manually correct it. Once you’re happy that it got things right, put an MCP on it or a CLI or whatever. And your LLMs will know what to do from there.
- DeepSeek V4.1 Flash
- Kimi K3
- GLM 5.3 (and flash)
- hy4-preview
- Grok 4.6
All of these can be acessed using a $10/mo OpenCode Go subscription.
1 Fable 5.1 38.8%
2 GPT-6 Astra 33.8%
3 Gemini 3.8 Flash 31.2%
4 GLM 5.3 28.8%
5 Grok 4.6 23.8%
5 Muse Spark 1.3 23.8%
7 Kimi K3 18.8%
8 GPT-5.6 Sol 16.2%
See number 4, 5 and 7.
My intuition from the example full instructions are that the tasks are poorly specified which results in ~60% failures due to bad assumptions and missing requirements.
Key questions, are they topped in performance? Is there some next leap?
if i can help answer any questions please feel free to email me at janak@withspecific.com, happy to send over my phone number as well :)
It's probably very hit or miss like everything with LLMs but I was really surprised it performed that badly
How does that work?
My gut feeling is that any serious real-world company with a proprietary codebase worth looking at would not be handing out the crown jewels to a third party. License or not.
I don't doubt somebody licensed their codebase to them, I just have my doubts about who the "who" could be.
At any rate, I'm not sure it matters whose codebase it is. I'd even say that a shitty codebase might make for a better test.
https://artificialanalysis.ai/evaluations/terminalbench-v4-0
happy to share more if you are interested.
Hardcoded constants everywhere
Multiple same-line macro invocations in C
Random indexes in production code
Hideous tokenizer code in C
It randomly decided going from no explicit partition key which would randomly add to a Kafka partition to partioning key company was the right solution.
In fact, that was fairly counterproductive since this codebase dumps events to the topic per company so you'd get a lumpy distribution where only one partition is ever active at once (starving other consumers)
The fact it decided it should add this without prompting was a bit shocking and I'll probably try to make it explicitly clear in the context not to "do random shit that wasn't asked for"
its under this section 'Missed requirements are the most common failure' on realswe.withspecific.com
we also have the setup in the blog. the reason for lower success rates is that we gave models ambitious tasks that real engineers worked on for weeks.
I’m not sure what the methodology of these are, but they certainly don’t match what I experience. Maybe I need to look deeper for relevant benchmarks.
To me, having to give extremely specific instructions and provide tight guidelines and guardrails defeats the purpose of agentic coding agents almost completely. At that point I might as well do the task myself.
With Fable I can start with a general ask like "I'm trying to do X, can you investigate and tell me what the shape would look like" and have it poke around and think, ask me questions with single-choice or multiple-choice answers, then break the task into small chunks, each of which becomes a ticket.
With Astra, it's like pulling teeth. It often does not understand what I'm trying to do, takes things literally, does not go above and beyond (i.e. infer intent), and stops way too short of the actual goal. I have to constantly prod it and it's frankly exhausting.
Otherwise I wouldn’t see others in the exact same codebase struggle and underutilize agents while others thrive using the exact same ones.
This happens to me more commonly in large projects (>100k LOC) and in those projects it seems to happen every few sessions. I feel this specific benchmark would be impacted by this more than the smaller contrived benchmarks.
I think the bigco folks saw all the "it wrote all this code but the tests didn't pass" or "it wrote the feature but it's super brittle" and tuned the newer model+harness combinations incredibly aggressively to try to turn a lazy prompt into "median Enterprise Architecture design suggestions" to bring up the baseline, but in a way that slows you down if you don't want that.
I'm not on big enough subscriptions to want to burn a lot time just evaluating Fable/Astra comparatively until they're cheaper, heh. I can steer any of the cheaper ones just fine anyway.
on the other hand, having fable plan and orchestrate with opus implemention + fable reviews, is my go-to. if you give fable your guidelines up front or in your {claude,agents}.md, it will keep opus on a tight leash. opus can still write great code almost on par with fable, but it needs to be tightly constrained.
My own experience is opus being lousy at an extremely niche math task, but it was still easier for me to describe what it needed to do to get code and correct issues in its reasoning/working than to write myself. a minor model number change later and it's nailing everything, despite my opt-out. Its is astronomically unlikley others were working on this also, especially at that level, especially this application.
so, safe to say they _all_ train models on chats, the only difference being if you "opt out" you at least have some defence later when they steal your work and claim it as their models original output.
There's a funny middle where they may not know the exact answer confidently, but do know where to get them, or aspects of the problem. That's still a potential lift in the competition . So we also check the traces to see if there is any knowledge during intermediate step DB queries that it shouldt have based on the previous steps.
I gave a recorded talk at BSides Las Vegas a few weeks ago on 'adversarial evals' that gets into the kind of cheating we've been having to deal with.
So it does not make sense that the same companies would then magically sign-off on allowing their entire codebase to be spoon-fed into a whole bunch of LLMs for benchmarking.
But the phrase "don't do random shit", semantically, from a monkey's paw perspective, could imply curtailing the model's creativity and 'thinking out of the box' capacity, that might have existed in its 'reasoning' process. So I'm always concerned if it's possible that, adding these phrases might be part of the reason why a model performs dumber than it should.
I don't have empirical evidence to support that supposition though.
So if you're doing something very ordinary, fewer instructions result in better results. If you're doing something fairly off-piste, you have to give instructions to that effect and accept less creativity. For situations where you want it to do something extremely specific, tons of instructions and accept that you're going to get much closer output but much worse "intelligence"
Another way to think about it is Type 1 and Type 2 errors or sensitivity and specificity from statistical testing - do you want an agent that solves any problem but goes off the rails 10% of the time, or do you want an agent that can only solve 10% of the problems but nails them 100% of the time (sensitivity and specificity, respectively)
I much prefer getting alerted when there's more than 1 approach to the problem and it's discovered mid-implementation.
I don't want to do the grunt work of writing code, but I do want to know the architecture and be responsible for the decisions.
Fable is also very good at pushing back when I propose something that will cost me. E.g. I'm working on a configuration layer above nix to manage my homelab fleet declaratively, and I tend to get into "config as new language", where Fable just goes - let's not do that.
Oh man! This also is a pet peeve of mine with Fable. I will look at what it's doing and say "Shouldn't it be done this way?" and then it will spend forever arguing with me that it should be done the way it wanted to do it. It seems to get stuck in a certain way of thinking and will insist its way is right until I can really prove it - or just go over to Astra.