I am hoping someone with more free time than myself can contribute some things in the RF engineering domain in the 'engineering-sciences' section. There's some problems out there that will definitely stump even a smart LLM.

> You can tell that Claude really does grasp a wide array of highly specific scientific and mathematical nuances... where's codex is just basically for coding and that's it.

If you have time, can you elaborate or give some examples of mathematical nuances?

I am evaluating Sol and Fable on a fairly large dataset of subtly flawed informal mathematical arguments (task is to identify and name propositions with substantially incorrect proofs in a larger body of text), and Sol is saturating the benchmark, while Fable is below 50% even with the most generous grading.

I don't work in the natural sciences, so I suspect you mean something different by "mathematical nuance".

So why is Sol the one solving Erdős problems?

IMHO it's roughly task depth (fable) vs breadth (opus). Fable is great at tracing and debugging sometimes, but otherwise shorthand for confabulation. It's persistent but ungovernable, struggles to switch contexts, and goes insane with too much freedom to explore. Don't point it at anything that looks like a "system" for actual work (but mapping or planning might be ok). Opus is maybe not as creative, but it's more stable and more trustworthy. Opus driving Fable could be awesome, but Fable unleashed/unsupervised on longer horizon tasks or things that require more methodical changes on lots of components seems like a disaster every time I try it.

Since this kind of thing is always down to harness, project-type, and other structural constraints, of course your mileage may vary. Fable is probably great for pen-testing, or as a decision-making kernel of other kinds of applications, and way better than Opus at those things. Probably fine for code-review or changing a codebase of a few thousand lines in any language. Actually building that codebase or changing an even bigger one? Woof.

How this fits in with science? IDK but I bet other existing causal reasoning benchmarks might tell the whole story and this is back to stability again. Sometimes having a smart idea is really important! But more often it's important to just not forget what you were doing. What was I talking about? Oh look a squirrel

This is measuring on scientific tasks though. I haven't used Fable lately but when I was playing with it when it was new its "safety" features made it practically impossible to use for biomedical science -- it once refused to work on a pipeline of mine that was analyzing pathogenicity islands in bacteria (presumably because it had guardrails to that effect to stop potential bioterrorists and the like)

Benchmark does check correctness. Team is working to write a paper that will likely contain failure mode analysis; checking for instruction following could be a good idea.

Agents are given instructions in markdown format, allowed to read data and libraries sandboxed in a Docker container, and evaluated on deterministic pytests on the outcome. Two things the team aimed to enforce to add tasks we could trust: - Scientific workflows are often simulations that are correct upto numerical tolerances (the scientist decides what's reasonable), so task verifiers' evaluate results of agent-written code within the tolerances. - There could be multiple solution codes to a scientific workflow, and the team tried to ensure the verifier tests accommodate those. Not overfit to the oracle reference code, written by the scientist.

Instruction following is implicitly assumed, if the model gives up and doesn't complete the task it counts as a failure because the verifier tests fail.

> I worry this doesn’t check correctness

Then it's not a valid benchmark. I agree though they're not reliable enough to just put results in a paper.

I love it when people handwave tech developments of a truly gargantuan scale

>AI should have started with science from the beginning, not after 4 years.

DeepMind has entered the chat.

Do they especially train for scientific performance?

I generally trust vibe code a lot more than researcher code.

"Vibecoded" science is well on its way to being better than your "real Science™". Withholding the future is not an option.

How is an llm parser_spec to parser better than something like lex?