This is not a benchmark for testing them against an Einstein.

Notably, Pangram is very conservative, and it's not difficult to manually get an LLM generated passage of text to turn human-written. So a score of near-100% AI generated means the writer didn't do even very light editing for a large part of the text.

There are extremely good reasons to be skeptical of fully LLM-written content. Our attention spans and our online platforms were built in a time where a long, data-supported article with references was expensive to produce. The time to write it was vastly longer than the time to read it, which means you could usually rely on some good faith, baseline level of accuracy and thinking on the writer's part.

With LLM-generated content, it's very difficult to know if 5 minutes, 5 hours or 5 days went into writing of the content. On the surface, it all looks similar, but the 5 minute version usually communicates very little or very shallow ideas, makes factual errors, and is generally lacking a lot of context. It's fast food writing.

These low effort versions of content take way more to read than they take to write. And combined with the obtuseness of the writing style, it all places undue burden on the reader to figure out the underlying message, because a lot of it has been mangled by the writing process.

I think LLMs are hugely helpful for writing, but to use their proper potential one needs to use them for feedback and engage with them at a level deeper than simply "write an article about X" or "rewrite this paragraph", and the text then doesn't obviously read AI generated as a bonus - I think nobody really has a problem with this.

Per the writing, reading AI writing is like having something taste "chemical". Not very specific, but still a very recognizable and bad taste that makes it hard to enjoy and marks the thing as low quality.

Ultimately I hope the result provided value by stimulating debate. To me it feels like lots of people rejecting the idea that a Chinese open weights model could be THAT good. Yeap, so the writeup is formatted with AI, but the benchmark and the results has taken meeeeeee weeks of effort to bring it together. The point is I am not going head-to-head against AA with my spare time.

And the concept is based on a Grand Tour - of if you are Brit Top Gear TV show where we put 1 star in a reasonably priced car (my humble, little test harness)

Hence 'same drive, same track. The LLM is the Star'. The irony is entirely lost on some of these folk. I am not, in my sparetime trying to be Artificial Analysis. But actually the point is....do you really trust Artificial Analysis: or do you trust a benchmark that is free and open for you all to pull down and run (and adapt to your needs, in your circumstances and your problems).

If there is anyone who wants to look at things with open eyes instead of the group think, it's all there for deep review

> I'm seeing a worrying trend on HN. Nearly for all articles, there is one unquantified, unproven comment at the top saying it's 100% AI — no proof, just baseless emotion of what fits the commenter's writing style. This is the new witch hunt, or virtue trolling.

Was that the case? Genuine question. Here the em dash is an actual em dash symbol, where in the other paragraph you used what looks like a minus symbol. It’s also the type of construct used by Claude.

https://reinvently.co.uk/about/

The parent was clearly not stating anything confidently.

> Look at the later comments, they have substance oriented discussions.

There are two sentences in the parent comment. Why is the top response (yours) pointedly ignoring the sentence with substance?

Other than having all the AI tells, what proof can there be? Are you seriously asking that, because there is no way to provide hard proof that we should just stop pointing out obvious AI tells?

In this article, _I_ get unstuck right at the very first paragraph:

> Same driver, same track. The LLM is the star. Seventeen leading models driven round the identical 28-realworld task lap — one harness, same verbatim prompts, deterministic grading — and the results go on the board.

It jsut doesn't make much sense to me. At best, I think it can be glossed as... "I made an arbitrary benchmark which I'm not going to explain, and I plotted the results."

------

Getting my own opinions out, this is blatant slop. It claims to be "deterministic grading", but then almost the _entire_ webpage is editorialization. Examples:

* "If you only run one model, run glm-5.3"

* "opus-5 posts the best rubric on the default panel"

* "deepseek-v4-pro is nominally cheaper still at $0.0029 [...] treat it as a batch-only option."

* " It performed well on what it completed"

And this really has to be policed. Once some tipping point is reached and too much of the HN homepage is AI slop, the site is dead.

But beyond that, can't you see how terrible the writing is? This is unadulterated AI slop.

I won’t be surprise if 80% of content in social media - including maybe HN - is AI generated as of now.

Further, you ignored the actual contents of my comments, to latch onto a superficial aspect. Please tell me: why do these results contradict existing attempts at benchmarking LLMs, which were designed with considerably more effort? Because the website certainly doesn't explain why in a way that is human-readable, which is why I asked.

EDIT: as explained by another commenter below, its because Fable refused to perform some of the tasks.

If authors would be open about their use of AI right in the title, I could decide for myself if I want to read it or not and spend the time, and there would be no need to write a comment to warn other users not to waste theirs.

So the root of the problem is authors not disclosing properly. Not commenters telling others about the clickscam the author is pulling.

And the commenter really shouldn’t need to prove it’s AI written. The author needs to prove it’s not. If you want my attention and time, you should have a good reason.

Right now, authors on HN are just abusing the fact that I need to click their bullshit to find out that it’s bullshit I wouldn’t have read if it was disclosed properly from the start.

If I want llm output, why would I read your writing if I could just ask one myself and get an equal result?

Also, it’s short-sighted to think it’s just the writing style that makes people think it’s fully or partially AI-generated. This website has a ton of people with decades of experience and domain knowledge in their fields.

For people with domain knowledge, it’s quite easy to spot flaws someone without domain knowledge generating words about something they have zero expertise in, or only having it explained by the machine that is making things up, is making.

I don’t see the need for moderation to police people who think something is AI generated. You tell people they can just ignore the article, but they can’t, they have to click and read to find out, and on the other hand you could also just ignore their comment if you don’t like it, or disagree and tell everyone why you think this is not AI-generated bullshit wasting others people time to generate clicks.

EDIT: You were correct, Fable and Opus reject some of the coding tasks, which is why they score lower. Thanks for explaining.

EDIT2: I believe this benchmark is invalid, my Opus 5 runs the supposedly rejected tasks just fine.

The reason for both of those things is, as you point out, the benchmark is very obviously saturated

Fable is good in a few very specific domains (graphics programming) but otherwise it's an overhyped model. Far too expensive too. Opus 5 is outright better in every metric.

Fable is very steerable. For me it makes the "simplest thing that works" and - once I tuned it - very understandable explanations. (Unsteered, I agree that Fable's writing style is too cryptic.)

I can let it loose on a 12+ hour task and it will test and verify everything autonomously. These days most of my work is done by letting Fable just run overnight, while my days consist of meetings/planning to figure out what I want Fable to build next.

Incidentally, if you want to look at this in more depth - my previous post about the eval framework got like zero response; that's where the results, methodology, code for running the evals, evals are all shared.

Anyone can run this to verify it for themselves

https://github.com/ed-is-ai/featherbench