undefined | Dark Hacker News

14 points by jiwidi 80 days ago

shiandow 80 days ago |

These seem to be different tests? One has 6 tasks the other has 30.

yorwba 80 days ago | |

Yeah, of those 6 tasks, only "halluc-doc-http-handler" isn't within 1% of the previous result. 86.6% is 13/15 rounded down, so if they sampled 15 attempts for that task, the probability of getting 100% when the true success rate was 13/15 would be (13/15)^15 > 0.11, which is not all that unlikely.

ekidd 80 days ago | |

Thank you. Those are not the same test at all. I agree that something weird is up with Opus. But this post doesn't actually prove anything, and the title is misleading editorializing.

jiwidi 80 days ago |

See original opus 4.6 sitting at 16% hallucination and the retest on 12th of april at 33%

They definitely must be doing some quantization or optimization to meet demand, otherwise why would model performance degrade this much? It's been crazy for me personally

dns_snek 80 days ago | |

I'm the last person to defend any of these companies, but it's not a retest. The set of tasks is clearly different and results for the original tasks are nearly identical. It differs on one task where it previously had fabrications = 0 and now it's 1, which dropped the score from 100% to 86%

Combining multiple tests on the same leaderboard like this is nonsense, there should be a separate leaderbaord for the new tasks where every model is tested again.

Putting it on the original leaderboard as "Opus 4.6 (April 12)" is so obviously inappropriate that it smells like deception. You could say that the leaderboard is hallucinated.

metalman 80 days ago |

do people get the simple fact of data sets becoming more and more polluted, one from AI, and secondly from an increasingly deranged human population that is hyper focused on getting some extra financial advantage at ANY other cost, and a huge part of that is about "reputational management", again with zero limits on missinformation. a sane society would pull the plug, now.