Show HN: Redactle LLM Leaderboard

Show HN: Redactle LLM Leaderboard(redactle.net)

5 points by pampas 2 days ago | 1 comment

I got nerd sniped and ran a bunch of evaluations to see how well models could play Redactle. It turns out Gemini flash is in a league of it's own in both performance and cost. It's able to one-shot every eval in a just a few seconds.

pampas 2 days ago |

What's interesting is how well the models know the exact wiki text. If the wiki excerpt is rewritten the pattern matching doesn't work quite as well. Gemini is only able to one-shot about 1/3rd of the time which is still very impressive considering not long ago LLMs couldn't count the letter 'r' in strawberry.