Hey there! Author here. I built this over Labor Day weekend.
The idea is pretty simple: these assistants all claim to know you now, so I wanted to see if that can be measured at all.
You answer 15 A/B questions privately, then give the same ones to your ChatGPT/Claude/Gemini and paste its guesses back. I compare that against the same model with no history of you.
A few important bits:
- raw accuracy isn't enough. A model that knows nothing about you can still guess a lot from population priors
- scoring is relative. If your assistant puts 80 on your real answer and the no-history model puts 60, yours wins that question. Near ties are ignored
- the A-F grade is still provisional. About 1 in 10 no-knowledge runs can still get a C just by chance
- no LLM calls happen while you're taking it. Baseline predictions are generated ahead of time and cached
The experiment isnt perfectly controlled either. Your assistant gets all 15 questions together in the normal app. The baseline was generated through the API one at a time. ChatGPT also needed an extra line telling it to actually guess instead of hedging at 50/50.
The validation I care about most is cross-person scoring. Take Alan's AI guesses and score them against Bob's actual answers. That should fall back toward zero. I just need more data before thats useful.
I wrote 116 questions and kept 62. A bunch got cut because the model already had an obvious preferred answer, or because they were basically age/country/job questions in disguise.
Code + question bank: https://github.com/skilledDeveloper/knowmenot
The annoying part has been writing the questions. A lot of A/B questions look balanced, but one answer is still obviously the "reasonable person" answer to a model.
curious what grades people get, and which model.
For anyone who wants to help grow the question bank: PRs against items/pool.json are welcome, you get credit. Details: https://github.com/skilledDeveloper/knowmenot#contributing