Calibration game is a game to get better at identifying hallucination in LLMs.

Prompts and hallucination labels (using ChatGPT) are obtained from Alpaca and HaluEval. You can use your own dataset to calibrate users to the responses of a different LLM (https://github.com/Viktour19/calibrationgame).

ML engineers are starting to come around to the fact that hallucination in LLMs is really tough to mitigate. This is an active area of research and we may have LLMs with no hallucination in the future. At the moment, we need to design user experiences that align users with this behaviour. The hope with the Calibration game is that it equips engineers to calibrate users to the responses of LLMs used in their applications, e.g., as part of the onboarding process.

Would love to hear some feedback.