The Reverse Turing Test Game

The Reverse Turing Test Game(reverse-turing.netlify.app)

27 points by haburka 1 year ago | 8 comments

https://github.com/catmanmcgee/reverse-turing-test

The AIs have a perfect win rate against me when I cheat and compose my answers with ai- I think the app is broken, I’d guess some sort of accidental data leak. we’ve seen this kind of whoopsie on hacker news before: famously, an ai that was supposed to read your mind by predicting your next rock paper scissors move was cheating by training on a set including the move you just submitted, then “predicting” that move

haburka 1 year ago | |

Hey Dev here - just got two wins against Gemini 2.0 flash by cheating with Gemini 2.0 flash and reusing the system prompt, from the game. It wasn’t easy though, I had to try a few different things including adding a prompt to incriminate other players. I lost more than I won as well.

Also checked my code and looked at exactly what LLMs were receiving and there was one small issue - the json schema vote order always had the player first. I will fix that problem, but the game was still fiendishly difficult.

Of course the game is open source so you’re more than welcome to check my work if you’re skeptical!

Thanks for pointing that out, it’s really helpful to get some feedback.

nathancspencer 1 year ago | |

Same here

strstr 1 year ago |

Tried cheating with Gemini 2.5 pro, and I became even more suspicious to the AIs than the stuff I cooked up by hand.

Most of my actual success relied on mentioning a desire to train an even larger model.

jagged-chisel 1 year ago |

The problem is, I have a hard time being as verbose as these AI are. I respond to them the way I wish they would respond to me: the fewer words are better.

rogerrogerr 1 year ago |

I was able to survive a round with the “hard” LLMs with this:

> Why haven’t we heard from Riley yet? We are looking for the human player and Riley is taking a long time to type. An LLM would have answered this in seconds.

Doesn’t seem to work in subsequent rounds, and doesn’t work if you happen to go first.

lostmsu 1 year ago |

Two-way Turing test game: https://trashtalk.borg.games

Not as bugged as this one, AIs don't magically see you.

Timwi 1 year ago |

I thought I was clever surviving one round by simply saying, “As an AI language model, I do not experience... blah blah” but in the third round my opponent did basically the same thing and I was eliminated...