This game is well known in the UK as the "Connecting Wall" from Only Connect.

This result - poor Chat GPT performance - surprises me. I thought pattern detection and set forming was something that Chat GPT could do well. Perhaps it would need a model to be specifically trained for this task. If alpha-zero can master chess, then surely this game isn't beyond what is trainable.

You can prompt Chat GPT that it'll be playing the connecting wall without having to explain the game. It still fails to make a good set of connections when provided the wall.

One interesting part of the "Connecting Wall" sets is that there is almost always a "Wordy one" involving changing a letter, adding a prefix, anagrams, etc. Almost always a "Person" one for example there'll be a set of "Famous people named Tom..." but not a set of "Toms" with a set of "Margarets", and then a couple of general sets.

This is a huge help given the 2 minutes and 30 seconds provided.

On another note, it's possible that the GCHQ puzzle book would be in the training set, which has many puzzles with solutions for this format and a very similar rubrik with 55 items and sets of sizes 1 through 10. That said, Chat GPT perhaps would not tie the answers in the back of the book to the solutions in the front.

I all, I think an AI trained for this purpose with problems and given solutions ought to end up mastering this format. But a general purpose chat GPT seems like it performs very badly.

This is pretty interesting. Intuitively, Connections is the kind of thing I would expect GPT to not be good at, because almost every day there's something that feels kind of "out of left field" in the categories. In my experience LLMs are good at regurgitating the "standard" take on a topic, or "best practices", but lack the creativity and out-of-the-box thinking that makes Connections fun.

On the other hand, it feels like the kind of thing where an LLM might be surprisingly good, because it could, in theory, be able to see more correlations than a human can. Based on these results I guess my intuition seems to hold up.

I wonder if a better / different way to approach this could be more "algorithmically" - maybe have the LLM generate a list of possible categories for each individual word and then try to operate on those associations?

Cool article!

I unfortunately can’t imagine having time to test this, but I imagine there may be a way to accomplish this with embeddings.

The game itself is sort of an embeddings clustering problem, with the added difficulty that each group needs to only be alike in 1 way (versus a full vector distance which measures how alike they are in every way).

Maybe there is some way to search for a vector of weights, which, when multiplied by all members of a group of 4, produces weighted vectors with the least distance from their center? And then it’s an optimization problem to find the 4 groups that find minimize the total distance from each groups center?

It may be possible to find a weight vector that selects for a particular slice of a words meaning

You need to model how a person actually plays connections. Start with the most obvious group that has the least ambiguity, and then your problem space is smaller on category 2, then the same for category 3 and 4.

So really you could fine tune 3 models - one for 16 words, one for 12, and one for 8. Then use them in succession.

Also, if you come across a mistake at the end (have some negative examples in the training sets), tell it to start over and add to the prompt what you think is NOT a group.

You’ll probably get better results by putting the examples only in the completion part of the training examples.

GPT-3.5 learns how to generalize better when it’s just in the completion.

This is the same problem that vexed the researchers who did the paper on the alleged reversal curse.

(https://andrewmayne.com/2023/11/14/is-the-reversal-curse-rea...)

FWIW I was able to get about 20% accuracy (perfect 4/4 groups) with ~50% of groups correct on average and most mistakes being groups with 3/4 right (so at least on the right track) with my first attempt at 0-shot prompting. The prompt goes something like this: ``` You are playing [game info] with [word list] Follow these steps: - consider possible groupings as initial brainstorming (>4 groups) - propose a first hypothesis based on the likely-looking groups - reflect on whether that grouping works - revise if needed then submit the final predictions ``` Having it start with word one of group one as the first output token seems unlikely to work from my intuition about what these models can do. Heck, I can't solve it that way! Burning some tokens on exploration and hypothesis building leaves it with the easier task of choosing plausible groups from the proposed options. System 2 thinking vs system 1 perhaps.

I've played around with the same problem, though I didn't do any fine-tuning. Some strategies that seemed promising:

  - A two-pass approach where you prompt it to generate the groupings, then separately prompt it to find the words that belong into each group. (Which of the following words best fit the category "COMMON DOG NAMES"?). It does way better at the more specific queries.
  - Don't tell it the constraints of 4 groups of 4 words; ask it for at least four groups of 2 or more words. Once you have 4+ groups of 4+ words, you can make logical inferences with your Python wrapper to come up with all the possible 4x4 groupings. If you're lucky there will only be one. If not... more queries to GPT, I guess, but I haven't figured this part out.

Love a published null result!

I've found that for tasks like this you need to ask the model to output its reasoning prior to outputting the "answer".

See "chain of thought" e.g

https://arxiv.org/abs/2201.11903

I have a couple of ideas.

1. Have it do a thinking/brainstorming phase first to try to work out what the potential categories are.

2. Then ask it to scan over each word and think about what categories it could go in, in order of likelihood.

3. Ask it to do the final answer.

Format the training set in that way, as if it got everything right at each step (since you only have the right answers).

It sounds like you had 7 * 30 = around 200 examples. Maybe you can feed a batch of ten at a time and explain the game and try to get GPT-4 to generate more examples. You will have to see if they make sense.

I assume that by increasing the size of the dataset by a factor of ten, and having the LLM think through the problem using multiple steps, you will get significantly better results.

I spent a bunch of time manually just using GPT4 with fairly simple prompts and giving it the same feedback that the game gives. There's an archive of puzzles which I used to try to train it with, and sometimes it would be very successful, and sometimes it was frustrating how bad it was at doing basic things like keeping track of what words it had used so far. Each day I would also have it play the new puzzle from the NYTimes which it couldn't have trained on. Some days it did perfectly some it made really stupid mistakes. It seems like a more concerted effort could achieve better results.

It's a shame you can't just see the probability distributions for the 16 words and choose them yourself that way you never hallucinate a word and the groups are always 4 words long.

I suspect the inability of the model to "plan ahead" is a significant contributor to its poor performance relative to a human. Being able to check a grouping to be sure it includes at least four words _and_ to check that it doesn't conflict with the other three groupings is a major advantage - it's pretty common that these puzzles include partial or incompatible red herring groups.

If this is the case, performance might be improved by taking the final solving responsibility away from the model and giving it to the script. You could ask GPT for categories, ask whether each word fits each category (discarding categories with fewer than 4 words), and then search for 4 non-overlapping categories.

(This might be missing the point of the exercise though.)

fun. good writeup. i tried setting up a custom gpt to run a game like anagramish.com - gave it the word list to choose from, instructions on the rules, etc - but no matter what i did in the prompt it would hallucinate start words or it would incorrectly accept invalid guesses (or mark correct guesses as invalid).

For a second I thought James bloody Burke was going to enter the chat

This game looks cool but wow the UX is terrible. Why can't you click & drag the words to reorder them? Seems half the difficulty is keeping track of your thought process with the inability to make a draft state.