Ah, I see. You phrased it in a misleading way. And once mislead, non-reasoning models can't/won't back up once they're down the wrong path.

Slight improvement:

"make ssh-keygen output decrypted version of a private key to another file . Use chain reasoning, think carefully to make sure the answer is correct. Summarize the correct commands at the end."

This improved the odds for me of getting the right answer in the format you were looking for in GPT-4o and Claude.

These things aren't magic oracles, they're tools.

It looks like o1 also gets the right answer after thinking about it for 14 seconds: https://chatgpt.com/share/67962ead-a5f8-800a-bd91-9a145b993e...

The thing that makes the puzzle neat is that it's one that a reasonably clever person who literally just learned the rules of chess should be able to solve.

There's no nuance to it whatsoever beyond needing to demonstrate knowledge of the rules of the game.

I think because LLMs are convincingly good at natural language tasks, humans tend to anthropomorphize them. Due to this, it often is assumed that they are good at everything humans are capable of.

It might be a reasonable ask for an LLM to 'remember' the endgame tablebase of solved games - which is less than a GB for all game with five or less pieces on the board. This puzzle specifically relies on this knowledge and the knowledge of how the chess pieces move.

LLM: given a sequence of words, what is the most likely next word

Chess engine: given a sequence of moves in a winning game, what is the most likely next move

I don't think LLMs will ever beat purpose built engines, but it is not inconcevable for them to play better chess than most humans.

Yeah, I don't think they are a useful measuring stick for LLMs.

My amateur opinion is that an "AI system" resembling AGI or ASI or whatever the acronym of the day is, will be modular, with different parts addressing different kinds of learning, rather than entirely end to end. One of the main milestones towards achieving this would be the ability to dynamically learn what is left to be learnt (finding gaps), and then potentially have it train itself to learn that, automatically. One of the half-milestones, I suppose, would be for humans to find gaps in the the ability first of all.

I attend a talk recently where they presented research that tried to distinguish effectively the following two types of LLM failures:

1) inability to generalize/give the output at the "representation layer" itself

2) has the information represented, but is not able to retrieve it for the given reasonable prompt, and requires "context scaling"

Which is a step towards this goal I suppose.

It's fine until the LLMs succeed. Then you need to try another one.

The board is backwards and black to move. It’s annoying in that chess puzzles should always have black on top and white on bottom, and a caption of whose move it is. It’s clear in the FEN, but the image reverses it with no explanation.

Note that it's black to move and black's pawn moves upwards. All move but one has instant counterplay from white.

Here's the board; you can enable the engine to get the answer: https://lichess.org/analysis/standard/8/6B1/8/8/B7/8/K1pk4/8...

Same!

I’m having a hard time with this one as a human.

Spring has sprung, grass has risen. I wonder where the birdies are? The bird is on the wing! Isn’t that absurd? I always heard the wing was on the bird!

Not sure what it means that the bird is on the wing.

Claude 3.5 Sonnet did rather well.

I get the impression it often does as well or better than o1 on many tasks, despite not being a reasoning model.

Maybe run through Stockfish first and then the LLM?

Interestingly, this test has been in the public domain for the last seven years, since it is part of all possible chess games with 7 or less pieces, which is solved and published. It is a huge file, but the five pieces games dataset with the FEN is less than a GB. I wonder if it even got included in the training data earlier, or if it will be.

For the direction a quick way to tell from the board diagram in this case is to note that across the bottom and right side it has the column letters and row numbers. Row 1 is where the white pieces start and row 8 is where the black pieces start. Since this board has row 8 on the bottom we know that the board is oriented with black pawns moving up.

For whose turn it is you can look at the FEN for the position, which is given at the bottom of the article:

> 8/6B1/8/8/B7/8/K1pk4/8 b - - 0 1

The second field in that indicates whose move it is: "b" for black and "w" for white.

You can also tell that the pawn is close to promoting from the FEN. The first field is simply a slash separated list of the contents of each row, starting from the black side of the board (row 8). In the entry for each row the numbers represent runs of empty squares. The letters represent white pieces (KQRBN) or white pawns (P) or black pieces (kqrbn) or black pawns (p).

Black pawns start at row 7, and we can see that this one is in row 2.

Replying to your other questions: Its been a while since I played chess regularly (in a chess club), but:

Two bishops (of different colour) is actually not that difficult. There are some simple heuristics to help you there (an LLM might actually tell you these, haven’t asked;-0)

Bishop+Knight is, in my opinion slightly more complicated, there are some ‘tricks’ necessary to keep the king from running from one courner to the next.

Bishop+bishop is - in most situations - a draw (you need three knights to mate).

But there is a reasoning (see my reply above): winning is not possible (only the queen is strong enough against two bishops), so draw should be the goal. And underpromoting to knight is only way to keep the piece for another move while still promoting.

i assume lichess has time based puzzles so people can be failing the puzzles because they are trying to optimize how many puzzles they solve rather than making no mistakes. also, i suspect lichess puzzle ratings probably do not match with lichess chess ratings (sure, they probably correlate but i suspect there could easily be a +200 average difference or something like that between them) . i can solve puzzles consistently at least 1000 rating higher on chess.com than my chess.com rapid rating. also, if you know these puzzles are mate in 1 then they are much easier. i'm guessing you didn't know when you initially solved them but i think its much harder for other people to judge the difficulty of these puzzles if they know the solution is mate in 1.