Chessvision.ai – Analyze chess position from websites, images or video

pkacprzak 7 years ago |

Hi, many thanks to innerspirit for posting this! I'm the author of the app and you can find discussions about it on reddit as well:

r/MachineLearning :https://www.reddit.com/r/MachineLearning/comments/b8jdho/p_d...

r/chess: https://www.reddit.com/r/chess/comments/b826h5/created_chess...

If you have any opinions and suggestions I'd love to hear them!

MisterOctober 7 years ago | |

Very cool! This'd be absolutely great for getting more value / use out of the pile of Everyman books that I [and, I'd hazard a guess, a solid chunk of chessplayers in general] have sitting on the shelf that don't get used as much as they should

0db532a0 7 years ago | | |

What are the best three books for beginners?

MisterOctober 7 years ago | | |

Great question! There are lots of great beginner books out there, but if I had to pick three:

1) "A First Book of Morphy" by del Rosario -- takes the very well-considered principles of GM Reuben Fine and illustrates them with [mostly] Paul Morphy's games. Of all the books I've ingested on the subject of chess, this one has stuck with me most. Even my kid loves it!

2) "Winning Chess Strategy for Kids" by Coakley -- it's not just for kids! And it's not just strategy! Covers fundamental tactical concerns such as pins, forks, etc in a straightforward way. Similar in some regards to Pandolfini's "Weapons of Chess" but pedagogically superior in my view.

3) "Silman's Complete Endgame Course" by Silman -- the old saw that one should study the endgame first is pretty true -- Silman is the best at teaching it -- man, if I had a dollar for every game I was winning in the middlegame and then lost in the endgame, I'd have... well, a lot of dollars

cheater extra: 3.5) "Bobby Fischer Teaches Chess" -- the old standby -- this is a tactics book first and foremost, and the main value of it is that it gives you puzzles of gradually increasing complexity so that you can really feel your comprehension improving and say to your self, "I'm getting it!" Truth is that getting discouraged is the thing that stalls or scares off many / most novice chessplayers

When you start getting up in the 1100-1300 range, pick up "My System" by Nimzowitsch and "How to Reassess your Chess" by Silman, "Soviet Middlegame Technique" by Romanovsky -- and after then, start learning openings in more detail. The biggest mistake most beginners make, including me, is building up a repertoire of openings before having a solid grasp of the fundamentals. Truth is, below 1300, most players are "off book" by the ~tenth move anyway, so learning the intricacies of the Nimzo-Indian isn't gonna do a person much good at that stage.

0db532a0 7 years ago | | |

Thanks a lot. Appreciated.

humanfromearth 7 years ago | |

Yay! Cheating now will be so easy :) Is the engine in use stockfish?

pkacprzak 7 years ago | | |

It displays lichess board analysis as iframe http://lichess.org/analysis/ and this board uses stockfish by default.

smallgovt 7 years ago |

It’d be great to convert this into a mobile app that lets you take a photo and then loads it into a game analysis board. We would use this all the time at my local chess club. Sometimes we just want to know what the best move is and other times we want to save the game so we can continue it later.

I would pay $10 or $1.99 a month for this.

Food for thought!

rtkwe 7 years ago | |

This would be much much harder with a real board with 3D pieces.

smallgovt 7 years ago | | |

I see. Well, one thing that may help is that 95% of chessboards (at least at the clubs I've visited) use the same piece set: https://www.amazon.com/WE-Games-Complete-Tournament-Chess/dp...

Also, a 10% error rate wouldn't be a big deal as long as it was easy to correct the position within a few seconds.

rtkwe 7 years ago | | |

That makes it vaguely possible yeah but going from a flat on 2D grid to a 3D set of pieces at an angle is a whole different set of problems.

zeptomu 7 years ago | | |

Although it is more difficult in 3D, it is still a very solvable problem, that won't work always, but in 99% of cases if the camera angle is not very low.

Indeed, the checkerboard itself gives valuable information about the pose of the checkerboard, it is even used for calibration in multi-view geometry: https://en.wikipedia.org/wiki/Chessboard_detection

tgarv 7 years ago | | |

I wonder how much easier it would be if you narrow down the likely set of pieces for each square. There are obviously some positions for some pieces that are invalid (bishop on the wrong colored square), but there are probably a lot of other positions that are so uncommon that they could be discounted.

levesque 7 years ago | | |

What would make it easier IMHO is to make it a top-down version only. Take a photo from the top, then the program breaks down the board into 8x8 squares, feeds each square into a classification algorithm that you will train on a bunch of hand labeled images. Fine tune the model as you gather more data.

rtkwe 7 years ago | | |

Probably not much. How many are actually impossible if you factor in pawn promotion though? The only one that comes to mind immediately for any individual piece is pawns on their own first row and a board without two kings.

darepublic 7 years ago | | |

Standard object detection can't work here?

rtkwe 7 years ago | | |

Have to detect and accurately place them in space to map them to the board.

jonwachob91 7 years ago |

Just to clarify an important detail that pkacprzak (the author) has been asked many times now by the streamers he is using the images of (GoldDustTori and GM Ben Finegold) - NONE of the streamers actually used this app during a game, as that would be considered outside assistance and cheating. The app is intended for viewers to follow along, not to enable cheating (even though the cheaters will use it for that purpose).

chess93 7 years ago |

One feature that would be useful for me would be the ability to convert a video to a PGN of the game. This could, for example, be used to strip game data from streamed chess en masse and then analyze the chess games in aggregate.

epiceric 7 years ago |

Maybe it's just me, but I'd be wary of any browser extension that captures and analyzes my screen over-the-wire, especially if it is closed-source like this one.

Not to say that this is necessarily malicious. But I personally wouldn't recommend this to anybody as of right now, unless you want to risk leaking potentially sensitive data to a third-party.

EGreg 7 years ago |

I thought they meant like in vivo in 3D, such as the scene in Independence Day

novalis78 7 years ago | |

That would rock for chess tournaments. Webcam next to the board and run analysis for the audience...

owldimoon 7 years ago | | |

Hey! I'm working on exactly this same problem for an undergraduate Computer Vision course right now. It's not going well! Chess piece recognition is hard. I definitely think the electronics approach described in the other comment is a much more reliable way to go.

joemi 7 years ago | | |

If you ever do get it working, I think it'll be fun to push it to over-recognize chess games everywhere, like the deepdream stuff over-recognizing animals (or whatever) everywhere.

danielvf 7 years ago | | |

Sounds like a blast!

Are you restricted to to a single game piece set?

This might be hugely easier if you can keep state between turn analyses. IE, if a white rook disappears from A5 and some unidentified white piece appears at C5, then it’s probably the rook.

Ayeceedee 7 years ago | | |

We weren't given many constraints related to the chess piece recognition itself. The course instead asks us to implement a CV research paper, and we chose an existing research project which focused on chess piece recognition.

That lack of constraints led us into running face first into issues of generalisation and variability within datasets. As in, exactly what you allude to with limiting the piece sets.

I think in my undergraduate naivety my aspirations were too high with what could reasonably be accomplished. I've spent a lot of time trying to improve an aspect of the project that really didn't need to be improved, which prevented meaningful progress.

Now finals are coming up and I feel terribly stressed. Having trouble functioning. Brain fog, etc. I feel so sad right now.

EDIT: I keep forgetting my password so apparently I have multiple throwaway now. Sorry.

yeldarb 7 years ago | | |

What challenges are you running into? What sort of data augmentation are you doing?

psuter 7 years ago | | |

We built a barely working version of that during a hackathon a while back: https://github.com/chesseye/chesseye (README has a link to a video).

It takes a lot of shortcuts, works with just the right lightning, etc., but worked great as a proof-of-concept :)

We got away with not identifying the pieces by just detecting the color, assuming the game started from the initial position, and assuming only legal moves (the whole game is unambiguous using these assumptions).

It's all old-school computer vision with hand-written features, and I'm pretty confident there is tons of low-hanging fruit, but who has the time.

pkacprzak 7 years ago | | |

If I understand correctly (watched the video) you start from the initial position and track moves that are made in order to update the position to reflect the move, is this correct?

psuter 7 years ago | | |

That's correct. If you're curious about the architecture: the vision part detects a chessboard, then corrects the perspective and restricts the image to just the square of the board, then looks at each square and has some simple thresholds to decide if it's occupied by a piece, and of what color. From there, the camera is treated as a black box sensor that continuously streams two 64-bit masks, for where it thinks it sees white and black pieces. There is a second program (controller) that turns that stream into a stream of chess positions (and a Unix pipe in between). The sensor is faulty of course and the controller has logic for ignoring bits from the mask where there cannot possibly be pieces etc.

novalis78 7 years ago | | |

That’s very cool, thanks for sharing!

pkacprzak 7 years ago | | |

actually, most high-level tournaments use electronic dgt boards to solve this problem in a more flawless way

bluGill 7 years ago | | |

Those boards are ~$1000 each. High level tournaments use them, but low level tournaments don't unless some rich person has bought the club a bunch. Most tournaments are low level and are designed not to turn a profit: the entry fees either pay for the room rent, or go to the prize winnings: the more people who enter the more prize money there is.

MisterOctober 7 years ago | | |

Yeah, the DGT board is perfect for on-the-fly tournament analysis -- but a computer vision tool would be great for times when you don't have access to the/a DGT board or its data

Set one up at the park and get some insight on those hustled blitz games!

heinrichhartman 7 years ago |

Pretty sure this will be (ab-)used for cheating in online-chess.

pkacprzak 7 years ago | |

There are already available apps that allow cheating in more "convenient" (if I can even say that) way for cheaters. They analyze DOM and display the best move etc. without any latency and heavy processing

elcomet 7 years ago | |

Isn't it much easier to run the moves yourself on another board with computer help?

jczhang 7 years ago | | |

not if you're playing bullet!

mcs_ 7 years ago | |

will be nice if chess.com (and others) could tell you that your opponent is "chessvisioning".

typo_hunter 7 years ago |

Small typo on the front page: "Postition From a Video" -> "Position From a Video"

pkacprzak 7 years ago | |

Thanks a lot, I fixed the typo

Romanulus 7 years ago |

Should have called it CheckMate!

hgasimov 7 years ago |

Seems cool. Does it use a server or it does all the calculations in the browser?

pkacprzak 7 years ago | |

Server side

unholiness 7 years ago | | |

This is kind of a bummer for privacy reasons. Sending your application screenshots of my browser feels not-great.

To be honest, I trust you, even as a random stranger on hacker news. But I don't trust that you'll never sell this, and I don't trust whoever you you might sell it to.

pkacprzak 7 years ago | | |

I understand your concerns and would like to clarify any potential doubts. The client app (the extension itself) has only currentTab permission which means that it has access only the tab where user opened the extension and it takes a screenshot only when a user explicitly clicks Scan button. The source code of the extension is completely public - everyone can unpack it and verify that indeed it does exactly that. So this actually reduces to the same functionality as for example using any photo app with cloud processing - i.e. you take the photo/screenshot explicitly and no other information is sent to the server.