Chess trainer

He pushed the chess board across the sofa. “A game of chess?” my 12-year-old nephew asked. I acquiesced, and we finished with one win each. “Finally! I managed to win my first game against an adult!” He jumped for joy and loudly exclaimed. He had won fair and square. My small mistakes had snowballed into a loss. I was happy for him, but the ‘L’ was seared into my mind. How could I actually improve at this game?

Over the next few days, my mind churned as I played game after game on Chess.com. Moments of "hmm I'm unsure of what to do next, no pieces are threatened for now" would often be turned against me. I would then tap Chess.com’s ‘Review Game’ button with anticipation, hoping another complimentary review was available. The review highlighted moves where I lost an advantage, through a mistake or a missed opportunity. But I still struggled to understand what I should have been trying to do before that mistake.

And then I thought to myself, “Why not just vibe-code my own trainer?” And vibe I did.

The Perfect Player + a tinge of Chaos

Two problems kept coming up:

• How could I choose a move when several looked okay, but one might leave me walking into a trap or a fork a few moves later?
• How could I distinguish an advantage I had created from one my opponent had handed me?

I wanted the trainer’s score to help me investigate those questions. My starting idea was the ‘Perfect Player’: after each ply (one move by either player), take the resulting position and let two strong players continue the game, with some variability with each move (instead of just choosing the 'best' move):

1. Ask Stockfish for its best move + up to 3 strong alternatives.
2. If there are no alternatives: Choose the best move.
3. Else: Choose the best move 90% of the time. Share the remaining 10% among those alternatives, giving higher-scoring moves more weight.

For that occasional exploration, I used the weights (1 + score / 100)², with the score measured in centipawns. This favours higher evaluations while still giving the other qualifying moves a chance. The Win/Loss/Draw outcome is then turned into a score:

• A win gives 1 point.
• A draw gives 0.5 points.
• A loss gives 0 points.

Some continuations hit a time or move limit before finishing and we exclude them. Then, repeat the same routine a bunch of times.

Some Metrics

With the outcomes collected, obvious metric we want is:

Expected score = sum_of_points ÷ completed trials

This estimates the score from a given position. But with a limited number of trials, how much confidence should I place in that average? I wanted the graph to show that uncertainty too. More completed trials should generally give me a narrower range. Here is how I calculated that range:

1. 'Binarize' the outcomes:
   
   1) Win vs Did-not-win
   
   2) Lose vs Did-not-lose

2. Apply Wilson Confidence Intervals to each case to get their respective intervals:
   
   win_upper_boundary, win_lower_boundary
   
   lose_upper_boundary, lose_lower_boundary

   *We use Wilson Confidence Intervals because the intervals should be within [0,1] ie does not make sense to have a "probability_lower_boundary(win) < 0" or "probability_upper_boundary(win) > 1"

3. Construct the joint-boundaries:
   
   score_lower_boundary = 0.5(1 + probability_lower_boundary(win) - probability_upper_boundary(lose))
   
   score_upper_boundary = 0.5(1 + probability_upper_boundary(win) - probability_lower_boundary(lose))

   We can do this because:
   
   E[score] = 1.0 * p(win) + 0.5 * p(draw) = p(win) + 0.5 * (1 - p(win) - p(loss)) = 0.5(1 + p(win) - p(loss))

A Quick Look

Choose one of my games from the dropdown. Me means my side of the board; Opponent switches to the other player's perspective.

1. Click inside the board, then press j to go back or k to play the next move. The Previous and Next buttons do the same thing, one ply at a time.
2. Follow the Decision Impact line to see how the estimated score changes. The shaded band shows uncertainty in that estimate. Click a point to jump to its position.
3. Use Most impactful moves to find the largest swings. Its Before and After buttons let you inspect what changed.

Open the replay in a full view

Check-mate?

My curiosity went beyond the obvious blunders (hoping that with more games, my blundering rate would go down). I wanted to understand how an advantage arose: had I created an opportunity, or had my opponent handed me one?

With the ply-score series, I think I have a good-enough approximation of the pulse of the game. When it shifts, I have somewhere to look: rewind, inspect the position, and try another continuation. What changed? What did I overlook? Could I recognise that opportunity next time?

Whether this makes me a better player remains to be seen. For now, those “hmm, what should I do next?” positions have become something I can return to and investigate.