AlphaZero[1]https://arxiv.org/abs/1712.01815 is one of those papers that lives in my head rent-free. I’m fascinated by the idea that you can solve seemingly difficult problems by just throwing enough compute at a problem with the right kind of objective – even if Yoshua Bengio thinks reinforcement learning is evil[2]https://www.youtube.com/watch?v=PZqDFs2sbiY&t=3687s.
I wanted to see how far I could push a tiny neural network at chess if it had to figure everything out from scratch.
So I built ZeroChess[3]https://github.com/tasercake/zerochess, a minimal chess learner inspired by AlphaZero’s techniques.
The setup
Not trying to beat Stockfish here. My goal is just to understand the core of what makes learning from self-play so powerful by implementing it from scratch.
To that end, we need:
A rules engine
If you wish to teach a bot to play chess, you must first invent the universe[4]https://www.youtube.com/watch?v=BkHCO8f2TWs.
For our purpose the universe consists of the board, the pieces, and the rules. This is non-negotiable and I definitely couldn’t just have used the chess crate.
This was a pain, but the upside is I get to decide exactly how the game state is represented and structure it in a way that’s easier to pass down to the neural network down the line.
The model
A tiny[5]~3.1M param CNN neural network to predict the ‘value’ of the current position and a probability distribution over all possible moves (including illegal ones).
Monte-Carlo Tree Search
Monte Carlo Tree Search (MCTS) guided by our neural network, à la AlphaZero. The network provides move probabilities and a position evaluation, and MCTS uses those to explore the game tree.
The training loop
- Self-play generates training data
- Network trains on it and gets better
- Better network generates better training data
Rinse and repeat.
The method
How will we know if it works?
We track the win-rate against known opponents! Elo-like tracking across training checkpoints
- Win rate vs a random legal-move player
- Win rate vs a shallow minimax bot
Results
After 3 full days of self-play game generation and training on my M4 MacBook Pro, ZeroChess beats an opponent that plays a completely random legal move each turn.
