MuZero

MuZero is a reinforcement learning algorithm developed by DeepMind that learns a model of an environment for planning without being given its rules or dynamics. It was introduced in a 2019 preliminary paper and described in a 2020 Nature paper. MuZero was evaluated on Go, chess, shogi, and Atari games, where it achieved performance comparable to or better than previous systems in the reported experiments.


MuZero was trained via self-play, with no access to rules, opening books, or endgame tablebases. The trained algorithm used the same convolutional and residual architecture as AlphaZero, but with 20 percent fewer computation steps per node in the search tree.

History

MuZero really is discovering for itself how to build a model and understand it just from first principles.

— David Silver, DeepMind, Wired

On November 19, 2019, the DeepMind team released a preprint introducing MuZero.

Derivation from AlphaZero

MuZero (MZ) is a combination of the high-performance planning of the AlphaZero (AZ) algorithm with approaches to model-free reinforcement learning. The combination allows for more efficient training in classical planning regimes, such as Go, while also handling domains with much more complex inputs at each stage, such as visual video games.

MuZero was derived directly from AZ code, sharing its rules for setting hyperparameters. Differences between the approaches include:

  • AZ's planning process uses a simulator. The simulator knows the rules of the game. It has to be explicitly programmed. A neural network then predicts the policy and value of a future position. Perfect knowledge of game rules is used in modeling state transitions in the search tree, actions available at each node, and termination of a branch of the tree. MZ does not have access to the rules, and instead learns one with neural networks.
  • AZ has a single model for the game (from board state to predictions); MZ has separate models for representation of the current state (from board state into its internal embedding), dynamics of states (how actions change representations of board states), and prediction of policy and value of a future position (given a state's representation).
  • MZ's hidden model may be complex, and it may turn out it can host computation; exploring the details of the hidden model in a trained instance of MZ is a topic for future exploration.
  • MZ does not expect a two-player game where winners take all. It works with standard reinforcement-learning scenarios, including single-agent environments with continuous intermediate rewards, possibly of arbitrary magnitude and with time discounting. AZ was designed for two-player games that could be won, drawn, or lost.

Comparison with R2D2

The previous state of the art technique for learning to play the suite of Atari games was R2D2, the Recurrent Replay Distributed DQN.

MuZero surpassed both R2D2's mean and median performance across the suite of games, though it did not do better in every game.

Training and results

MuZero used 16 third-generation tensor processing units (TPUs) for training, and 1000 TPUs for selfplay for board games, with 800 simulations per step and 8 TPUs for training and 32 TPUs for selfplay for Atari games, with 50 simulations per step.

AlphaZero used 64 second-generation TPUs for training, and 5000 first-generation TPUs for selfplay. As TPU design has improved (third-generation chips are 2x as powerful individually as second-generation chips, with further advances in bandwidth and networking across chips in a pod), these are comparable training setups.

R2D2 was trained for 5 days through 2M training steps.

Initial results

MuZero matched AlphaZero's performance in chess and shogi after roughly 1 million training steps. It matched AZ's performance in Go after 500,000 training steps and surpassed it by 1 million steps. It matched R2D2's mean and median performance across the Atari game suite after 500 thousand training steps and surpassed it by 1 million steps, though it never performed well on 6 games in the suite.

MuZero was viewed as a significant advancement over AlphaZero, and a generalizable step forward in unsupervised learning techniques. The work was seen as advancing understanding of how to compose systems from smaller components, a systems-level development more than a pure machine-learning development.

While only pseudocode was released by the development team, Werner Duvaud produced an open source implementation based on that.

MuZero has been used as a reference implementation in other work, for instance as a way to generate model-based behavior.

In late 2021, a more efficient variant of MuZero was proposed, named EfficientZero. It "achieves 194.3 percent mean human performance and 109.0 percent median performance on the Atari 100k benchmark with only two hours of real-time game experience".

In early 2022, a variant of MuZero was proposed to play stochastic games (for example 2048, backgammon), called Stochastic MuZero, which uses afterstate dynamics and chance codes to account for the stochastic nature of the environment when training the dynamics network.

In February 2022, DeepMind reported a first application of MuZero to a real-world task, collaborating with YouTube to improve video compression in the open-source VP9 codec. A version called MuZero-RC replaced VP9's default rate-control mechanism, selecting the quantisation parameter for each frame, and achieved an average 4% bitrate reduction across a diverse set of videos without quality degradation.

See also

References

  1. ↑ Schrittwieser, Julian; Antonoglou, Ioannis; Hubert, Thomas; Simonyan, Karen; Sifre, Laurent; Schmitt, Simon; Guez, Arthur; Lockhart, Edward; Hassabis, Demis; Graepel, Thore; Lillicrap, Timothy (23 December 2020). "Mastering Atari, Go, chess and shogi by planning with a learned model". Nature. 588 (7839): 604–609. arXiv:1911.08265. doi:10.1038/s41586-020-03051-4.
  2. ↑ Schrittwieser, Julian; Antonoglou, Ioannis; Hubert, Thomas; Simonyan, Karen; Sifre, Laurent; Schmitt, Simon; Guez, Arthur; Lockhart, Edward; Hassabis, Demis; Graepel, Thore; Lillicrap, Timothy (2020). "Mastering Atari, Go, chess and shogi by planning with a learned model". Nature. 588 (7839): 604–609. arXiv:1911.08265. Bibcode:2020Natur.588..604S. doi:10.1038/s41586-020-03051-4. PMID 33361790. S2CID 208158225.
  3. ↑ "What AlphaGo Can Teach Us About How People Learn". Wired. ISSN 1059-1028. Retrieved 2020-12-25.
  4. ↑ Silver, David; Hubert, Thomas; Schrittwieser, Julian; Antonoglou, Ioannis; Lai, Matthew; Guez, Arthur; Lanctot, Marc; Sifre, Laurent; Kumaran, Dharshan; Graepel, Thore; Lillicrap, Timothy; Simonyan, Karen; Hassabis, Demis (5 December 2017). "Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm". arXiv:1712.01815 [cs.AI].
  5. ↑ Kapturowski, Steven; Ostrovski, Georg; Quan, John; Munos, Remi; Dabney, Will. RECURRENT EXPERIENCE REPLAY IN DISTRIBUTED REINFORCEMENT LEARNING. ICLR 2019 – via Open Review.
  6. ↑ Shah, Rohin (27 November 2019). "[AN #75]: Solving Atari and Go with learned game models, and thoughts from a MIRI employee - LessWrong 2.0". www.lesswrong.com. Retrieved 2020-06-07.
  7. ↑ Wu, Jun. "Reinforcement Learning, Deep Learning's Partner". Forbes. Retrieved 2020-07-15.
  8. ↑ "Machine Learning & Robotics: My (biased) 2019 State of the Field". cachestocaches.com. Retrieved 2020-07-15.
  9. ↑ Duvaud, Werner (2020-07-15), werner-duvaud/muzero-general, retrieved 2020-07-15
  10. ↑ van Seijen, Harm; Nekoei, Hadi; Racah, Evan; Chandar, Sarath (2020-07-06). "The LoCA Regret: A Consistent Metric to Evaluate Model-Based Behavior in Reinforcement Learning". arXiv:2007.03158 [cs.stat].
  11. ↑ Ye, Weirui; Liu, Shaohuai; Kurutach, Thanard; Abbeel, Pieter; Gao, Yang (2021-12-11). "Mastering Atari Games with Limited Data". arXiv:2111.00210 [cs.LG].
  12. ↑ Antonoglou, Ioannis; Schrittwieser, Julian; Ozair, Serjil; Hubert, Thomas; Silver, David (2022-01-28). "Planning in Stochastic Environments with a Learned Model". Retrieved 2023-12-12.
  13. ↑ "MuZero's first step from research into the real world". DeepMind. 11 February 2022. Retrieved 20 June 2026.