This is a step-by-step tutorial from DQN to Rainbow. Every chapter contains both of theoretical backgrounds and object-oriented implementation. Just pick any topic in which you are interested, and learn! You can run them directly in the cloud with molab — no local setup needed.

Built with marimo — a reactive Python notebook that runs as a pure .py file with better reproducibility, git diffs, and interactive UI.

Please feel free to open an issue or a pull-request if you have any idea to make it better. :)

If you want a tutorial for policy gradient methods, please see PG is All You Need.

Contents

1. DQN [GitHub] [Preview]
2. DoubleDQN [GitHub] [Preview]
3. PrioritizedExperienceReplay [GitHub] [Preview]
4. DuelingNet [GitHub] [Preview]
5. NoisyNet [GitHub] [Preview]
6. CategoricalDQN [GitHub] [Preview]
7. N-stepLearning [GitHub] [Preview]
8. Rainbow [GitHub] [Preview]
9. Rainbow IQN [GitHub] [Preview]

Click "Run in molab" on the preview page to open an interactive session where you can edit and run the notebook.

Prerequisites

# Install mise
curl https://mise.run | sh

# Clone the project
git clone https://github.com/Curt-Park/rainbow-is-all-you-need.git
cd rainbow-is-all-you-need

# Install Python + Create venv + Install Python packages
make init
make setup

How to Run

Run and experiment with any notebook:

make run notebook=01_dqn.py

Development

make format    # run the formatter
make lint      # run the linter

Related Papers

1. V. Mnih et al., "Human-level control through deep reinforcement learning." Nature, 518 (7540):529–533, 2015.
2. van Hasselt et al., "Deep Reinforcement Learning with Double Q-learning." arXiv preprint arXiv:1509.06461, 2015.
3. T. Schaul et al., "Prioritized Experience Replay." arXiv preprint arXiv:1511.05952, 2015.
4. Z. Wang et al., "Dueling Network Architectures for Deep Reinforcement Learning." arXiv preprint arXiv:1511.06581, 2015.
5. M. Fortunato et al., "Noisy Networks for Exploration." arXiv preprint arXiv:1706.10295, 2017.
6. M. G. Bellemare et al., "A Distributional Perspective on Reinforcement Learning." arXiv preprint arXiv:1707.06887, 2017.
7. R. S. Sutton, "Learning to predict by the methods of temporal differences." Machine learning, 3(1):9–44, 1988.
8. M. Hessel et al., "Rainbow: Combining Improvements in Deep Reinforcement Learning." arXiv preprint arXiv:1710.02298, 2017.
9. W. Dabney et al., "Implicit Quantile Networks for Distributional Reinforcement Learning." arXiv preprint arXiv:1806.06923, 2018.

Contributors

Thanks goes to these wonderful people (emoji key):

This project follows the all-contributors specification. Contributions of any kind welcome!