Awesome Deep RL

A curated list of awesome Deep Reinforcement Learning resources.

Contents

• Libraries
• Benchmark Results
• Environments
• Competitions
• Timeline
• Books
• Tutorials
• Blog

Libraries

• AgileRL - A Deep Reinforcement Learning library focused on improving development by introducing RLOps - MLOps for reinforcement learning.
• Berkeley Ray RLLib - An open-source library for reinforcement learning that offers both high scalability and a unified API for a variety of applications.
• Berkeley Softlearning - A reinforcement learning framework for training maximum entropy policies in continuous domains.
• Catalyst - Accelerated DL & RL.
• ChainerRL - A deep reinforcement learning library built on top of Chainer.
• DeepMind Acme - A research framework for reinforcement learning.
• DeepMind OpenSpiel - A collection of environments and algorithms for research in general reinforcement learning and search/planning in games.
• DeepMind TRFL - TensorFlow Reinforcement Learning.
• DeepRL - Modularized Implementation of Deep RL Algorithms in PyTorch.
• DeepX machina - A library for real-world Deep Reinforcement Learning which is built on top of PyTorch.
• d3rlpy - An offline deep reinforcement learning library.
• Facebook ELF - A platform for game research with AlphaGoZero/AlphaZero reimplementation.
• Facebook ReAgent - A platform for Reasoning systems (Reinforcement Learning, Contextual Bandits, etc.)
• garage - A toolkit for reproducible reinforcement learning research.
• Google Dopamine - A research framework for fast prototyping of reinforcement learning algorithms.
• Google TF-Agents - TF-Agents is a library for Reinforcement Learning in TensorFlow.
• K-Scale Labs - ksim - A modular and easy-to-use framework for training policies in simulation.
• K-Scale Labs - ksim-gym - K-Sim Gym: Making robots useful with RL. Built on top of K-Sim.
• MAgent - A Platform for Many-agent Reinforcement Learning.
• Maze - Application-oriented deep reinforcement learning framework addressing real-world decision problems.
• MushroomRL - Python library for Reinforcement Learning experiments.
• NervanaSystems coach - Reinforcement Learning Coach by Intel AI Lab.
• OpenAI Baselines - High-quality implementations of reinforcement learning algorithms.
• OpenRL - An open-source general reinforcement learning research framework.
• pytorch-a2c-ppo-acktr-gail - PyTorch implementation of Advantage Actor Critic (A2C), Proximal Policy Optimization (PPO), Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation (ACKTR) and Generative Adversarial Imitation Learning (GAIL).
• pytorch-rl - Model-free deep reinforcement learning algorithms implemented in Pytorch.
• reaver - A modular deep reinforcement learning framework with a focus on various StarCraft II based tasks.
• RLgraph - Modular computation graphs for deep reinforcement learning.
• RLkit - Reinforcement learning framework and algorithms implemented in PyTorch.
• rlpyt - Reinforcement Learning in PyTorch.
• RLtools - The fastest deep reinforcement learning library for continuous control, implemented in pure, dependency-free C++ (Python bindings available as well).
• skrl - Modular reinforcement learning library (on PyTorch and JAX) with support for NVIDIA Isaac Gym, Omniverse Isaac Gym and Isaac Lab.
• SLM Lab - Modular Deep Reinforcement Learning framework in PyTorch.
• Stable Baselines - A fork of OpenAI Baselines, implementations of reinforcement learning algorithms.
• TensorForce - A TensorFlow library for applied reinforcement learning.
• Tianshou - Tianshou (天授) is a reinforcement learning platform based on pure PyTorch.
• TorchRL - An open-source Reinforcement Learning (RL) library for PyTorch.
• UMass Amherst Autonomous Learning Library - A PyTorch library for building deep reinforcement learning agents.
• Unity ML-Agents Toolkit - Unity Machine Learning Agents Toolkit.
• vel - Bring velocity to deep-learning research.
• DI-engine - A generalized decision intelligence engine. It supports various Deep RL algorithms.

Benchmark Results

• DeepMind bsuite
• OpenAI baselines-results
• OpenAI Baselines
• OpenAI Spinning Up
• ray rl-experiments
• rl-baselines-zoo
• SLM Lab
• vel
• What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study
• yarlp

Environments

• AI2-THOR - A near photo-realistic interactable framework for AI agents.
• Animal-AI Olympics - An AI competition with tests inspired by animal cognition.
• Berkeley rl-generalization - Modifiable OpenAI Gym environments for studying generalization in RL.
• BTGym - Scalable event-driven RL-friendly backtesting library. Build on top of Backtrader with OpenAI Gym environment API.
• Carla - Open-source simulator for autonomous driving research.
• CuLE - A CUDA port of the Atari Learning Environment (ALE).
• Deepdrive - End-to-end simulation for self-driving cars.
• DeepMind AndroidEnv - A library for doing RL research on Android devices.
• DeepMind DM Control - The DeepMind Control Suite and Package.
• DeepMind Lab - A customisable 3D platform for agent-based AI research.
• DeepMind pycolab - A highly-customisable gridworld game engine with some batteries included.
• DeepMind PySC2 - StarCraft II Learning Environment.
• DeepMind RL Unplugged - Benchmarks for Offline Reinforcement Learning.
• Facebook EmbodiedQA - Train embodied agents that can answer questions in environments.
• Facebook Habitat - A modular high-level library to train embodied AI agents across a variety of tasks, environments, and simulators.
• Facebook House3D - A Rich and Realistic 3D Environment.
• Facebook natural_rl_environment - natural signal Atari environments, introduced in the paper Natural Environment Benchmarks for Reinforcement Learning.
• Google Research Football - An RL environment based on open-source game Gameplay Football.
• GVGAI Gym - An OpenAI Gym environment for games written in the Video Game Description Language, including the Generic Video Game Competition framework.
• gym-doom - Doom environments based on VizDoom.
• gym-duckietown - Self-driving car simulator for the Duckietown universe.
• gym-gazebo2 - A toolkit for developing and comparing reinforcement learning algorithms using ROS 2 and Gazebo.
• gym-ignition - Experimental OpenAI Gym environments implemented with Ignition Robotics.
• gym-idsgame - An Abstract Cyber Security Simulation and Markov Game for OpenAI Gym
• gym-super-mario - 32 levels of original Super Mario Bros.
• Gym4ReaL - A Gymnasium-based benchmarking suite for testing reinforcement learning algorithms on real-world scenarios, including water management, energy management in microgrids, financial trading, and more.
• Holodeck - High Fidelity Simulator for Reinforcement Learning and Robotics Research.
• home-platform - A platform for artificial agents to learn from vision, audio, semantics, physics, and interaction with objects and other agents, all within a realistic context
• ma-gym - A collection of multi agent environments based on OpenAI gym.
• mazelab - A customizable framework to create maze and gridworld environments.
• Meta-World - An open source robotics benchmark for meta- and multi-task reinforcement learning.
• Microsoft AirSim - Open source simulator for autonomous vehicles built on Unreal Engine / Unity, from Microsoft AI & Research.
• Microsoft Jericho - A learning environment for man-made Interactive Fiction games.
• Microsoft Malmö - A platform for Artificial Intelligence experimentation and research built on top of Minecraft.
• Microsoft MazeExplorer - Customisable 3D environment for assessing generalisation in Reinforcement Learning.
• Microsoft TextWorld - A text-based game generator and extensible sandbox learning environment for training and testing reinforcement learning (RL) agents.
• MineRL - MineRL Competition for Sample Efficient Reinforcement Learning.
• MuJoCo - Advanced physics simulation.
• OpenAI Coinrun - Code for the environments used in the paper Quantifying Generalization in Reinforcement Learning.
• OpenAI Gym Retro - Retro Games in Gym.
• OpenAI Gym Soccer - A multiagent domain featuring continuous state and action spaces.
• OpenAI Gym - A toolkit for developing and comparing reinforcement learning algorithms.
• OpenAI Multi-Agent Particle Environment - A simple multi-agent particle world with a continuous observation and discrete action space, along with some basic simulated physics.
• OpenAI Neural MMO - A Massively Multiagent Game Environment.
• OpenAI Procgen Benchmark - Procedurally Generated Game-Like Gym Environments.
• OpenAI Roboschool - Open-source software for robot simulation, integrated with OpenAI Gym.
• OpenAI RoboSumo - A set of competitive multi-agent environments used in the paper Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments.
• OpenAI Safety Gym - Tools for accelerating safe exploration research.
• Personae - RL & SL Methods and Envs For Quantitative Trading.
• Pommerman - A clone of Bomberman built for AI research.
• pybullet-gym - Open-source implementations of OpenAI Gym MuJoCo environments for use with the OpenAI Gym Reinforcement Learning Research Platform
• PyGame Learning Environment - Reinforcement Learning Environment in Python.
• RLBench - A large-scale benchmark and learning environment.
• RLGym - A python API to treat the game Rocket League as an OpenAI Gym environment.
• RLTrader - A cryptocurrency trading environment using deep reinforcement learning and OpenAI's gym.
• RoboNet - A Dataset for Large-Scale Multi-Robot Learning.
• rocket-lander - SpaceX Falcon 9 Box2D continuous-action simulation with traditional and AI controllers.
• Stanford Gibson Environments - Real-World Perception for Embodied Agents.
• Stanford osim-rl - Reinforcement learning environments with musculoskeletal models.
• Unity ML-Agents Toolkit - Unity Machine Learning Agents Toolkit.
• UnityObstableTower - A procedurally generated environment consisting of multiple floors to be solved by a learning agent.
• VizDoom - Doom-based AI Research Platform for Reinforcement Learning from Raw Visual Information.
• RLCard - A research platform for reinforcement learning in card games.
• DouZero - A research platform for reinforcement learning in DouDizhu (Chinese poker).

Competitions

• AWS DeepRacer League 2019
• Flatland Challenge 2019
• Kaggle Connect X Competition 2020
• NeurIPS 2019: Animal-AI Olympics
• NeurIPS 2019: Game of Drones
• NeurIPS 2019: Learn to Move - Walk Around
• NeurIPS 2019: MineRL Competition
• NeurIPS 2019: Reconnaissance Blind Chess
• NeurIPS 2019: Robot open-Ended Autonomous Learning
• Unity Obstacle Tower Challenge 2019

Check AICrowd for the latest list of major RL competitions

Timeline

• 1947: Monte Carlo Sampling
• 1958: Perceptron
• 1959: Temporal Difference Learning
• 1983: ASE-ALE — the first Actor-Critic algorithm
• 1986: Backpropagation algorithm
• 1989: CNNs
• 1989: Q-Learning
• 1991: TD-Gammon
• 1992: REINFORCE
• 1992: Experience Replay
• 1994: SARSA
• 1999: Nvidia invented the GPU
• 2007: CUDA released
• 2012: Arcade Learning Environment (ALE)
• 2013: DQN
• 2015 Feb: DQN human-level control in Atari
• 2015 Feb: TRPO
• 2015 Jun: Generalized Advantage Estimation
• 2015 Sep: Deep Deterministic Policy Gradient (DDPG)
• 2015 Sep: DoubleDQN
• 2015 Nov: DuelingDQN
• 2015 Nov: Prioritized Experience Replay
• 2015 Nov: TensorFlow
• 2016 Feb: A3C
• 2016 Mar: AlphaGo beats Lee Sedol 4-1
• 2016 Jun: OpenAI Gym
• 2016 Jun: Generative Adversarial Imitation Learning (GAIL)
• 2016 Oct: PyTorch
• 2017 Mar: Model-Agnostic Meta-Learning (MAML)
• 2017 Jul: Distributional RL
• 2017 Jul: PPO
• 2017 Aug: OpenAI DotA 2 1:1
• 2017 Aug: Intrinsic Cusiority Module (ICM)
• 2017 Oct: Rainbow
• 2017 Oct: AlphaGo Zero masters Go without human knowledge
• 2017 Dec: AlphaZero masters Go, Chess and Shogi
• 2018 Jan: Soft Actor-Critic
• 2018 Feb: IMPALA
• 2018 Jun: Qt-Opt
• 2018 Nov: Go-Explore solved Montezuma’s Revenge
• 2018 Dec: AlphaZero becomes the strongest player in history for chess, Go, and Shogi
• 2019 Apr: OpenAI Five defeated world champions at DotA 2
• 2019 May: FTW Quake III Arena Capture the Flag
• 2019 Aug: AlphaStar: Grandmaster level in StarCraft II
• 2019 Sep: Emergent Tool Use from Multi-Agent Interaction
• 2019 Oct: Solving Rubik’s Cube with a Robot Hand
• 2020 Mar: Agent57 outperforms the standard human benchmark on all 57 Atari games
• 2020 Nov: AlphaFold for protein folding
• 2020 Dec: MuZero masters Go, chess, shogi and Atari without rules
• 2021 Aug: Generally capable agents emerge from open-ended play

Books

• Algorithms for Reinforcement Learning. Szepesvari et. al.
• An Introduction to Deep Reinforcement Learning. Francois-Lavet et. al.
• Deep Reinforcement Learning Hands-On. Lapan
• Deep Reinforcement Learning in Action. Zai & Brown
• Foundations of Deep Reinforcement Learning. Graesser & Keng
• Grokking Deep Reinforcement Learning. Morales
• Reinforcement Learning: An Introduction. Sutton & Barto.

Tutorials

• Andrew Karpathy Deep Reinforcement Learning: Pong from Pixels
• Arthur Juliani Simple Reinforcement Learning in Tensorflow Series
• Berkeley Deep Reinforcement Learning Course
• David Silver UCL Course on RL 2015
• Deep RL Bootcamp 2017
• DeepMind UCL Deep RL Course 2018
• DeepMind Learning Resources
• dennybritz/reinforcement-learning
• higgsfield/RL-Adventure-2
• higgsfield/RL-Adventure
• The Hugging Face Deep Reinforcement Learning Class 🤗
• MorvanZhou/Reinforcement Learning Methods and Tutorials
• OpenAI Spinning Up
• Sergey Levine CS294 Deep Reinforcement Learning Fall 2017
• Udacity Deep Reinforcement Learning Nanodegree
• Reinforcement Learning Fundamental
• PPOxFamily: DRL Tutorial Course

Blogs

• Alex Irpan
• Andrew Karpathy
• Berkeley AI Research
• Chris Olah
• David Ha
• DeepMind
• Distill
• Eric Jang
• Facebook AI
• Google AI
• Lilian Weng
• Matthew Rahtz
• OpenAI
• The Gradient
• Uber AI