Reinforcement Learning Resources

My personal collection of reinforcement learning resources.

Most of the content is in subject folders such as alphago and model-based.

Textbooks and Courses

Sutton & Barto - Reinforcement Learning: An Introduction - 2nd Edition - highly awaited second edition released in 2019

UCL Lectures - David Silver (Head of Reinforcement Learning at DeepMind) - slides - 10 lecture videos

Open AI Spinning Up in Deep Reinforcement Learning - notes - lecture

Getting Started Coding

To train a reinforcement learning agent you need three things - an environment (look at Open AI gym), an agent and code to run the experiments.

Part of learning how to use reinforcement learning is gaining familiarity with how quickly an agent should learn on a toy environment.

Three useful environments are:

• Cartpole - discrete action space
• Pendulum - continuous action space
• MountainCar - difficult exploration problem

Start out building simple agents and code to run experiments. A list of agents to work through is given below:

• dynamic programming or value/policy iteration,
• cross entropy method (optional),
• REINFORCE,
• DQN,
• PPO,
• SAC.

Hints:

• Don't make the common mistake of building an environment and agent at the same time! You won't be sure where the problem is when debugging.
• Run experiments over multiple random seeds - expect to see variance in learning across only these random seeds.
• Reinforcement learning is sample inefficient - especially when all experiments need to be done across at least 3 random seeds.

More Resources

If you like this check out:

• ml-resources,
• programming resources,
• rl-course.