RL Games: High performance RL library

Note: The next release will be 2.0.0 (unreleased). It migrates fully from gym to gymnasium and requires Python >= 3.11 (CI runs 3.11 and 3.12). EnvPool support has been restored: envpool >= 1.2.6 on Python 3.12+, with an envpool 1.2.5 fallback on Python 3.11 (NumPy 2.x, MuJoCo 3.x compatible; MyoSuite target randomization requires 1.2.6 and therefore Python >= 3.12).

Discord Channel Link

• https://discord.gg/hnYRq7DsQh

Papers and related links

• Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning: https://arxiv.org/abs/2108.10470
• DeXtreme: Transfer of Agile In-Hand Manipulation from Simulation to Reality: https://dextreme.org/ https://arxiv.org/abs/2210.13702
• Transferring Dexterous Manipulation from GPU Simulation to a Remote Real-World TriFinger: https://s2r2-ig.github.io/ https://arxiv.org/abs/2108.09779
• Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge? https://arxiv.org/abs/2011.09533
• Superfast Adversarial Motion Priors (AMP) implementation: https://twitter.com/xbpeng4/status/1506317490766303235 https://github.com/NVIDIA-Omniverse/IsaacGymEnvs
• OSCAR: Data-Driven Operational Space Control for Adaptive and Robust Robot Manipulation: https://cremebrule.github.io/oscar-web/ https://arxiv.org/abs/2110.00704
• EnvPool: A Highly Parallel Reinforcement Learning Environment Execution Engine: https://arxiv.org/abs/2206.10558 and https://github.com/sail-sg/envpool
• TimeChamber: A Massively Parallel Large Scale Self-Play Framework: https://github.com/inspirai/TimeChamber
• DextrAH-RGB: Visuomotor Policies to Grasp Anything with Dexterous Hands: https://dextrah-rgb.github.io/ https://arxiv.org/abs/2412.01791

Some results on the different environments

• NVIDIA Isaac Gym (legacy — use rl_games <= 1.6.5)

• Dextreme

• DexPBT

• MJLab (MuJoCo Lab) — quadruped and humanoid locomotion, dexterous manipulation

WujiHand in-hand cube reorientation (wuji-mjlab): on the unmodified task, rl_games trains to 19.1–20.1 goal reaches per episode (three seeds) vs 16.9 for the reference rsl-rl fork at the same training budget, reaches the reference's final score in 1.3–1.4 h instead of 2.1 h, and scales to 21.1 on 2 GPUs (reference score in 1.0 h). Full comparison, the training recipe and the stability notes in docs/MJLAB.md.

• Starcraft 2 Multi Agents
• DeepMind Control Suite
• EnvPool — high-throughput MuJoCo / Atari / DM Control vectorized envs
• MyoSuite — musculoskeletal control, envpool-vectorized
• Random Envs

SAC

SAC matches or exceeds published reference scores on MuJoCo continuous control (envpool, 1M frames, 3 seeds). Full table, plots and reproduction in docs/SAC_BENCHMARKS.md.

Humanoid keeps improving well past the standard 1M-frame budget — a single run extended to 5M frames reaches 7,066, about 40% above the reference mean.

Implemented in PyTorch

• PPO with the support of asymmetric actor-critic variant
• SAC
• Support of end-to-end GPU accelerated training pipelines: MJLab (MuJoCo Warp), Isaac Lab. Legacy Isaac Gym: rl_games <= 1.6.5
• Masked actions support
• Multi-agent training, decentralized and centralized critic variants
• Self-play

Implemented in Tensorflow 1.x (was removed in this version):

• Rainbow DQN
• A2C
• PPO

Quickstart: Colab in the Cloud

Explore RL Games quick and easily in colab notebooks:

• Mujoco training Mujoco gymnasium training example.
• MJLab training GPU-parallel robot locomotion with MJLab (MuJoCo Warp): trains a Unitree Go1 walking policy in minutes, renders it to video, probes command tracking. Local GPU required.
• MJLab training on Colab Same pipeline for Colab GPUs — L4/A100 recommended (T4 untested).
• Onnx discrete space export example with Cartpole
• Onnx continuous space export example with Pendulum
• Onnx continuous space with LSTM export example with Pendulum

Installation

For maximum training performance, PyTorch >= 2.2 with CUDA is recommended.

pip install rl-games

Or clone the repo and install the latest version from source:

pip install -e .

With optional extras (e.g. Atari, Mujoco, EnvPool):

pip install -e ".[atari,mujoco,envpool]"

Available extras: atari, mujoco, envpool. (pufferlib is no longer an extra: its current releases pin numpy<2 / old gymnasium, conflicting with this project's floors — pip install pufferlib --no-deps; the extra returns with pufferlib 4.0.)

For high-throughput vectorized MuJoCo / Atari / DM Control training, install the envpool extra and see docs/ENVPOOL.md.

Using uv (recommended)

uv is a fast Python package manager. To create a virtual environment and install rl_games:

# 3.12+ resolves envpool 1.2.6; a 3.11 venv falls back to envpool 1.2.5
uv venv --python 3.12
source .venv/bin/activate
uv pip install -e ".[mujoco,envpool]"

Citing

If you use rl-games in your research please use the following citation:

@misc{rl-games2021,
title = {rl-games: A High-performance Framework for Reinforcement Learning},
author = {Makoviichuk, Denys and Makoviychuk, Viktor},
month = {May},
year = {2021},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/Denys88/rl_games}},
}

Development setup

uv venv --python 3.11  # project floor (Isaac Lab); .[envpool] resolves envpool 1.2.5 here
source .venv/bin/activate
uv pip install -e ".[atari,mujoco]"

Training

NVIDIA Isaac Gym (legacy)

Isaac Gym Preview is end-of-life and not supported by current rl_games (it requires Python <= 3.8). To run legacy Isaac Gym / IsaacGymEnvs projects use the last supporting release, rl-games==1.6.5, and its documentation: https://github.com/Denys88/rl_games/tree/v1.6.5 — see docs/ISAAC_GYM.md. For current GPU-accelerated training see MJLab (MuJoCo Warp) and Isaac Lab.

Other

Atari Pong

python runner.py --train --file rl_games/configs/atari/ppo_pong.yaml
python runner.py --play --file rl_games/configs/atari/ppo_pong.yaml --checkpoint nn/PongNoFrameskip.pth

Experiment tracking

rl_games support experiment tracking with Weights and Biases.

python runner.py --train --file rl_games/configs/atari/ppo_breakout_torch.yaml --track
WANDB_API_KEY=xxxx python runner.py --train --file rl_games/configs/atari/ppo_breakout_torch.yaml --track
python runner.py --train --file rl_games/configs/atari/ppo_breakout_torch.yaml --wandb-project-name rl-games-special-test --track
python runner.py --train --file rl_games/configs/atari/ppo_breakout_torch.yaml --wandb-project-name rl-games-special-test -wandb-entity openrlbenchmark --track

Population Based Training

rl_games includes the DexPBT-lineage PBT observers (previously maintained downstream in IsaacGymEnvs and Isaac Lab), usable with any backend via rl_games.common.pbt — see docs/PBT.md.

Multi GPU

We use torchrun to orchestrate any multi-gpu runs.

torchrun --standalone --nnodes=1 --nproc_per_node=2 runner.py --train --file rl_games/configs/ppo_cartpole.yaml

Triton Kernels

When Triton is installed, rl_games automatically uses custom Triton kernels for performance-critical operations like GAE (Generalized Advantage Estimation). This replaces the Python for-loop with a single fused GPU kernel.

Triton is enabled by default. To disable:

RLG_NO_TRITON=1 python runner.py --train --file rl_games/configs/mujoco/ant.yaml

Run the benchmark to see speedups on your hardware:

python benchmarks/bench_triton_gae.py

Config Parameters

Custom network example:

simple test network
This network takes dictionary observation. To register it you can add code in your init.py

from rl_games.envs.test_network import TestNetBuilder 
from rl_games.algos_torch import model_builder
model_builder.register_network('testnet', TestNetBuilder)

simple test environment example environment

Additional environment supported properties and functions

Release Notes

1.6.5

• Added torch.compile support with configurable modes. Provides 10-40% performance improvement. Requires torch 2.2 or newer.
  • Default mode is reduce-overhead for balanced compilation time and runtime performance
  • Configurable via torch_compile parameter in yaml configs (true/false/"default"/"reduce-overhead"/"max-autotune")
  • Separate compilation modes for actor and central value networks
  • See torch.compile documentation for detailed configuration and mode selection guidance
• Fixed critical bugs in asymmetric actor-critic (central_value) training:
  • Fixed incorrect device reference in update_lr() method
  • Fixed infinite loop when iterating over dataset
  • Added proper __iter__ method to PPODataset class
• Fixed variance calculation in RunningMeanStd to use population variance
• Fixed get_mean_std_with_masks function.
• Fixed missing central value optimizer state in checkpoint save/load
• Added myosuite support.
• Added auxilary loss support.
• Update for tacsl release: CNN tower processing, critic weights loading and freezing.
• Fixed SAC input normalization.
• Fixed SAC agent summary writer to use configured directory instead of hardcoded 'runs/'
• Fixed default player config num_games value.
• Fixed applying minibatch size per env.
• Added concat_output support for RNN.
• SAC improvements:
  • Fixed missing gamma_tensor initialization bug
  • Removed hardcoded torch.compile decorators (now respects YAML config)
  • Optimized tensor operations and removed unnecessary clones
• Environment wrapper fixes:
  • Fixed tuple/list observation handling for compatibility with various gym environments
  • Added proper numpy to torch tensor conversion in cast_obs
  • Fixed missing gym import in envpool wrapper
• Ray integration improvements:
  • Moved Ray import to lazy loading (only when RayVecEnv is used)
  • Added configurable Ray initialization with ray_config parameter
  • Added proper cleanup with close() method for Ray actors
  • Default 1GB object store memory allocation

1.6.1

• Fixed Central Value RNN bug which occurs if you train ma multi agent environment.
• Added Deepmind Control PPO benchmark.
• Added a few more experimental ways to train value prediction (OneHot, TwoHot encoding and crossentropy loss instead of L2).
• New methods didn't. It is impossible to turn it on from the yaml files. Once we find an env which trains better it will be added to the config.
• Added shaped reward graph to the tensorboard.
• Fixed bug with SAC not saving weights with save_frequency.
• Added multi-node training support for GPU-accelerated training environments like Isaac Gym. No changes in training scripts are required. Thanks to @ankurhanda and @ArthurAllshire for assistance in implementation.
• Added evaluation feature for inferencing during training. Checkpoints from training process can be automatically picked up and updated in the inferencing process when enabled.Enhanced
• Added get/set API for runtime update of rl training parameters. Thanks to @ArthurAllshire for the initial version of fast PBT code.
• Fixed SAC not loading weights properly.
• Removed Ray dependency for use cases it's not required.
• Added warning for using deprecated 'seq_len' instead of 'seq_length' in configs with RNN networks.

1.6.0

• Added ONNX export colab example for discrete and continious action spaces. For continuous case LSTM policy example is provided as well.
• Improved RNNs training in continuous space, added option zero_rnn_on_done.
• Added NVIDIA CuLE support: https://github.com/NVlabs/cule
• Added player config everride. Vecenv is used for inference.
• Fixed multi-gpu training with central value.
• Fixed max_frames termination condition, and it's interaction with the linear learning rate: https://github.com/Denys88/rl_games/issues/212
• Fixed "deterministic" misspelling issue.
• Fixed Mujoco and Brax SAC configs.
• Fixed multiagent envs statistics reporting. Fixed Starcraft2 SMAC environments.

1.5.2

• Added observation normalization to the SAC.
• Returned back adaptive KL legacy mode.

1.5.1

• Fixed build package issue.

1.5.0

• Added wandb support.
• Added poetry support.
• Fixed various bugs.
• Fixed cnn input was not divided by 255 in case of the dictionary obs.
• Added more envpool mujoco and atari training examples. Some of the results: 15 min Mujoco humanoid training, 2 min atari pong.
• Added Brax and Mujoco colab training examples.
• Added 'seed' command line parameter. Will override seed in config in case it's > 0.
• Deprecated horovod in favor of torch.distributed (#171).

1.4.0

• Added discord channel https://discord.gg/hnYRq7DsQh :)
• Added envpool support with a few atari examples. Works 3-4x time faster than ray.
• Added mujoco results. Much better than openai spinning up ppo results.
• Added tcnn(https://github.com/NVlabs/tiny-cuda-nn) support. Reduces 5-10% of training time in the IsaacGym envs.
• Various fixes and improvements.

1.3.2

• Added 'sigma' command line parameter. Will override sigma for continuous space in case if fixed_sigma is True.

1.3.1

• Fixed SAC not working

1.3.0

• Simplified rnn implementation. Works a little bit slower but much more stable.
• Now central value can be non-rnn if policy is rnn.
• Removed load_checkpoint from the yaml file. now --checkpoint works for both train and play.

1.2.0

• Added Swish (SILU) and GELU activations, it can improve Isaac Gym results for some of the envs.
• Removed tensorflow and made initial cleanup of the old/unused code.
• Simplified runner.
• Now networks are created in the algos with load_network method.

1.1.4

• Fixed crash in a play (test) mode in player, when simulation and rl_devices are not the same.
• Fixed variuos multi gpu errors.

1.1.3

• Fixed crash when running single Isaac Gym environment in a play (test) mode.
• Added config parameter clip_actions for switching off internal action clipping and rescaling

1.1.0

• Added to pypi: pip install rl-games
• Added reporting env (sim) step fps, without policy inference. Improved naming.
• Renames in yaml config for better readability: steps_num to horizon_length amd lr_threshold to kl_threshold

Custom stop callback

In addition to the built-in score_to_win, max_epochs, and max_frames stop conditions, training can be terminated by a user-defined callback. The callback receives the algorithm instance and returns True to stop. It is checked once per epoch on rank 0 and broadcast to other ranks under multi-GPU. Works with PPO (continuous and discrete) and SAC.

The signature:

def my_stop(algo) -> bool:
    # algo exposes: epoch_num, frame, last_mean_rewards, mean_rewards,
    # game_rewards, game_lengths, config, ...
    return algo.last_mean_rewards > 18.0 and algo.epoch_num >= 100

There are two ways to wire it in.

A. Manual algo instantiation

Build the agent yourself, set the callback, then call train():

import yaml
from rl_games.torch_runner import Runner

with open('rl_games/configs/atari/ppo_pong.yaml') as f:
    cfg = yaml.safe_load(f)

runner = Runner()
runner.load(cfg)
runner.load_config(runner.default_config)

agent = runner.algo_factory.create(
    runner.algo_name, base_name='run', params=runner.params
)
agent.stop_fn = lambda algo: algo.last_mean_rewards > 18.0 and algo.epoch_num >= 100
agent.train()

B. Through Runner.run

Pass the callable in the runtime args dict (programmatic), or reference it from YAML by import path (string).

Programmatic:

def my_stop(algo):
    return algo.last_mean_rewards > 18.0 and algo.epoch_num >= 100

runner = Runner()
runner.load(cfg)
runner.run({'train': True, 'stop_fn': my_stop})

YAML — set config.stop_fn to either pkg.mod:function or pkg.mod.function:

params:
  config:
    stop_fn: my_project.stops:reward_plateau

The string is resolved by importlib at training start. args['stop_fn'] (programmatic) takes precedence over config['stop_fn'] (YAML) if both are set.

stop_fn must be a callable taking the algo and returning bool; an ValueError is raised at startup otherwise.

Troubleshouting

• Some of the supported envs are not installed with setup.py, you need to manually install them
• Starting from rl-games 1.1.0 old yaml configs won't be compatible with the new version:
  • steps_num should be changed to horizon_length amd lr_threshold to kl_threshold

Known issues

• Running a single environment with Isaac Gym can cause crash, if it happens switch to at least 2 environments simulated in parallel