AmineAndam04/cleanmarl

Single file implementations of Deep Multi-agent Reinforcement Learning

Python

117 commits

updated Sep 1, 2026

cleanmarl-docs.readthedocs.io

See the code

README

CleanMARL

CleanMARL provides single-file, clean, and educational implementations of Deep Multi-Agent Reinforcement Learning (MARL) algorithms in PyTorch, following the same philosophy of CleanRL.

Main Features

Implementations of key MARL algorithms: VDN, QMIX, COMA, MADDPG, FACMAC, IPPO, and MAPPO.
A documentation for algorithms, code and training details.
We support continuous and discrete actions.
We support parallel environments and recurrent policies.
Tensorboard and Weights & Biases logging.

We provide more details in our documentation.

Check the old_jax branch for JAX implementations (non-jax envs only).

Quick Start

Prerequisites:

Python >=3.9

Installation:

git clone https://github.com/AmineAndam04/cleanmarl.git
cd cleanmarl
pip install .

To run experiment you can run for example:

python  cleanmarl/vdn.py --env_type="pz" --env_name="simple_spread_v3" --env_family="mpe" --use_wnb --wnb_project="cleanmarl-test" --wnb_entity="cleanmarl-test" --total_timesteps=1000000

python  cleanmarl/mappo.py --env_type="smaclite" --env_name="3m"

Supported Algorithms

Algorithm	Variants Implemented
Value Decomposition Networks (VDN)	`vdn.py` `vdn_lstm.py` `vdn_multienvs.py`
QMIX	`qmix.py` `qmix_lstm.py` `qmix_multienvs.py`
Counterfactual Multi-Agent (COMA)	`coma.py` `coma_lstm.py` `coma_multienvs.py` `coma_lstm_multienvs.py`
Multi-Agent Deep Deterministic Policy Gradient (MADDPG)	`maddpg.py` `maddpg_multienvs.py` `maddpg_lstm.py` `maddpg_continuous`
Factored Multi-Agent Centralized Policy Gradients (FACMAC)	`facmac.py` `facmac_multienvs.py` `facmac_continuous`
Independent Proximal Policy Optimization (IPPO)	`ippo.py` `ippo_lstm.py` `ippo_multienvs.py` `ippo_lstm_multienvs.py` `ippo_continuous`
Multi-Agent Proximal Policy Optimization (MAPPO)	`mappo.py` `mappo_lstm.py` `mappo_multienvs.py` `mappo_lstm_multienvs.py` `mappo_continuous`

Supported environments

We use marlbench to interact with MARL environments. marlbench is a tool that provides (1) a common API for MARL envs, (2) vectorized envs, and (2) common wrappers (normalization, clipping ...)

Install using uv pip install marlbench

Git repo: marlbench

Environment	Action space	Installation
Level-Based Foraging	Discrete	`pip install lbforaging`
Multi-Robot Warehouse	Discrete	`pip install rware`
SMAClite	Discrete	Install it from its GitHub repository
PettingZoo	Discrete or continuous	`pip install pettingzoo` and install the extra dependencies for the family you use
MaMuJoCo	Continuous	`pip install gymnasium-robotics`
MAgent2	Discrete	`pip install magent2`
SMAC	Discrete	Follow the instructions in the SMAC repository
SMACv2	Discrete	Follow the instructions in the SMACv2 repository

multiagent-reinforcement-learning

reinforcement-learning

Similar repos

Replicable-MARL/MARLlib

One repository is all that is necessary for Multi-agent Reinforcement Learning (MARL)

1,350

instadeepai/Mava

🦁 A research-friendly codebase for fast experimentation of multi-agent reinforcement learning in…

937

vwxyzjn/cleanrl

High-quality single file implementation of Deep Reinforcement Learning algorithms with…

10,492

TimeBreaker/MARL-papers-with-code

Multi-Agent Reinforcement Learning (MARL) papers with code

433

HorizonRobotics/alf

Agent Learning Framework https://alf.readthedocs.io

367

hijkzzz/pymarl2

Fine-tuned MARL algorithms on SMAC (100% win rates on most scenarios)

710

TimeBreaker/MARL-resources-collection

A Collection of Multi-Agent Reinforcement Learning (MARL) Resources

267

TimeBreaker/Multi-Agent-Reinforcement-Learning-papers

Multi-Agent Reinforcement Learning (MARL) papers

317