An Extended Benchmarking of Multi-Agent Reinforcement Learning Algorithms in Complex Fully Cooperative Tasks
Python
60
263 commits
updated Jul 23, 2026
Image was generated by ImageGen at 12th Feb 2025
PyMARLzoo+ is based on EPyMARL, and includes
Algorithms (9) maintained from EPyMARL:
If you find PyMARLzoo+ useful or use PyMARLzoo+ in your research, please cite it in your publications.
@inproceedings{10.5555/3709347.3743796,
author = {Papadopoulos, George and Kontogiannis, Andreas and Papadopoulou, Foteini and Poulianou, Chaido and Koumentis, Ioannis and Vouros, George},
title = {An Extended Benchmarking of Multi-Agent Reinforcement Learning Algorithms in Complex Fully Cooperative Tasks},
year = {2025},
isbn = {9798400714269},
publisher = {International Foundation for Autonomous Agents and Multiagent Systems},
address = {Richland, SC},
abstract = {Multi-Agent Reinforcement Learning (MARL) has recently emerged as a significant area of research. However, MARL evaluation often lacks systematic diversity, hindering a comprehensive understanding of algorithms' capabilities. In particular, cooperative MARL algorithms are predominantly evaluated on benchmarks such as SMAC and GRF, which primarily feature team game scenarios without assessing adequately various aspects of agents' capabilities required in fully cooperative real-world tasks such as multi-robot cooperation and warehouse, resource management, search and rescue, and human-AI cooperation. Moreover, MARL algorithms are mainly evaluated on low dimensional state spaces, and thus their performance on high-dimensional (e.g., image) observations is not well-studied. To fill this gap, this paper highlights the crucial need for expanding systematic evaluation across a wider array of existing benchmarks. To this end, we conduct extensive evaluation and comparisons of well-known MARL algorithms on complex fully cooperative benchmarks, including tasks with images as agents' observations. Interestingly, our analysis shows that many algorithms, hailed as state-of-the-art on SMAC and GRF, may underperform standard MARL baselines on fully cooperative benchmarks. Finally, towards more systematic and better evaluation of cooperative MARL algorithms, we have open-sourced PyMARLzoo+, an extension of the widely used (E)PyMARL libraries, which addresses an open challenge from[46], facilitating seamless integration and support with all benchmarks of PettingZoo, as well as Overcooked, PressurePlate, Capture Target and Box Pushing.},
booktitle = {Proceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems},
pages = {1613–1622},
numpages = {10},
keywords = {benchmarking, fully cooperative multi-agent reinforcement learning, image-based observations, open-source framework},
location = {Detroit, MI, USA},
series = {AAMAS '25}
}
Before installing PyMARLzoo+, we recommend creating a conda environment:
conda create -y -n pymarlzooplus python=3.8.18
conda activate pymarlzooplus
To install and use PyMARLzoo+ as a package, run the following commands:
pip install pymarlzooplus
To build it from source, run the following commands:
git clone https://github.com/AILabDsUnipi/pymarlzooplus.git
cd pymarlzooplus
pip install -e .
Note that before running an atari environment (from PettingZoo) for the first time, you have to run:
AutoROM -y
The docker image is available at Docker Hub.
Pull the image from Docker Hub with the following command:
docker pull ailabdsunipi/pymarlzooplus:latest
You can run the image with the following command:
docker run -it --rm -v /path/to/your/project:/home ailabdsunipi/pymarlzooplus:latest bash
You can also use the provided Docker files in docker/ to build and run a Docker container with the pymarlzooplus installed as a headless machine.
To build the image, run the container and get into the container, run the following commands after you clone the repository:
docker compose -f docker/docker-compose.yml up --build
docker exec -it $(docker ps | awk 'NR > 1 {print $1}') /bin/bash
In the following instructions, <algo> (or "algo") can be replaced with one of the following algoriths:
You can specify the algorithm arguments:
"algo_args".
"algo" and "scenario"): from pymarlzooplus import pymarlzooplus
params_dict = {
"config": "algo",
"algo_args": {},
"env-config": "gymma",
"env_args": {
"key": "scenario",
"time_limit": 50,
"seed": 2024
}
}
pymarlzooplus(params_dict)
Find all the LBF arguments in the configuration file pymarlzooplus/config/envs/gymma.yaml.
<algo> and <scenario>):
python pymarlzooplus/main.py --config=<algo> --env-config=gymma with env_args.time_limit=25 env_args.key=<scenario>
Available scenarios (replace "scenario" or <scenario>) we run experiments (our experiments were conducted with v2):
All the above scenarios are compatible both with our training framework and the environment API.
As a package (replace "algo" and "scenario"):
from pymarlzooplus import pymarlzooplus
params_dict = {
"config": "algo",
"algo_args": {},
"env-config": "gymma",
"env_args": {
"key": "scenario",
"time_limit": 500,
"seed": 2024
}
}
pymarlzooplus(params_dict)
Find all the RWARE arguments in the configuration file pymarlzooplus/config/envs/gymma.yaml.
From source (replace <algo> and <scenario>):
python pymarlzooplus/main.py --config=<algo> --env-config=gymma with env_args.time_limit=500 env_args.key=<scenario>
Before running an experiment, edit the configuration file in pymarlzooplus/config/envs/gymma.yaml.
Available scenarios we run experiments (our experiments were conducted with v1):
All the above scenarios are compatible both with our training framework and the environment API.
As a package (replace "algo" and "scenario"):
from pymarlzooplus import pymarlzooplus
params_dict = {
"config": "algo",
"algo_args": {},
"env-config": "gymma",
"env_args": {
"key": "scenario",
"time_limit": 25,
"seed": 2024
}
}
pymarlzooplus(params_dict)
Find all the MPE arguments in the configuration file pymarlzooplus/config/envs/gymma.yaml.
From source (replace <algo> and <scenario>):
python pymarlzooplus/main.py --config=<algo> --env-config=gymma with env_args.time_limit=25 env_args.key=<scenario>
Before running an experiment, edit the configuration file in pymarlzooplus/config/envs/gymma.yaml.
Available scenarios we run experiments:
More available scenarios:
All the above scenarios are compatible both with our training framework and the environment API.
As a package (replace "algo", "task", and 0000):
from pymarlzooplus import pymarlzooplus
params_dict = {
"config": "algo",
"algo_args": {},
"env-config": "pettingzoo",
"env_args": {
"key": "task",
"time_limit": 0000,
"render_mode": "rgb_array", # Options: "human", "rgb_array
"image_encoder": "ResNet18", # Options: "ResNet18", "SlimSAM", "CLIP"
"image_encoder_use_cuda": True, # Whether to load image-encoder in GPU or not.
"image_encoder_batch_size": 10, # How many images to encode in a single pass.
"partial_observation": False, # Only for "Emtombed: Cooperative" and "Space Invaders"
"trainable_cnn": False, # Specifies whether to return image-observation or the encoded vector-observation
"seed": 2024
}
}
pymarlzooplus(params_dict)
Find all the PettingZoo arguments in the configuration file pymarlzooplus/config/envs/pettingzoo.yaml. Also, there you can find useful comments for each task's arguments.
From source (replace <algo>, <time-limit>, and <task>):
python pymarlzooplus/main.py --config=<algo> --env-config=pettingzoo with env_args.time_limit=<time-limit> env_args.key=<task>
Before running an experiment, edit the configuration file in pymarlzooplus/config/envs/pettingzoo.yaml.
Available tasks we run experiments:
The above tasks are compatible both with our training framework and the environment API. Below, we list more tasks which are compatible only with the environment API:
As a package (replace "algo", "scenario", and "rewardType"):
from pymarlzooplus import pymarlzooplus
params_dict = {
"config": "algo",
"algo_args": {},
"env-config": "overcooked",
"env_args": {
"time_limit": 500,
"key": "scenario",
"reward_type": "rewardType",
}
}
pymarlzooplus(params_dict)
Find all the Overcooked arguments in the configuration file pymarlzooplus/config/envs/overcooked.yaml.
From source (replace <algo>, <scenario>, and <reward_type>):
python pymarlzooplus/main.py --config=<algo> --env-config=overcooked with env_args.time_limit=500 env_args.key=<scenario> env_args.reward_type=<reward_type>
Before running an experiment, edit the configuration file in pymarlzooplus/config/envs/overcooked.yaml.
Available scenarios we run experiments:
More available scenarios:
Reward types available:
All the above scenarios are compatible both with our training framework and the environment API.
As a package (replace "algo" and "scenario"):
from pymarlzooplus import pymarlzooplus
params_dict = {
"config": "algo",
"algo_args": {},
"env-config": "pressureplate",
"env_args": {
"key": "scenario",
"time_limit": 500,
"seed": 2024
}
}
pymarlzooplus(params_dict)
Find all the Pressure Plate arguments in the configuration file pymarlzooplus/config/envs/pressureplate.yaml.
From source (replace <algo> and <scenario>):
python pymarlzooplus/main.py --config=<algo> --env-config=pressureplate with env_args.key=<scenario> env_args.time_limit=500
Before running an experiment, edit the configuration file in pymarlzooplus/config/envs/pressureplate.yaml.
Available scenarios we run experiments:
More available scenarios:
All the above scenarios are compatible both with our training framework and the environment API.
As a package (replace "algo" and "scenario"):
from pymarlzooplus import pymarlzooplus
params_dict = {
"config": "algo",
"algo_args": {},
"env-config": "capturetarget",
"env_args": {
"time_limit": 60,
"key" : "scenario",
}
}
pymarlzooplus(params_dict)
Find all the Capture Target arguments in the configuration file pymarlzooplus/config/envs/capturetarget.yaml.
From source (replace <algo> and <scenario>):
python pymarlzooplus/main.py --config=<algo> --env-config=capturetarget with env_args.key=<scenario> env_args.time_limit=60
Before running an experiment, edit the configuration file in pymarlzooplus/config/envs/capturetarget.yaml. Also, there you can find useful comments for the rest of arguments.
Available scenario:
The above scenario is compatible both with our training framework and the environment API.
As a package (replace "algo" and "scenario"):
from pymarlzooplus import pymarlzooplus
params_dict = {
"config": "algo",
"algo_args": {},
"env-config": "boxpushing",
"env_args": {
"key": "scenario",
"time_limit": 60,
"seed": 2024
}
}
pymarlzooplus(params_dict)
Find all the Box Pushing arguments in the configuration file pymarlzooplus/config/envs/boxpushing.yaml.
From source (replace <algo> and <scenario>):
python pymarlzooplus/main.py --config=<algo> --env-config=boxpushing with env_args.key=<scenario> env_args.time_limit=60
Before running an experiment, edit the configuration file in pymarlzooplus/config/envs/boxpushing.yaml.
Available scenario:
The above scenario is compatible both with our training framework and the environment API.
Except from using our training framework, you can use PyMARLzoo+ in your custom training pipeline as follows:
# Import package
from pymarlzooplus.envs import REGISTRY as env_REGISTRY
# Example of arguments for PettingZoo
args = {
"env": "pettingzoo",
"env_args": {
"key": "pistonball_v6",
"time_limit": 900,
"render_mode": "rgb_array",
"image_encoder": "ResNet18",
"image_encoder_use_cuda": True,
"image_encoder_batch_size": 10,
"centralized_image_encoding": False,
"partial_observation": False,
"trainable_cnn": False,
"kwargs": "('n_pistons',10),",
"seed": 2024
}
}
# Initialize environment
env = env_REGISTRY[args["env"]](**args["env_args"])
n_agns = env.get_n_agents()
n_acts = env.get_total_actions()
# Reset the environment
obs, state = env.reset()
done = False
# Run an episode
while not done:
# Render the environment (optional)
env.render()
# Insert the policy's actions here
actions = env.sample_actions()
# Apply an environment step
reward, done, extra_info = env.step(actions)
obs = env.get_obs()
state = env.get_state()
info = env.get_info()
# Terminate the environment
env.close()
The code above can be used as is (changing the arguments accordingly) with the fully cooperative tasks (see section Run training to find out which tasks are fully cooperative).
n_agns (int)
Number of agents in the environment.
n_acts (int)
Number of available actions per agent (all agents share the same action space).
reward (float)
Sum of all agents’ rewards for the current step.
done (bool)
False if any agent is still active; True only when all agents are done.
extra_info (dict)
Typically is an empty dictionary.
info (dict)
Contains a single key, "TimeLimit.truncated", which is False if any agent’s time limit has not been exceeded.
obs (tuple of numpy arrays)
One array per agent.
(cnn_features_dim,) (default 128).(3, h, w).state (numpy array)
The global state:
(cnn_features_dim * n_agns,).(n_agents, 3, h, w).In the file pymarlzooplus/scripts/api_script.py, you can find a complete example for all the supported environments.
For a description of each environment's arguments, please see the corresponding config file in pymarlzooplus/config/envs.
We include a script named search.py which reads a search configuration file (e.g. the included search.config.example.yaml) and runs a hyperparameter search in one or more tasks. The script can be run using:
python pymarlzooplus/search.py run --config=pymarlzooplus/config/search.config.example.yaml --seeds 5 locally
In a cluster environment where one run should go to a single process, it can also be called in a batch script like:
python pymarlzooplus/search.py run --config=pymarlzooplus/config/search.config.example.yaml --seeds 5 single 1
where 1 is an index to the particular hyperparameter configuration and can take values from 1 to the number of different combinations.
To visualize experiment results, use the utils/plot_utils.py . Plots are defined by YAML configs in pymarlzooplus/config/plots/.
Run plots from the pymarlzooplus root directory.
Command:
python utils/plot_utils.py <mode> --config <path_to_config.yaml>
Modes:
single: Plot a single experiment.multiple: Aggregates multiple seeds/algorithms for one environment.average: Averages results across multiple environments (using .pkl files from multiple mode).Configuration
config/plots/.
If --config is omitted, a default is used based on the mode:
single -> config/plots/single.yamlmultiple -> config/plots/multiple.yamlaverage -> config/plots/average.yaml~ for your home directory in paths (e.g., ~/results/sacred/...).Run plots from the pymarlzooplus root directory.
python utils/plot_utils.py single --config config/plots/single.yaml
python utils/plot_utils.py multiple --config config/plots/multiple.yaml
python utils/plot_utils.py average --config config/plots/average.yaml
You can save the learnt models to disk by setting save_model = True, which is set to False by default. The frequency of saving models can be adjusted using save_model_interval configuration. Models will be saved in the result directory, under the folder called models. The directory corresponding each run will contain models saved throughout the experiment, each within a folder corresponding to the number of timesteps passed since starting the learning process.
Learnt models can be loaded using the checkpoint_path parameter, after which the learning will proceed from the corresponding timestep.
PyMARLzoo+ supports integration with Weights & Biases (W&B). We thank Aditya Parameshwaran for this contribution. To enable logging to W&B:
requirements.txt):conda run -n pymarlzooplus pip install -U wandb
or check if wandb is already installed:
conda run -n pymarlzooplus pip show wandb
conda run -n pymarlzooplus wandb login
Alternatively, you can export your API key:
export WANDB_API_KEY=<your_api_key>
pymarlzooplus/config/default.yaml) by setting enable: True inside the wandb section:wandb:
enable: True # set to True to activate W&B logging
project: "your-project-name" # W&B project name (e.g. "pymarlzooplus")
entity: "" # W&B entity/user (optional, leave empty for default)
run_name: null # if null, an automatic name is built from the env, algorithm and seed
notes: "" # optional notes/description for the run
tags: [] # optional list of tags
save_dir: "results/wandb" # directory where W&B stores run files
sync_tensorboard: False
resume: False # whether to attempt to resume a previous run
id: null # if provided and resume is True, use this run id
WANDB_MODE=offline python pymarlzooplus/main.py --config=<algo> --env-config=<env> with env_args.key=<scenario>
or set it in the config:
wandb:
enable: False
# ...
Notes:
wandb isn't installed, the code will continue without failing.wandb.init) lives in Logger.setup_wandb() (pymarlzooplus/utils/logging_setup.py).Logger are automatically forwarded to W&B when enabled.Installation error for box2d-py, run:
sudo apt-get install swig
In case you get ImportError: Library "GLU" not found., run:
sudo apt-get install python3-opengl
In case you get IndexError: list index out of range for pyglet in a non-headless machine using render=True,
set __GLX_VENDOR_LIBRARY_NAME=nvidia and __NV_PRIME_RENDER_OFFLOAD=1 as environment variables.
For example, if you want to run your own file example.py , run:
__GLX_VENDOR_LIBRARY_NAME=nvidia __NV_PRIME_RENDER_OFFLOAD=1 python example.py
In case you get ERROR: Failed building wheel for multi-agent-ale-py, probably the problem is with the compiler (if cmake is already installed), so run:
sudo apt-get install g++-9 gcc-9
export CC=gcc-9 CXX=g++-9
In case you get SystemError: ffi_prep_closure(): bad user_data (it seems that the version of the libffi library seen at runtime is different from the 'ffi.h' file seen at compile-time), probably for "waterworld_v4" of PettingZoo, run:
pip install --upgrade --force-reinstall pymunk cffi
In case you get ImportError: ../bin/../lib/libstdc++.so.6: 'version GLIBCXX_3.4.32' not found, run:
conda install -c conda-forge libstdcxx-ng
If you encounter a bug, have a feature request, or any other questions about the project, we encourage you to open an issue. Please follow these steps:
Thank you for contributing to the project and helping us improve!
If you use PyMARLzoo+ in your research, please cite our paper using the following bibtex:
@inproceedings{papadopoulos2025extended,
title={An Extended Benchmarking of Multi-Agent Reinforcement Learning Algorithms in Complex Fully Cooperative Tasks},
author={Papadopoulos, George and Kontogiannis, Andreas and Papadopoulou, Foteini and Poulianou, Chaido and Koumentis, Ioannis and Vouros, George},
booktitle={Proceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems},
pages={1613--1622},
year={2025}
}
All the source code that has been taken from the EPyMARL and PyMARL repository was licensed (and remains so) under the Apache License v2.0 (included in LICENSE file).
Any new code is also licensed under the Apache License v2.0
An Extended Benchmarking of Multi-Agent Reinforcement Learning Algorithms in Complex Fully Cooperative Tasks
Python
60
263 commits
updated Jul 23, 2026
Image was generated by ImageGen at 12th Feb 2025
PyMARLzoo+ is based on EPyMARL, and includes
Algorithms (9) maintained from EPyMARL:
If you find PyMARLzoo+ useful or use PyMARLzoo+ in your research, please cite it in your publications.
@inproceedings{10.5555/3709347.3743796,
author = {Papadopoulos, George and Kontogiannis, Andreas and Papadopoulou, Foteini and Poulianou, Chaido and Koumentis, Ioannis and Vouros, George},
title = {An Extended Benchmarking of Multi-Agent Reinforcement Learning Algorithms in Complex Fully Cooperative Tasks},
year = {2025},
isbn = {9798400714269},
publisher = {International Foundation for Autonomous Agents and Multiagent Systems},
address = {Richland, SC},
abstract = {Multi-Agent Reinforcement Learning (MARL) has recently emerged as a significant area of research. However, MARL evaluation often lacks systematic diversity, hindering a comprehensive understanding of algorithms' capabilities. In particular, cooperative MARL algorithms are predominantly evaluated on benchmarks such as SMAC and GRF, which primarily feature team game scenarios without assessing adequately various aspects of agents' capabilities required in fully cooperative real-world tasks such as multi-robot cooperation and warehouse, resource management, search and rescue, and human-AI cooperation. Moreover, MARL algorithms are mainly evaluated on low dimensional state spaces, and thus their performance on high-dimensional (e.g., image) observations is not well-studied. To fill this gap, this paper highlights the crucial need for expanding systematic evaluation across a wider array of existing benchmarks. To this end, we conduct extensive evaluation and comparisons of well-known MARL algorithms on complex fully cooperative benchmarks, including tasks with images as agents' observations. Interestingly, our analysis shows that many algorithms, hailed as state-of-the-art on SMAC and GRF, may underperform standard MARL baselines on fully cooperative benchmarks. Finally, towards more systematic and better evaluation of cooperative MARL algorithms, we have open-sourced PyMARLzoo+, an extension of the widely used (E)PyMARL libraries, which addresses an open challenge from[46], facilitating seamless integration and support with all benchmarks of PettingZoo, as well as Overcooked, PressurePlate, Capture Target and Box Pushing.},
booktitle = {Proceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems},
pages = {1613–1622},
numpages = {10},
keywords = {benchmarking, fully cooperative multi-agent reinforcement learning, image-based observations, open-source framework},
location = {Detroit, MI, USA},
series = {AAMAS '25}
}
Before installing PyMARLzoo+, we recommend creating a conda environment:
conda create -y -n pymarlzooplus python=3.8.18
conda activate pymarlzooplus
To install and use PyMARLzoo+ as a package, run the following commands:
pip install pymarlzooplus
To build it from source, run the following commands:
git clone https://github.com/AILabDsUnipi/pymarlzooplus.git
cd pymarlzooplus
pip install -e .
Note that before running an atari environment (from PettingZoo) for the first time, you have to run:
AutoROM -y
The docker image is available at Docker Hub.
Pull the image from Docker Hub with the following command:
docker pull ailabdsunipi/pymarlzooplus:latest
You can run the image with the following command:
docker run -it --rm -v /path/to/your/project:/home ailabdsunipi/pymarlzooplus:latest bash
You can also use the provided Docker files in docker/ to build and run a Docker container with the pymarlzooplus installed as a headless machine.
To build the image, run the container and get into the container, run the following commands after you clone the repository:
docker compose -f docker/docker-compose.yml up --build
docker exec -it $(docker ps | awk 'NR > 1 {print $1}') /bin/bash
In the following instructions, <algo> (or "algo") can be replaced with one of the following algoriths:
You can specify the algorithm arguments:
"algo_args".
"algo" and "scenario"): from pymarlzooplus import pymarlzooplus
params_dict = {
"config": "algo",
"algo_args": {},
"env-config": "gymma",
"env_args": {
"key": "scenario",
"time_limit": 50,
"seed": 2024
}
}
pymarlzooplus(params_dict)
Find all the LBF arguments in the configuration file pymarlzooplus/config/envs/gymma.yaml.
<algo> and <scenario>):
python pymarlzooplus/main.py --config=<algo> --env-config=gymma with env_args.time_limit=25 env_args.key=<scenario>
Available scenarios (replace "scenario" or <scenario>) we run experiments (our experiments were conducted with v2):
All the above scenarios are compatible both with our training framework and the environment API.
As a package (replace "algo" and "scenario"):
from pymarlzooplus import pymarlzooplus
params_dict = {
"config": "algo",
"algo_args": {},
"env-config": "gymma",
"env_args": {
"key": "scenario",
"time_limit": 500,
"seed": 2024
}
}
pymarlzooplus(params_dict)
Find all the RWARE arguments in the configuration file pymarlzooplus/config/envs/gymma.yaml.
From source (replace <algo> and <scenario>):
python pymarlzooplus/main.py --config=<algo> --env-config=gymma with env_args.time_limit=500 env_args.key=<scenario>
Before running an experiment, edit the configuration file in pymarlzooplus/config/envs/gymma.yaml.
Available scenarios we run experiments (our experiments were conducted with v1):
All the above scenarios are compatible both with our training framework and the environment API.
As a package (replace "algo" and "scenario"):
from pymarlzooplus import pymarlzooplus
params_dict = {
"config": "algo",
"algo_args": {},
"env-config": "gymma",
"env_args": {
"key": "scenario",
"time_limit": 25,
"seed": 2024
}
}
pymarlzooplus(params_dict)
Find all the MPE arguments in the configuration file pymarlzooplus/config/envs/gymma.yaml.
From source (replace <algo> and <scenario>):
python pymarlzooplus/main.py --config=<algo> --env-config=gymma with env_args.time_limit=25 env_args.key=<scenario>
Before running an experiment, edit the configuration file in pymarlzooplus/config/envs/gymma.yaml.
Available scenarios we run experiments:
More available scenarios:
All the above scenarios are compatible both with our training framework and the environment API.
As a package (replace "algo", "task", and 0000):
from pymarlzooplus import pymarlzooplus
params_dict = {
"config": "algo",
"algo_args": {},
"env-config": "pettingzoo",
"env_args": {
"key": "task",
"time_limit": 0000,
"render_mode": "rgb_array", # Options: "human", "rgb_array
"image_encoder": "ResNet18", # Options: "ResNet18", "SlimSAM", "CLIP"
"image_encoder_use_cuda": True, # Whether to load image-encoder in GPU or not.
"image_encoder_batch_size": 10, # How many images to encode in a single pass.
"partial_observation": False, # Only for "Emtombed: Cooperative" and "Space Invaders"
"trainable_cnn": False, # Specifies whether to return image-observation or the encoded vector-observation
"seed": 2024
}
}
pymarlzooplus(params_dict)
Find all the PettingZoo arguments in the configuration file pymarlzooplus/config/envs/pettingzoo.yaml. Also, there you can find useful comments for each task's arguments.
From source (replace <algo>, <time-limit>, and <task>):
python pymarlzooplus/main.py --config=<algo> --env-config=pettingzoo with env_args.time_limit=<time-limit> env_args.key=<task>
Before running an experiment, edit the configuration file in pymarlzooplus/config/envs/pettingzoo.yaml.
Available tasks we run experiments:
The above tasks are compatible both with our training framework and the environment API. Below, we list more tasks which are compatible only with the environment API:
As a package (replace "algo", "scenario", and "rewardType"):
from pymarlzooplus import pymarlzooplus
params_dict = {
"config": "algo",
"algo_args": {},
"env-config": "overcooked",
"env_args": {
"time_limit": 500,
"key": "scenario",
"reward_type": "rewardType",
}
}
pymarlzooplus(params_dict)
Find all the Overcooked arguments in the configuration file pymarlzooplus/config/envs/overcooked.yaml.
From source (replace <algo>, <scenario>, and <reward_type>):
python pymarlzooplus/main.py --config=<algo> --env-config=overcooked with env_args.time_limit=500 env_args.key=<scenario> env_args.reward_type=<reward_type>
Before running an experiment, edit the configuration file in pymarlzooplus/config/envs/overcooked.yaml.
Available scenarios we run experiments:
More available scenarios:
Reward types available:
All the above scenarios are compatible both with our training framework and the environment API.
As a package (replace "algo" and "scenario"):
from pymarlzooplus import pymarlzooplus
params_dict = {
"config": "algo",
"algo_args": {},
"env-config": "pressureplate",
"env_args": {
"key": "scenario",
"time_limit": 500,
"seed": 2024
}
}
pymarlzooplus(params_dict)
Find all the Pressure Plate arguments in the configuration file pymarlzooplus/config/envs/pressureplate.yaml.
From source (replace <algo> and <scenario>):
python pymarlzooplus/main.py --config=<algo> --env-config=pressureplate with env_args.key=<scenario> env_args.time_limit=500
Before running an experiment, edit the configuration file in pymarlzooplus/config/envs/pressureplate.yaml.
Available scenarios we run experiments:
More available scenarios:
All the above scenarios are compatible both with our training framework and the environment API.
As a package (replace "algo" and "scenario"):
from pymarlzooplus import pymarlzooplus
params_dict = {
"config": "algo",
"algo_args": {},
"env-config": "capturetarget",
"env_args": {
"time_limit": 60,
"key" : "scenario",
}
}
pymarlzooplus(params_dict)
Find all the Capture Target arguments in the configuration file pymarlzooplus/config/envs/capturetarget.yaml.
From source (replace <algo> and <scenario>):
python pymarlzooplus/main.py --config=<algo> --env-config=capturetarget with env_args.key=<scenario> env_args.time_limit=60
Before running an experiment, edit the configuration file in pymarlzooplus/config/envs/capturetarget.yaml. Also, there you can find useful comments for the rest of arguments.
Available scenario:
The above scenario is compatible both with our training framework and the environment API.
As a package (replace "algo" and "scenario"):
from pymarlzooplus import pymarlzooplus
params_dict = {
"config": "algo",
"algo_args": {},
"env-config": "boxpushing",
"env_args": {
"key": "scenario",
"time_limit": 60,
"seed": 2024
}
}
pymarlzooplus(params_dict)
Find all the Box Pushing arguments in the configuration file pymarlzooplus/config/envs/boxpushing.yaml.
From source (replace <algo> and <scenario>):
python pymarlzooplus/main.py --config=<algo> --env-config=boxpushing with env_args.key=<scenario> env_args.time_limit=60
Before running an experiment, edit the configuration file in pymarlzooplus/config/envs/boxpushing.yaml.
Available scenario:
The above scenario is compatible both with our training framework and the environment API.
Except from using our training framework, you can use PyMARLzoo+ in your custom training pipeline as follows:
# Import package
from pymarlzooplus.envs import REGISTRY as env_REGISTRY
# Example of arguments for PettingZoo
args = {
"env": "pettingzoo",
"env_args": {
"key": "pistonball_v6",
"time_limit": 900,
"render_mode": "rgb_array",
"image_encoder": "ResNet18",
"image_encoder_use_cuda": True,
"image_encoder_batch_size": 10,
"centralized_image_encoding": False,
"partial_observation": False,
"trainable_cnn": False,
"kwargs": "('n_pistons',10),",
"seed": 2024
}
}
# Initialize environment
env = env_REGISTRY[args["env"]](**args["env_args"])
n_agns = env.get_n_agents()
n_acts = env.get_total_actions()
# Reset the environment
obs, state = env.reset()
done = False
# Run an episode
while not done:
# Render the environment (optional)
env.render()
# Insert the policy's actions here
actions = env.sample_actions()
# Apply an environment step
reward, done, extra_info = env.step(actions)
obs = env.get_obs()
state = env.get_state()
info = env.get_info()
# Terminate the environment
env.close()
The code above can be used as is (changing the arguments accordingly) with the fully cooperative tasks (see section Run training to find out which tasks are fully cooperative).
n_agns (int)
Number of agents in the environment.
n_acts (int)
Number of available actions per agent (all agents share the same action space).
reward (float)
Sum of all agents’ rewards for the current step.
done (bool)
False if any agent is still active; True only when all agents are done.
extra_info (dict)
Typically is an empty dictionary.
info (dict)
Contains a single key, "TimeLimit.truncated", which is False if any agent’s time limit has not been exceeded.
obs (tuple of numpy arrays)
One array per agent.
(cnn_features_dim,) (default 128).(3, h, w).state (numpy array)
The global state:
(cnn_features_dim * n_agns,).(n_agents, 3, h, w).In the file pymarlzooplus/scripts/api_script.py, you can find a complete example for all the supported environments.
For a description of each environment's arguments, please see the corresponding config file in pymarlzooplus/config/envs.
We include a script named search.py which reads a search configuration file (e.g. the included search.config.example.yaml) and runs a hyperparameter search in one or more tasks. The script can be run using:
python pymarlzooplus/search.py run --config=pymarlzooplus/config/search.config.example.yaml --seeds 5 locally
In a cluster environment where one run should go to a single process, it can also be called in a batch script like:
python pymarlzooplus/search.py run --config=pymarlzooplus/config/search.config.example.yaml --seeds 5 single 1
where 1 is an index to the particular hyperparameter configuration and can take values from 1 to the number of different combinations.
To visualize experiment results, use the utils/plot_utils.py . Plots are defined by YAML configs in pymarlzooplus/config/plots/.
Run plots from the pymarlzooplus root directory.
Command:
python utils/plot_utils.py <mode> --config <path_to_config.yaml>
Modes:
single: Plot a single experiment.multiple: Aggregates multiple seeds/algorithms for one environment.average: Averages results across multiple environments (using .pkl files from multiple mode).Configuration
config/plots/.
If --config is omitted, a default is used based on the mode:
single -> config/plots/single.yamlmultiple -> config/plots/multiple.yamlaverage -> config/plots/average.yaml~ for your home directory in paths (e.g., ~/results/sacred/...).Run plots from the pymarlzooplus root directory.
python utils/plot_utils.py single --config config/plots/single.yaml
python utils/plot_utils.py multiple --config config/plots/multiple.yaml
python utils/plot_utils.py average --config config/plots/average.yaml
You can save the learnt models to disk by setting save_model = True, which is set to False by default. The frequency of saving models can be adjusted using save_model_interval configuration. Models will be saved in the result directory, under the folder called models. The directory corresponding each run will contain models saved throughout the experiment, each within a folder corresponding to the number of timesteps passed since starting the learning process.
Learnt models can be loaded using the checkpoint_path parameter, after which the learning will proceed from the corresponding timestep.
PyMARLzoo+ supports integration with Weights & Biases (W&B). We thank Aditya Parameshwaran for this contribution. To enable logging to W&B:
requirements.txt):conda run -n pymarlzooplus pip install -U wandb
or check if wandb is already installed:
conda run -n pymarlzooplus pip show wandb
conda run -n pymarlzooplus wandb login
Alternatively, you can export your API key:
export WANDB_API_KEY=<your_api_key>
pymarlzooplus/config/default.yaml) by setting enable: True inside the wandb section:wandb:
enable: True # set to True to activate W&B logging
project: "your-project-name" # W&B project name (e.g. "pymarlzooplus")
entity: "" # W&B entity/user (optional, leave empty for default)
run_name: null # if null, an automatic name is built from the env, algorithm and seed
notes: "" # optional notes/description for the run
tags: [] # optional list of tags
save_dir: "results/wandb" # directory where W&B stores run files
sync_tensorboard: False
resume: False # whether to attempt to resume a previous run
id: null # if provided and resume is True, use this run id
WANDB_MODE=offline python pymarlzooplus/main.py --config=<algo> --env-config=<env> with env_args.key=<scenario>
or set it in the config:
wandb:
enable: False
# ...
Notes:
wandb isn't installed, the code will continue without failing.wandb.init) lives in Logger.setup_wandb() (pymarlzooplus/utils/logging_setup.py).Logger are automatically forwarded to W&B when enabled.Installation error for box2d-py, run:
sudo apt-get install swig
In case you get ImportError: Library "GLU" not found., run:
sudo apt-get install python3-opengl
In case you get IndexError: list index out of range for pyglet in a non-headless machine using render=True,
set __GLX_VENDOR_LIBRARY_NAME=nvidia and __NV_PRIME_RENDER_OFFLOAD=1 as environment variables.
For example, if you want to run your own file example.py , run:
__GLX_VENDOR_LIBRARY_NAME=nvidia __NV_PRIME_RENDER_OFFLOAD=1 python example.py
In case you get ERROR: Failed building wheel for multi-agent-ale-py, probably the problem is with the compiler (if cmake is already installed), so run:
sudo apt-get install g++-9 gcc-9
export CC=gcc-9 CXX=g++-9
In case you get SystemError: ffi_prep_closure(): bad user_data (it seems that the version of the libffi library seen at runtime is different from the 'ffi.h' file seen at compile-time), probably for "waterworld_v4" of PettingZoo, run:
pip install --upgrade --force-reinstall pymunk cffi
In case you get ImportError: ../bin/../lib/libstdc++.so.6: 'version GLIBCXX_3.4.32' not found, run:
conda install -c conda-forge libstdcxx-ng
If you encounter a bug, have a feature request, or any other questions about the project, we encourage you to open an issue. Please follow these steps:
Thank you for contributing to the project and helping us improve!
If you use PyMARLzoo+ in your research, please cite our paper using the following bibtex:
@inproceedings{papadopoulos2025extended,
title={An Extended Benchmarking of Multi-Agent Reinforcement Learning Algorithms in Complex Fully Cooperative Tasks},
author={Papadopoulos, George and Kontogiannis, Andreas and Papadopoulou, Foteini and Poulianou, Chaido and Koumentis, Ioannis and Vouros, George},
booktitle={Proceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems},
pages={1613--1622},
year={2025}
}
All the source code that has been taken from the EPyMARL and PyMARL repository was licensed (and remains so) under the Apache License v2.0 (included in LICENSE file).
Any new code is also licensed under the Apache License v2.0