Paper: Benchmarking LLMs' Swarm intelligence Code: https://github.com/x66ccff/swarmbench
Large Language Models (LLMs) show potential for complex reasoning, yet their capacity for emergent coordination in Multi-Agent Systems (MAS) when operating under strict swarm-like constraints—limited local perception and communication—remains largely unexplored. Existing benchmarks often do not fully capture the unique challenges of decentralized coordination when agents operate with incomplete spatio-temporal information. To bridge this gap, we introduce SwarmBench, a novel benchmark designed to systematically evaluate the swarm intelligence capabilities of LLMs acting as decentralized agents. SwarmBench features five foundational MAS coordination tasks (Pursuit, Synchronization, Foraging, Flocking, Transport) within a configurable 2D grid environment, forcing agents to rely solely on local sensory input ($k\times k$ view) and local communication. We propose metrics for coordination effectiveness and analyze emergent group dynamics. Zero-shot evaluations of leading LLMs (e.g., deepseek-v3, o4-mini) reveal significant task-dependent performance variations. While some rudimentary coordination is observed, our results indicate that current LLMs significantly struggle with robust long-range planning and adaptive strategy formation under the uncertainty inherent in these decentralized scenarios. Assessing LLMs under such swarm-like constraints is crucial for understanding their utility in future decentralized intelligent systems. We release SwarmBench as an open, extensible toolkit—built on a customizable physical system—providing environments, prompts, evaluation scripts, and comprehensive datasets. This aims to foster reproducible research into LLM-based MAS coordination and the theoretical underpinnings of emergent collective behavior under severe informational decentralization.
Figure 1: Natural Swarm Intelligence Inspiration and SwarmBench Tasks.
Figure 2: Conceptual Architecture of SwarmBench.

Figure 3: Overview of LLM Performance on SwarmBench Tasks
k × k view) and local communication.This dataset contains comprehensive experimental datasets generated and used for benchmarking LLMs' swarm intelligence across various tasks and models.
Ensure you have Conda installed. Then, create the environment using the provided file:
conda env create -f environment.yaml
conda activate swarmbench
Execute the eval.py script to run the benchmark evaluations:
from swarmbench import SwarmFramework
if __name__ == '__main__':
name = 1
for task in ('Transport', 'Pursuit'): ##### {'Transport', 'Pursuit', 'Synchronization', 'Foraging', 'Flocking'}
for model in ('gpt-4o-mini', 'llama-3.1-70b'): ##### Models
for seed in (27, 42):
SwarmFramework.submit(
f'exp_{name}',
SwarmFramework.model_config(model, 'YOUR_API_KEY', 'YOUR_API_BASE'), ##### API
task,
log_dir='YOUR_LOG_DIR', ##### Logging
num_agents=10,
max_round=100,
width=10,
height=10,
seed=seed,
view_size=5
)
name += 1
SwarmFramework.run_all(max_parallel=4)
This will run the LLM agents through the defined tasks and log the results.
python load_dataset.py
@article{ruan2025benchmarkingllmsswarmintelligence,
title={Benchmarking LLMs' Swarm intelligence},
author={Kai Ruan and Mowen Huang and Ji-Rong Wen and Hao Sun},
year={2025},
eprint={2505.04364},
archivePrefix={arXiv},
primaryClass={cs.MA},
url={https://arxiv.org/abs/2505.04364},
}
Paper: Benchmarking LLMs' Swarm intelligence Code: https://github.com/x66ccff/swarmbench
Large Language Models (LLMs) show potential for complex reasoning, yet their capacity for emergent coordination in Multi-Agent Systems (MAS) when operating under strict swarm-like constraints—limited local perception and communication—remains largely unexplored. Existing benchmarks often do not fully capture the unique challenges of decentralized coordination when agents operate with incomplete spatio-temporal information. To bridge this gap, we introduce SwarmBench, a novel benchmark designed to systematically evaluate the swarm intelligence capabilities of LLMs acting as decentralized agents. SwarmBench features five foundational MAS coordination tasks (Pursuit, Synchronization, Foraging, Flocking, Transport) within a configurable 2D grid environment, forcing agents to rely solely on local sensory input ($k\times k$ view) and local communication. We propose metrics for coordination effectiveness and analyze emergent group dynamics. Zero-shot evaluations of leading LLMs (e.g., deepseek-v3, o4-mini) reveal significant task-dependent performance variations. While some rudimentary coordination is observed, our results indicate that current LLMs significantly struggle with robust long-range planning and adaptive strategy formation under the uncertainty inherent in these decentralized scenarios. Assessing LLMs under such swarm-like constraints is crucial for understanding their utility in future decentralized intelligent systems. We release SwarmBench as an open, extensible toolkit—built on a customizable physical system—providing environments, prompts, evaluation scripts, and comprehensive datasets. This aims to foster reproducible research into LLM-based MAS coordination and the theoretical underpinnings of emergent collective behavior under severe informational decentralization.
Figure 1: Natural Swarm Intelligence Inspiration and SwarmBench Tasks.
Figure 2: Conceptual Architecture of SwarmBench.

Figure 3: Overview of LLM Performance on SwarmBench Tasks
k × k view) and local communication.This dataset contains comprehensive experimental datasets generated and used for benchmarking LLMs' swarm intelligence across various tasks and models.
Ensure you have Conda installed. Then, create the environment using the provided file:
conda env create -f environment.yaml
conda activate swarmbench
Execute the eval.py script to run the benchmark evaluations:
from swarmbench import SwarmFramework
if __name__ == '__main__':
name = 1
for task in ('Transport', 'Pursuit'): ##### {'Transport', 'Pursuit', 'Synchronization', 'Foraging', 'Flocking'}
for model in ('gpt-4o-mini', 'llama-3.1-70b'): ##### Models
for seed in (27, 42):
SwarmFramework.submit(
f'exp_{name}',
SwarmFramework.model_config(model, 'YOUR_API_KEY', 'YOUR_API_BASE'), ##### API
task,
log_dir='YOUR_LOG_DIR', ##### Logging
num_agents=10,
max_round=100,
width=10,
height=10,
seed=seed,
view_size=5
)
name += 1
SwarmFramework.run_all(max_parallel=4)
This will run the LLM agents through the defined tasks and log the results.
python load_dataset.py
@article{ruan2025benchmarkingllmsswarmintelligence,
title={Benchmarking LLMs' Swarm intelligence},
author={Kai Ruan and Mowen Huang and Ji-Rong Wen and Hao Sun},
year={2025},
eprint={2505.04364},
archivePrefix={arXiv},
primaryClass={cs.MA},
url={https://arxiv.org/abs/2505.04364},
}