MedAgentSim: Self-Evolving Multi-Agent Simulations for Realistic Clinical Interactions

📣 Recent Updates

• [13/05/2025] 🎉 Our paper MedAgentSim: Self-Evolving Multi-Agent Simulations for Realistic Clinical Interactions has been accepted early at MICCAI 2025.
• [31/03/2025] 🔥 We release MedAgentSim: Self-Evolving Multi-Agent Simulations for Realistic Clinical Interactions.

🔍 Overview

MedAgentSim is an open-source simulated hospital environment designed to evaluate and enhance large language model (LLM) performance in dynamic diagnostic settings. Unlike prior approaches, our framework requires doctor agents to actively engage with patients through multi-turn conversations, requesting relevant medical examinations and imaging results to mimic real-world diagnostic processes.

Key features:

• Multi-Agent Architecture: Doctor, patient, and measurement agents interact in a realistic clinical setting
• Self-Improvement Mechanisms: Models iteratively refine their diagnostic strategies through experience
• Experience Replay: Past successful diagnoses inform future cases through knowledge retrieval
• Visual Game Simulation: Built with Phaser for an intuitive, interactive environment
• Multi-Modal Capabilities: Integration with vision language models for medical image interpretation

🚀 Quick Start

Prerequisites

# Clone the repository
git clone https://github.com/MAXNORM8650/MedAgentSim.git
cd MedAgentSim
conda env create -f environment.yml
conda activate mgent
pip install e .
# Install dependencies
pip install -r requirements.txt
# Ensure
pip install --upgrade torch torchao torchvision transformers
pip install --upgrade openai
python -m pip install replicate
python -m pip install  anthropic
python -m pip install groq
python -m pip install accelerate
python -m pip install openai-cost-tracker
python -m pip install django==2.2

Running the Simulation

# Start the server
python -m medsim.server

# In a separate terminal, launch the client
python -u -m medsim.simulate --doctor_llm meta-llama/Llama-3.2-3B-Instruct --patient_llm meta-llama/Llama-3.2-3B-Instruct --measurement_llm meta-llama/Llama-3.2-3B-Instruct --moderator_llm meta-llama/Llama-3.2-3B-Instruct

Visit http://localhost:8000/simulator_home in your browser. Make sure to keep that tab open and active during the simulation.

Host models using vLLM to query

vllm serve unsloth/Llama-3.2-11B-Vision-Instruct-unsloth-bnb-4bit --dtype 'auto'  --quantization "bitsandbytes" --load_format "bitsandbytes" --tensor-parallel-size 4 --max-model-len 8192 --limit-mm-per-prompt image=1

vllm serve meta-llama/Llama-3.2-3B-Instruct --tensor-parallel-size 4
vllm serve unsloth/Llama-3.3-70B-Instruct-bnb-4bit --quantization "bitsandbytes" --load_format "bitsandbytes"

🏥 Simulation Modes

MedAgentSim supports three core interaction modes:

1. Generation Mode: Patient agent autonomously creates cases, generating illnesses, symptoms, and test results
2. Dataset Mode: Patient responses derived from predefined medical datasets
3. Control Mode: Human users can control either the doctor or patient agent for real-time interaction

🧠 Model Support

MedAgentSim is compatible with various LLMs:

• Open-Source Models: LLaMA 3.3, Mistral, Mixtral, Qwen2
• Vision-Language Models: LLaVA 1.5, QwenVL
• Custom Models: Integrate your own models following our documentation

📊 Benchmarks

MedAgentSim has been evaluated on several medical benchmarks:

1. Launch the vLLM server (with appropriate GPUs):

CUDA_VISIBLE_DEVICES=0,1,2,3 vllm serve meta-llama/Llama-3.1-70B-Instruct \
    --tensor-parallel-size 4 \
    --max-model-len 8192

2. Run the simulation:

python medsim/main.py \
    --doctor_llm meta-llama/Llama-3.1-70B-Instruct \
    --patient_llm meta-llama/Llama-3.1-70B-Instruct \
    --measurement_llm meta-llama/Llama-3.1-70B-Instruct \
    --moderator_llm meta-llama/Llama-3.1-70B-Instruct \
    --agent_dataset MedQA

We also support ollama,

python medsim/main.py \
    --doctor_llm llama3.3:70b \
    --patient_llm llama3.3:70b \
    --measurement_llm llama3.3:70b \
    --moderator_llm llama3.3:70b \
    --agent_dataset MedQA

🧩 Project Structure

MedAgentSim/
├── assets/               # Images, CSS, and other static files
├── datasets/             # Sample datasets and medical knowledge base. Put your dataset here
├── docs/                 # Documentation. Coming soon
├── medsim/               # Core simulation code
│   ├── configs/          # configs for for models
│   ├── core/             # Agent implementations
│   ├── server/           # Simulation environment server
│   ├── simulate/         # Multi-agnet running interfaces
│   ├── utils/            # Utility functions
├── Simulacra/            # Backend support
├── MedPromptSimulate/    # Dignosis memory support
├── examples/             # Example scenarios and configurations
├── tests/                # Unit and integration tests. Coming soon
├── requirements.txt      # Python dependencies
├── LICENSE               # License information
└── README.md             # This file

Datasets

from datasets import load_dataset

# Load all files
ds = load_dataset("ItsMaxNorm/MedAgentSim-datasets")

# Load a specific file
ds = load_dataset("ItsMaxNorm/MedAgentSim-datasets", data_files="medqa_v1.parquet")

# Access the data
print(ds["train"][0])

👥 Contributing

We welcome contributions to MedAgentSim! Please see CONTRIBUTING.md for details on how to get started.

📄 License

This project is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (CC BY-NC-SA 4.0) - see https://creativecommons.org/licenses/by-nc-sa/4.0/ for details.

📚 Citation

If you use MedAgentSim in your research, please cite our paper:

@inproceedings{almansooriandkumarMedAgentSim,
  title={Self-Evolving Multi-Agent Simulations for Realistic Clinical Interactions},
  author={Mohammad Almansoori and Komal Kumar and Hisham Cholakkal},
  booktitle={International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year={2025}
}

🙏 Acknowledgements

• Thanks to AgentClinic, Microsoft PromptBase, and Generative Agents for open-sourcing their works
• Mohamed bin Zayed University of Artificial Intelligence for supporting this research
• We encourage you to support the following three talented artists whose assets were used in this project, especially if you are planning to use the assets included here for your own project:
  • Background art: PixyMoon (@_PixyMoon_)
  • Furniture/interior design: LimeZu (@lime_px)
  • Character design: ぴぽ (@pipohi)

Other Resources

• Open-source multi-agent framework: CAMEL-AI OWL
• Multi-agent framework: OpenMANAUS
• MedPrompt Blog: The Power of Prompting
• MedPrompt Research Paper: https://arxiv.org/abs/2311.16452
• MedPrompt+: Steering at the Frontier: Extending the Power of Prompting
• Microsoft Introduction to Prompt Engineering: https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/prompt-engineering
• Microsoft Advanced Prompt Engineering Guide: https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/advanced-prompt-engineering?pivots=programming-language-chat-completions

📞 Contact

For questions or support, please open an issue or contact the maintainers:

• GitHub: @MAXNORM8650

---

Star History Chart