DermAgent: A Self-Reflective Agentic System for Dermatological Image Analysis with Multi-Tool Reasoning and Traceable Decision-Making

A self-reflective agentic system for dermatological image analysis, built on LangChain/LangGraph.

DermAgent orchestrates seven specialist vision and language tools (PanDerm, MAKE, DermoGPT, Qwen3-VL, Case RAG, Guideline RAG, Ontology) within a Plan-Execute-Reflect framework, using GPT-4o as the reasoning backbone. A deterministic Critic module performs post-hoc auditing via confidence, coverage, and conflict gates to trigger targeted self-correction, delivering stepwise, traceable diagnostic reasoning.

Architecture

Project Structure

DermAgent/
├── .agents/skills/              # Repo-level Codex skills
│   └── mayo-dermnet-corpus-pipeline/ # Guideline corpus reproduction contract
├── skin_agent/                  # Core agent framework
│   ├── benchmark_agent.py       # Benchmark agent + Critic + AnswerParser
│   ├── configs.py               # Dataset task configurations
│   ├── tracing.py               # TraceLogger, TracingCallback
│   ├── profiler.py              # Performance profiling
│   ├── resume.py                # Checkpoint/resume for long runs
│   ├── prompts.md               # System prompts
│   ├── tools/
│   │   ├── base.py              # BaseSkinTool, input schemas
│   │   ├── skin_tools.py        # All 7 tool implementations
│   │   ├── executor.py          # Tool execution orchestration
│   │   └── derm_knowledge_tree/ # Disease ontology JSONs
│   └── utils/
│       ├── retry.py             # Rate-limit retry logic
│       └── image_utils.py       # Image path handling
├── benchmark/                   # Unified evaluation framework
│   ├── run.py                   # CLI runner for single-model baselines
│   ├── metrics.py               # Shared metrics (classification, multilabel, captioning, VQA)
│   ├── models/                  # Model wrappers (GPT-4o, LLaVA-Med, HuatuoGPT, etc.)
│   └── datasets/                # Dataset configs with prompts and class lists
├── scripts/                     # All runnable scripts
│   ├── build_qdrant_db.py       # Build image RAG vector database
│   ├── build_qdrant_rag.py      # Build text RAG (guidelines)
│   ├── run_task1_ham10000_500_agent_dermogpt_full_critic.sh
│   ├── run_task1_snu_500_critic.sh
│   ├── run_task2_task3_agent_critic.sh
│   ├── run_task3_loo_ablation.sh
│   └── *.py                     # Python runner scripts
├── baselines/                   # Agent-based baseline reproductions
│   ├── MDAgents/                # MDAgents agent baseline (NeurIPS 2024)
│   ├── MedAgent-Pro/            # MedAgent-Pro agent baseline
│   └── SkinVL/                  # SkinVL-PubMM baseline
├── data/                        # Benchmark CSV metadata
├── requirements.txt
└── .env.example

Setup

1. Environment

conda create -n dermagent python=3.10
conda activate dermagent

# Install PyTorch (match your CUDA version)
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu126

pip install -r requirements.txt

# Download NLTK data (needed for BLEU/ROUGE metrics)
python -c "import nltk; nltk.download('punkt'); nltk.download('punkt_tab')"

2. API Keys

cp .env.example .env
# Edit .env: set OPENAI_API_KEY (required) and other variables as needed

3. External Dependencies

The following external code/data directories are required but not included in this repository due to size or licensing. Place them at the project root:

For Text RAG models, pre-download the embedding and reranker models into model-weights/:

# Pre-download Qwen3 Embedding and Reranker for Text RAG
huggingface-cli download Qwen/Qwen3-Embedding-8B --local-dir model-weights/Qwen3-Embedding-8B
huggingface-cli download Qwen/Qwen3-Reranker-0.6B --local-dir model-weights/Qwen3-Reranker-0.6B

4. Datasets

Download the following datasets and place images in the expected directories:

Benchmark CSV metadata (split definitions) are included in data/.

5. RAG Vector Database

To reproduce the Mayo Clinic and DermNet source corpora without redistributing scraper code or article content, follow the repo-level Mayo + DermNet corpus pipeline skill. It documents the Python environment, final pipeline, and JSON contract expected by scripts/build_qdrant_rag.py.

Install and start Qdrant (vector database server):

# Option A: Docker (recommended)
docker run -d -p 6333:6333 -v $(pwd)/qdrant_storage:/qdrant/storage qdrant/qdrant

# Option B: Local binary — see https://qdrant.tech/documentation/guides/installation/

Build the Qdrant vector databases:

# Image-based case retrieval (requires Derm1M dataset + Derm1M/src/)
python scripts/build_qdrant_db.py

# Guideline-grounded text retrieval (requires RAG/ JSON files)
python scripts/build_qdrant_rag.py

6. Model Weights

Most tool models are auto-downloaded from HuggingFace on first use:

• PanDerm: DermLIP_PanDerm-base-w-PubMed-256 (~2GB VRAM)
• MAKE: MAKE (~2GB VRAM)
• Qwen3-VL: Qwen3-VL-8B-Instruct (~16GB VRAM, bfloat16)

Models that must be manually placed in model-weights/:

• DermoGPT-RL: See Step 3 above (~16GB VRAM, bfloat16)
• Qwen3-Embedding-8B / Qwen3-Reranker-0.6B: See Step 3 above (Text RAG)

Total GPU requirement: ~20-22GB for all tools loaded simultaneously.

Table 1: Main Results

Commands to reproduce DermAgent results:

# HAM10000 Diagnosis
bash scripts/run_task1_ham10000_500_agent_dermogpt_full_critic.sh

# SNU Diagnosis
bash scripts/run_task1_snu_500_critic.sh

# Derm7pt + SkinCon + SkinCAP
bash scripts/run_task2_task3_agent_critic.sh

Commands to reproduce baseline results (see baselines/README.md for agent baselines):

# Single-model MLLM baselines (e.g., GPT-4o on HAM10000)
cd benchmark && python run.py --model gpt4o --dataset HAM10000_500

# MDAgents agent baseline
cd baselines/MDAgents && python run_derm_benchmark.py --dataset HAM10000 --difficulty basic --model gpt-4o

# MedAgent-Pro agent baseline
cd baselines/MedAgent-Pro && python Derm_Case_level.py --task 1 \
    --csv-path ../../datasets/ham10000/HAM10000_benchmark_500.csv \
    --image-dir ../../datasets/ham10000 --max-samples 500

Table 2: Ablation Study (Leave-One-Out on SkinCAP)

Command to reproduce ablation results:

# Runs 6 leave-one-out experiments sequentially (removes one tool at a time)
bash scripts/run_task3_loo_ablation.sh

The full agent result (w/ Critic, ROUGE-L: 0.1948) is produced by the Task 3 portion of run_task2_task3_agent_critic.sh. The "w/o Critic" baseline (0.1727) is produced by the LOO script's full-tool run without the Critic module.

Tool Descriptions

Citation

Paper: arXiv:2605.14403 (MICCAI 2026, early accept).

@article{liu2026dermagent,
  title={DermAgent: A Self-Reflective Agentic System for Dermatological Image
         Analysis with Multi-Tool Reasoning and Traceable Decision-Making},
  author={Liu, Yize and Yan, Siyuan and Hu, Ming and Ju, Lie and Li, Xieji and
          Tang, Feilong and Feng, Wei and Ge, Zongyuan},
  journal={arXiv preprint arXiv:2605.14403},
  year={2026}
}

License

This project is licensed under the Apache License 2.0 — see the LICENSE file for the full text.

DermAgent redistributes adapted code from third-party projects under the baselines/ directory. See NOTICE and the per-baseline ATTRIBUTION.md files for upstream sources, citations, and license status.