Medical AI Diagnostic Orchestrator with Vision Module (MAI-DxO-Vision)

1. Project Background

Microsoft introduced the MAI-DxO (Medical AI Diagnostic Orchestrator) framework in 2025, which leverages multiple Large Language Models (LLMs) acting as different clinical roles (e.g., hypothesis-driving physician, challenging specialist) forming a "virtual expert panel". This architecture demonstrates superior performance compared to single human physicians in text-only sequential diagnostic tasks.

Key Challenge & Limitation: The current open-source MAI-DxO implementation (Open-MAI-Dx-Orchestrator) is purely text-driven and lacks the ability to directly process critical visual information in real-world medical scenarios, such as chest X-rays and other medical imaging.

---

2. Project Core Objective

This project aims to bridge this architectural gap by integrating a Vision Module into the existing open-source MAI-DxO text-based multi-agent framework, upgrading it to support multimodal inputs (clinical text records + medical images) for comprehensive medical diagnostic intelligence.

Recognizing the absence of a perfect open-source dataset combining high-quality real medical images with complex interactive diagnostic scenarios, we innovatively propose and implement a "Decoupled Evaluation Strategy" that rigorously validates system component usability and end-to-end connectivity without compromising academic rigor.

---

3. Core Implementation Methodology

The project validation logic is strictly divided into two decoupled phases:

Phase 1: Vision Component Development & Real-World Validation (Component-level Validation)

• Objective: Demonstrate that the vision module can accurately extract pathological features from real clinical medical images.
• Datasets Used: MIMIC-CXR and CT-RATE datasets (real chest X-rays / real chest CT scans + authentic radiologist reports)
• Approach:
  • Feed real medical images into the vision module (e.g., Vision Language Models)
  • Generate text-based image analysis reports
  • Evaluate similarity against ground truth radiologist reports using NLP metrics (BLEU, ROUGE, etc.)

Phase 2: Agent Integration & Robustness Validation (System-level Integration Testing)

• Objective: Validate the end-to-end pipeline: "visual input → text translation → multi-agent sequential reasoning → final diagnosis", and verify MAI-DxO's robustness to modality translation noise.
• Datasets Used: MedQA dataset (text-only clinical cases) + text-to-image generation models
• Approach (Closed-loop Mock Testing):
  1. Use a generative model to create synthetic medical images from MedQA clinical case descriptions
  2. Feed these synthetic images through the validated vision module to extract re-translated radiology reports
  3. Input the re-extracted reports along with clinical case information to the MAI-DxO agent group
  4. Evaluate whether MAI-DxO can reach correct diagnoses despite the visual-to-text translation noise
  5. Verify fidelity of synthetic images using: Text_GT → (image generation model) → synthetic image → (vision module) → Text_generated → (similarity comparison) → verify Text_generated ≈ Text_GT

---

4. Vision Module Overview

The Vision Module serves as the bridge between medical imaging and diagnostic reasoning:

• Purpose: Automatically converts medical images (chest X-rays, CTs, etc.) into detailed clinical text descriptions
• Function: Extracts and describes pathological findings, anatomical structures, and diagnostic observations from medical images
• Integration: Seamlessly feeds structured image analysis into the MAI-DxO framework alongside traditional clinical text records
• Technology: Leverages state-of-the-art Vision Language Models to ensure accurate and clinically relevant image interpretation

---

5. Installation & Setup

System Requirements

• Python: 3.10 or higher
• CUDA: 11.8 or higher (for GPU support)
• Memory: 16GB GPU VRAM minimum (varies by model size)

Vision Module Installation (Hulu-Med)

The vision module is built upon Hulu-Med, a transparent generalist medical vision-language model. Follow these steps to install:

Step 1. Create Conda Environment

# Create a new conda environment with Python 3.10
conda create -n medagent python=3.10
conda activate medagent

Step 2: Install PyTorch and Dependencies

# PyTorch and torchvision for CUDA 11.8
pip install torch==2.4.0 torchvision==0.19.0 --extra-index-url https://download.pytorch.org/whl/cu118

# Flash-attn for optimized attention
pip install flash-attn==2.7.3 --no-build-isolation --upgrade

# Transformers and accelerate
pip install transformers==4.51.2 accelerate==1.7.0

# Video processing and image handling
pip install decord ffmpeg-python imageio opencv-python

Step 3: Install Additional Dependencies

# Install remaining requirements
pip install -r requirements.txt

6. Vision Module Implementation

The vision module is implemented through a set of Python scripts in the vision_module/ directory:

Core Scripts

1. inference.py - Medical Image Inference

Purpose: Processes medical images (CT scans, X-rays) using Hulu-Med-14B to generate clinical findings descriptions.

Functionality:

• Loads Hulu-Med-14B model with memory optimization (gradient checkpointing, chunked image encoding)
• Processes medical images from specified directories
• Generates natural language clinical findings
• Outputs results in JSON format with image-text pairs

Usage:

python vision_module/inference.py

Output: JSON file containing generated medical findings for each image

---

2. benchmark.py - Evaluation & Quality Assessment

Purpose: Evaluates the quality of generated medical descriptions against ground truth reports using multiple metrics.

Supported Metrics:

• ROUGE (ROUGE-1, ROUGE-2, ROUGE-L): Lexical similarity and recall
• BLEU-4: Precision of medical terminology phrasing
• BERTScore: Semantic similarity using Bio_ClinicalBERT

Supports Multiple Datasets:

• CT Dataset: CT-RATE with reference reports
• X-ray Dataset: MIMIC-CXR with radiologist ground truth

Usage:

# Evaluate X-ray model outputs (default)
python vision_module/benchmark.py xray

# Evaluate CT model outputs
python vision_module/benchmark.py ct

Output: Evaluation metrics (ROUGE, BLEU, BERTScore) for both Findings and Impressions sections

---

3. evaluate.py - Batch Inference & Evaluation

Purpose: Processes a batch of medical volumes and stores evaluation results.

Functionality:

• Loads medical volume data from CT-RATE dataset
• Generates descriptions for each volume
• Stores both generated text and evaluation metadata
• Optimized for batch processing with memory management

Usage:

python vision_module/evaluate.py

7. Project Structure

comp5423-course-project/
├── README.md              # Project overview
├── README_EN.md          # English version
├── vision_module/        # Vision Module Implementation
│   ├── Hulu-Med/              # Hulu-Med model repository
│   ├── VISION_MODULE_REPORT.md      # Technical report (English)
│   ├── inference.py       # Medical image inference script
│   ├── benchmark.py       # Quality evaluation script
│   └── evaluate.py        # Batch evaluation script
├── mai_dx/              # MAI-DxO Framework
├── cases/               # Test cases and examples
├── results/             # Evaluation results
└── MedQA/              # MedQA dataset files