Medical Multimodal Fine-grained RAG Agent

Đồ án 2 — Chương trình Kỹ sư AI
Multimodal Fine-grained RAG-based AI Agent cho domain y tế: nhận query text + ảnh X-quang/MRI/pathology, thực hiện retrieval đa phương thức, fuse evidence, sinh câu trả lời có citation.

---

Trạng thái hiện tại

---

Kiến trúc tổng thể

Query (text + image)
        │
        ▼
┌─────────────────────────────────────────────────────┐
│  LAYER 1 — Data & Knowledge Base                    │
│  MedQA | BioASQ | VQA-RAD | ROCO | MIMIC-CXR |     │
│  PathVQA → Canonical JSONL → data/<dataset>/        │
└──────────────────────┬──────────────────────────────┘
                       │ ingest
┌──────────────────────▼──────────────────────────────┐
│  LAYER 2 — Multimodal Indexing                      │
│  BioMedBERT (1024-dim) → Qdrant text_chunks         │
│  BioCLIP (512-dim) + 5-patch → Qdrant image_patches │
│  Fallback: BM25 + TF-IDF → joblib index            │
└──────────────────────┬──────────────────────────────┘
                       │ query
┌──────────────────────▼──────────────────────────────┐
│  LAYER 3 — LangGraph Agent                         │
│  decompose_query (VLM / rule-based router)          │
│    ├── retrieve_text: Dense + BM25 + BGE rerank     │
│    └── retrieve_visual: BioCLIP → ROI crop → score  │
│  fuse_and_verify + generate_answer (Qwen2.5-VL)     │
└──────────────────────┬──────────────────────────────┘
                       │ evaluate
┌──────────────────────▼──────────────────────────────┐
│  LAYER 4 — Evaluation                               │
│  Recall@k | MRR | Routing accuracy | RAGAS          │
│  Ablation A/B/C/D | Error analysis                  │
└─────────────────────────────────────────────────────┘

---

Cài đặt nhanh

Requirements

• Python ≥ 3.10
• (Optional) CUDA GPU ≥ 16GB cho Qwen2.5-VL local inference
• (Optional) Docker cho Qdrant persistent storage

Baseline (không cần GPU)

# Clone và cài package
git clone <repo-url>
cd KS_Project_2
python -m pip install -e .

# Build index từ data đã có
python -m medical_rag build-index --data-dir data --index-dir data/processed/indexes

# Query thử
python -m medical_rag query "pneumonia treatment" --index-dir data/processed/indexes --top-k 5

Full system (có GPU + Docker)

# Cài full dependencies
python -m pip install -e ".[gpu,qdrant,agent,eval]"

# Start Qdrant
docker run -d -p 6333:6333 -v ${PWD}/qdrant_storage:/qdrant/storage qdrant/qdrant

# Build Qdrant index (cần model downloads lần đầu)
python -m medical_rag build-qdrant-index --data-dir data --limit 500

---

Recommended deployment for RTX 3050 4GB

For the target laptop profile, do not self-host Qwen-VL or build full embeddings locally. Use the split workflow below:

Key commands:

# local diagnostics
python -m medical_rag test-encoders --mock
python -m medical_rag test-qdrant --qdrant-url :memory:
python -m medical_rag test-openrouter

# Kaggle/Colab real indexing
python scripts/colab_workflow.py build-index-resumable --data-dir data --qdrant-url $QDRANT_URL --datasets all --modality both --image-mode full_only --max-minutes 100 --use-cloud-auth

# app on laptop
python demo/app.py --use-agent --use-qdrant

Full runbooks:

• docs/02-development/deployment.md
• docs/02-development/kaggle-workflow.md

---

Sử dụng

CLI commands

# Audit datasets
python -m medical_rag audit-data --data-dir data

# Build index
python -m medical_rag build-index --data-dir data --index-dir data/processed/indexes --limit 500

# Query
python -m medical_rag query "chest xray pneumonia" --index-dir data/processed/indexes --top-k 5

# Evaluate
python -m medical_rag evaluate --eval-file data/eval_cases.json --index-dir data/processed/indexes

# Ablation study
python -m medical_rag ablate --eval-file data/eval_cases.json --output-dir outputs/ablation

# Export static demo
python -m medical_rag export-demo --output-dir demo

Python API — Baseline pipeline

from medical_rag.pipeline import MedicalRAGPipeline
from medical_rag.config import RAGConfig
from medical_rag.indexing import load_indexes

config = RAGConfig(index_dir="data/processed/indexes").resolved()
pipeline = MedicalRAGPipeline(config)
answer = pipeline.run("What treats pneumonia?")
print(answer.answer)

Python API — Agent pipeline

from medical_rag.agents.graph import AgenticRAGPipeline

pipeline = AgenticRAGPipeline({
    "use_qdrant": True,         # requires Qdrant running
    "use_vlm_generation": True, # requires GPU or API key
    "use_fine_grained_visual": True,
    "index_dir": "data/processed/indexes",
})
result = pipeline.run("What does this chest X-ray show?", image_path="xray.jpg")
print(result["answer"])
print(result["citations"])

---

Cấu trúc thư mục

KS_Project_2/
├── data/                        # Datasets (không commit lên git)
│   ├── medqa/                   # 12,723 docs — text-only QA
│   ├── bioasq/                  # 8,216 docs — biomedical QA
│   ├── vqa_rad/                 # 2,244 — visual QA radiology
│   ├── roco/                    # 12,415 — medical image captions
│   ├── mimic_cxr/               # 30,633 — chest X-ray reports
│   ├── pathvqa/                 # 3,600 — pathology VQA
│   └── processed/               # Legacy processed JSONL + joblib index
│       └── indexes/             # rag_indexes.joblib (baseline index)
├── src/medical_rag/             # Main Python package
│   ├── agents/                  # LangGraph agent nodes (Phase 3)
│   ├── models/                  # BioCLIP, BioMedBERT, Qwen2.5-VL (Phase 1)
│   ├── ingestion/               # Chunker + Qdrant indexer (Phase 2)
│   ├── retrieval/               # Baseline BM25+TF-IDF retrievers
│   ├── data_tools/              # Dataset canonicalization
│   ├── pipeline.py              # Baseline MedicalRAGPipeline
│   ├── config.py                # RAGConfig (all settings)
│   ├── schema.py                # Pydantic data models
│   └── __main__.py              # CLI entrypoint
├── docs/                        # Documentation
├── tests/                       # Pytest smoke tests
├── scripts/                     # Dataset download utilities
├── outputs/                     # Evaluation results, ablation JSONs
├── demo/                        # Static HTML demo export
└── pyproject.toml               # Package config + dependencies

Xem thêm: docs/02-development/folder-structure.md

---

Datasets

---

Tech stack

---

Kết quả hiện tại

Kết quả tổng hợp từ TASK.md, outputs/ablation_full/, và outputs/benchmark/:

Advanced baseline:

[!NOTE] Exact Match và Token F1 thấp vì generator hiện trả lời dạng evidence draft/citation-rich, không phải short-form answer. Project đã bổ sung các quality proxy nhẹ như answer non-empty, citation coverage, evidence overlap và groundedness proxy để báo cáo công bằng hơn.

Điểm cần cải thiện tiếp:

• BioASQ retrieval còn yếu trong local benchmark.
• Clean OpenRouter/Qdrant benchmark đã ổn nhưng mới ở small set 6 cases.
• Gradio demo đã sẵn sàng để test local nhưng chưa được xác nhận như final deployment.

---

Đóng góp

Xem CONTRIBUTING.md và docs/02-development/coding-standards.md.

License

MIT — xem LICENSE.

Tài liệu tham khảo

Papers được tham khảo trong src/support_repo/:

• MMed-RAG (ICLR 2025): Medical multimodal RAG baseline
• HM-RAG (ACM MM 2025): Hierarchical multi-agent RAG
• VimRAG/VRAG: Progressive visual retrieval, coarse-to-fine
• A-MAR: Adaptive retrieval loop, structured reasoning