Hippocrates-X

Multimodal medical AI assistant that fuses chest X-ray analysis, clinical text understanding, audio transcription, and large language model reasoning into a single pipeline for clinical decision support.

Architecture

Image (CXR)  -->  CXformer (87M)         --+
                                            |
Clinical Text -->  Bio_ClinicalBERT (110M) -+--> FusionOrchestrator --> ReasoningEngine --> Response
                                            |        (GPT-4o / OpenBioLLM-8B)
Audio         -->  Distil-Whisper (756M)  --+

Backend: FastAPI, SQLAlchemy (async), PostgreSQL + pgvector, Alembic
Frontend: React, TypeScript, Mantine UI, Recharts
ML: PyTorch, HuggingFace Transformers
Deployment: Docker (multi-stage build), Docker Compose

Evaluation Results

Evaluated against public radiology benchmarks (NIH ChestX-ray14, MIMIC-CXR). Full reports in eval_report.json and eval_report_50.json.

Quick Start

Prerequisites

• Docker and Docker Compose
• OpenAI API key (or HuggingFace token for local models)

1. Configure

cp .env.example .env
# Edit .env with your API keys

2. Start

docker compose up -d

This starts PostgreSQL (pgvector) and the backend. The API is available at http://localhost:8000. The frontend is served from the same port.

3. Verify

python scripts/smoke_test.py

Project Structure

src/
  api/          FastAPI app, routes, schemas, dependency injection
  config/       Pydantic settings (reads .env)
  db/           SQLAlchemy models, repositories, engine
  models/       ML model wrappers
    vision.py     CXformer (m42-health/CXformer-base)
    nlp.py        Bio_ClinicalBERT (emilyalsentzer/Bio_ClinicalBERT)
    audio.py      Distil-Whisper (distil-whisper/distil-large-v3.5)
    reasoning.py  GPT-4o or Llama3-OpenBioLLM-8B
  services/     Business logic
    fusion.py              Multimodal orchestrator, model lifecycle
    consultation.py        Consultation session management
    patient_intelligence.py  AI deep-dive outside consultations
    follow_up.py           AI-generated follow-up recommendations
    analytics.py           Aggregate dashboard queries

app/            React frontend (Vite + Mantine)
alembic/        Database migrations
scripts/        Evaluation and testing tools
tests/          Pytest suite (async, SQLite-backed)

API Endpoints

Evaluation

Run the evaluation suite against local datasets in data/raw/:

# Evaluate on NIH ChestX-ray14 and MIMIC-CXR
python scripts/evaluate.py --dataset all --limit 50 --report eval_report.json

# Load datasets through the pipeline
python scripts/load_public_datasets.py --dataset inventory
python scripts/load_public_datasets.py --dataset nih --limit 20

# Benchmark latency
python scripts/benchmark.py

# Seed demo data (no downloads required)
python scripts/demo_seed.py

Supported Datasets

Testing

# Unit tests (SQLite-backed, no external dependencies)
pytest tests/ -v

# Smoke test against running server
python scripts/smoke_test.py

# Docker smoke test
docker compose --profile test up smoke-test

Configuration

All settings are configured via environment variables (.env):

Model Lifecycle

Models load on first use and unload after idle timeout. The FusionOrchestrator manages this:

• Vision (CXformer): 175 MB, 10 min TTL
• NLP (Bio_ClinicalBERT): 220 MB, 30 min TTL
• Audio (Distil-Whisper): 1.5 GB, 10 min TTL
• Reasoning (OpenBioLLM-8B): 16 GB, 30 min TTL (local) / negligible (OpenAI)

A machine with 16 GB RAM can run the full pipeline by loading models sequentially.

License

This project is for research and educational purposes.