MedUMM

MedUMM is an open platform for inference, evaluation, reporting, and post-training of medical unified multimodal models. Its stable interfaces let models, datasets, benchmarks, and training methods evolve independently.

Research use only. MedUMM is not a medical device and must not be used for diagnosis, treatment, or other clinical decisions.

v1.7: distributed post-training substrate

Version 1.7 adds a model-independent PyTorch training substrate instead of only generating torchrun commands. Model-specific trainers can now reuse initialized process groups, device-local DDP or FSDP, gradient accumulation with no_sync, FP32/FP16/BF16 autocast, gradient scaling and clipping, non-reentrant activation checkpointing, per-rank EMA, and resumable model/optimizer shards written through torch.distributed.checkpoint.

Checkpoints also preserve the scheduler, AMP scaler, EMA, Python/Torch/CUDA RNG, epoch, micro-batch position, optimizer step, and sample count. An incomplete checkpoint is never selected by resume_from: auto, and retention only removes completed checkpoints. The included distributed_reference trainer is a real small PyTorch optimization job used to test the substrate; its synthetic data is not a medical model or quality result.

python -m torch.distributed.run --standalone --nproc-per-node=2 \
  -m medumm post-train \
  --config configs/post_training/distributed_reference_ddp.yaml

sbatch --export=ALL,MEDUMM_STRATEGY=fsdp \
  scripts/slurm_distributed_training_v1.7.sh

The Slurm recipe deliberately interrupts after two optimizer steps, restarts from the sharded checkpoint, and validates all rank sidecars and EMA state. Public API, configuration, checkpoint layout, topology limits, and acceptance criteria are in docs/distributed-training-v1.7.md. Committed two-node DDP/FSDP recovery and one-A800 runtime evidence is available in docs/results/v1.7-distributed-training.json.

v1.6: independent medical benchmark matrix

Version 1.6 separates executable benchmarks from the dataset catalog. MedUMM now has 13 specialized medical benchmark adapters, two generic benchmark adapters, and one composite cross_task runner. The 34 dataset entries remain data resources; they are no longer presented as 34 complete evaluations.

Every specialized adapter fixes its compatible dataset family, normalized annotation contract, medical prompt, metric-suite version, audit gates, and audit → inference → dedicated scoring → report path. The matrix covers pathology VQA, medical MCQA, single-label and multilabel recognition, radiology reports, grounding, measurement, temporal reasoning, retrieval, calibration, fairness, safety, and paired robustness.

medumm benchmarks list
medumm benchmarks show medical_grounding
medumm benchmarks audit
medumm evaluate --config configs/evaluation/benchmarks_v1.6/medical_grounding.yaml

The committed deterministic fixture validates software contracts only. It is not reported as a medical quality score. All 13 paths passed Slurm job 437789 on node15 (21 aligned fixture predictions). See docs/specialized-benchmarks-v1.6.md for the benchmark/data distinction, all task contracts, commands, outputs, validation scope, and remaining real-dataset queue. The exact acceptance record is machine-readable.

v1.5: four more real medical model runtimes

Version 1.5 validates four additional open-weight medical multimodal releases through the same public medumm infer interface: MedMO-4B and MedMO-8B use Qwen3-VL generation, Lingshu-I-8B uses the native Transformers InternVL model with its upstream MPT prompt protocol, and Fleming-VL-8B uses its pinned official InternVL chat implementation. Each model has an immutable weight revision and an independently checked Python environment.

All four completed offline inference in Slurm job 437697 on one NVIDIA A800-SXM4-80GB. MedUMM now has committed GPU runtime evidence for 11 of 32 catalog models; the other 21 remain interface or contract claims only. MedSigLIP, MedGemma 1.5 4B, and MAIRA-2 were also probed and remain explicitly blocked by upstream access terms on the current server. See docs/real-model-adapters-v1.5.md and the machine-readable A800 evidence.

v1.4: explicit real-model adapters

Version 1.4 replaces broad catalog-family routing with one explicit adapter recipe for every one of the 32 model releases. Each recipe fixes the real model type/class, processor, prompt protocol, executor, immutable weight revision, and—when required—the pinned official source entry point. The new medumm models CLI audits recipe/resource/environment coverage and performs local asset, revision, access, source-commit, and import preflight checks.

Four additional real-weight paths passed one offline A800 Slurm acceptance job: PLIP and QuiltNet through native Transformers CLIP ranking, MedVLM-R1 through Qwen2-VL generation, and BiomedCLIP through a local pinned OpenCLIP snapshot and pinned BiomedBERT text encoder. This raises committed GPU evidence from 3 to 7 of 32 models. It does not turn the remaining 25 catalog entries into runtime claims; repository-specific executors and model-scale validation remain in the queue. See docs/real-model-adapters-v1.4.md and the A800 evidence.

v1.3: isolated model environments

Every one of the 32 catalog models now has an independent, immutable runtime contract. Each contract pins Python, CUDA, the base-container digest, Python packages, model revision, upstream source commits, access policy, import probes, GPU guidance, and validation evidence. Generated requirements, Docker and Apptainer definitions live under environments/models/<model>/; Modal images, local virtual environments, and Slurm/Apptainer execution all consume the same catalog so their dependency stacks cannot silently diverge.

medumm environments list
medumm environments show llava_med_v1_5_7b
bash scripts/setup_model_env.sh lingshu_7b
sbatch --export=ALL,MODEL_NAME=lingshu_7b scripts/slurm_model_environment.sh

Restricted models fail closed until --accept-terms is passed after the user accepts the upstream license. Contract/build validation is not reported as a successful model run: v1.4 adds four new runtime-validated paths to the three previously committed ones. See docs/model-environments-v1.3.md. The machine-readable validation record, including the cluster container-build privilege boundary, is in docs/results/v1.3-model-environments.json.

v1.2: optimized inference engines

Version 1.2 adds explicit vLLM and SGLang backends, OpenAI-compatible serving, continuous request batching, TP/PP/DP launch controls, and a shared inference performance benchmark. Emu3.5 has a separate fail-closed native vLLM adapter: it requires vLLM 0.11.0 plus BAAI's 20 official patches, sends conditional and unconditional token streams through the custom CFG scheduler, and supports two-GPU tensor parallel inference. Standard vLLM/SGLang HTTP servers do not claim that Emu3.5 CFG path.

Use medumm backends, medumm serve, and medumm benchmark-inference to inspect, launch, and measure the new paths. Reproducible setup, configs, A800 Slurm acceptance, metrics, capability boundaries, and validation levels are in docs/inference-optimization-v1.2.md. The vLLM and SGLang two-A800 paths passed real TP=2 sequential/concurrent benchmarks; the evidence record keeps the full Emu3.5 weight run explicitly blocked until its pinned assets and FlashAttention dependency are available.

v1.1: research post-training CLI

Version 1.1 adds one validated CLI contract for BAGEL SFT, RecA, Uni-CoT, IRG, UniGame, UniPath, and LatentUMM. Each method retains its real stage graph, data requirements, official/reference runtime boundary, and checkpoint dependencies. The CLI now supports route discovery, safe templates, preflight-only planning, pinned-source checks, medical data governance gates, child-process failure propagation, and checkpoint/log collection.

The dependency-light acceptance suite invokes the public CLI independently for all seven methods (13 ordered stages), while explicitly making no paper- fidelity claim. Every method receives its own preflight, CLI log, stage logs, checkpoints, and result. See docs/post-training-routes-v1.1.md. The successful Slurm evidence and the first failed attempt are both preserved in docs/results/v1.1-post-training-routes.json.

v1.0: medical-specific evaluation contracts

Version 1.0 adds independently registered, versioned metric suites for pathology VQA, annotated report factuality, spatial grounding, unit-aware measurement, and probabilistic calibration/selective prediction. Model candidate scores now survive the shared evaluation runner, and every suite can produce subgroup reports plus minimum-sample-gated disparity summaries.

The first real acceptance slice uses a pinned PathVQA test export with both yes/no and free-form questions through the native Lingshu adapter. Missing structured annotations remain explicitly unavailable rather than being counted as failures. See docs/clinical-evaluation-v1.0.md for contracts, schemas, limits, the reproducible server recipe, and the passed A800 evidence.

v0.9: architecture-diverse runtime slices

Version 0.9 turns the first items in the v0.8 validation queue into reproducible vertical slices:

• a native lingshu_7b Qwen2.5-VL adapter with multi-image input, immutable revision enforcement, official chat templating, and CUDA/Slurm evidence;
• source-pinned SLAKE and PathVQA exporters into the common medical VQA schema;
• a real PubMedCLIP contrastive path with per-sample candidate ranking;
• a fixed MedMNIST v2 PneumoniaMNIST exporter and zero-shot classification evaluation through the same benchmark/report contract;
• CPU asset-preparation and offline A800 acceptance jobs for both model families.

Both open slices passed a pinned A800 Slurm acceptance run and are marked runtime_validated. Gated MedSigLIP remains interface-validated until its upstream terms are accepted and weights are available. See docs/runtime-slices-v0.9.md and the committed machine-readable evidence.

v0.8: scale catalog for medical models and evaluation data

Version 0.8 registers 32 medical multimodal model releases and 34 evaluation datasets behind the stable platform interfaces. Every resource has an audited primary source, paper/code links, license and access level, revision policy, medical domains, tasks/modalities, an executor or normalized-dataset family, and an explicit validation status.

The catalog distinguishes interface_validated from runtime_validated. Only real pinned server runs receive the latter; a model name or downloadable weight is not counted as execution evidence. Gated and credentialed resources require explicit acceptance/access flags, and remote runs require immutable revisions. See docs/resource-catalog-v0.8.md. The v0.8 A800 acceptance evidence is stored in docs/results/v0.8-scale-catalog.json.

v0.7: advanced post-training and research methods

Version 0.7 adds a real parameter-efficient alignment layer rather than another classification-style training alias:

• one medical_alignment trainer for causal-LM SFT, DPO, SimPO, ORPO, and clinical-relevance-weighted DPO;
• LoRA and optional 4-bit QLoRA loading with a self-describing PEFT checkpoint;
• provenance-aware supervised/preference records with rationale, annotation source, safety category, specialty, task, and clinical-relevance fields;
• deterministic weighted multi-dataset mixtures with source-namespaced sample identifiers and content-stable fingerprints;
• data gates for license, de-identification, preference provenance, rationale, non-expert disclosure, and invalid relevance weights;
• token-level completion masking, frozen-reference DPO without a second model copy, gradient/history evidence, and independent adapter reload verification.

The real acceptance recipe performs LoRA-DPO on a pinned eight-pair UltraMedical-Preference slice. It uses a small Apache-2.0 Pythia-14M research model so the training system is cheap to reproduce; it is not a medical model quality claim. See docs/advanced-post-training-v0.7.md. The verified A800 evidence is stored in docs/results/v0.7-advanced-post-training.json.

v0.6: task-aware medicine, not natural-image classification

Version 0.6 adds a medical semantic layer across perception, reasoning, and long-form generation:

• eight stable intents for finding assessment, clinical description, anatomy localization, quantitative assessment, imaging context, diagnostic reasoning, report generation, and patient communication;
• a medical_tasks_jsonl contract with task, concept, evidence, case/turn, and reference-provenance fields;
• a medical_tasks benchmark and medical_task_core suite with task-specific success, concept/evidence coverage, negation-aware extra concepts, strict diagnosis, and uncertainty intervals;
• audit gates that distinguish expert/native task labels from transparent heuristic mappings;
• a balanced 24-sample VQA-RAD + LLaVA-Med A800 acceptance recipe covering six tasks supported by real source questions, without fabricating report or patient-communication labels.

The task hierarchy is informed by the open diagnosis, clinical explanation, and interaction goals described in VisionUnite. MedUMM does not redistribute its weights or treat its pretrained model license as part of this implementation. See docs/medical-tasks-v0.6.md for the schema, metrics, provenance rules, and server recipe. The verified A800 evidence is stored in docs/results/v0.6-medical-tasks.json.

v0.4: medical evaluation base

Version 0.4 turns the first real-model slice into a reusable medical evaluation foundation:

• a versioned evaluation protocol and independently registered metric suite;
• preflight dataset quality/governance audits with content and provenance hashes;
• closed/open VQA metrics, configurable subgroup reports, abstention, and seeded bootstrap confidence intervals;
• per-batch atomic prediction checkpoints, fingerprint-safe resume, automatic Slurm/torchrun sharding, and strict deterministic shard merge;
• protocol-aware score files, CSV exports, leaderboards, and real A800 evidence.

See docs/medical-evaluation-v0.4.md for the contract, CLI examples, artifacts, and acceptance criteria.

v0.3: first real medical model

Version 0.3 connects the stable platform to a real biomedical vision-language model and public medical benchmark:

• llava_med: Microsoft LLaVA-Med v1.5 Mistral-7B through its official model implementation;
• flaviagiammarino/vqa-rad: a revision-pinned VQA-RAD test slice normalized into the MedUMM dataset contract;
• real CUDA inference and generate → score → report evaluation on one A800;
• run evidence containing exact revisions, device/dtype, latency, peak allocated GPU memory, predictions, and grouped medical-VQA metrics.

This first vertical slice supports medical image understanding only. Generation, editing, broader datasets, and post-training remain platform capabilities or future real-model slices; v0.3 does not claim clinical model quality.

See docs/real-model-v0.3.md for the reproducible server recipe, pinned revisions, and license boundaries.

Stable platform interfaces

Version 0.2 establishes one four-layer platform architecture:

1. Application and API: schema-versioned YAML, one CLI, typed Python API, run manifests, reports, and leaderboards.
2. Task and execution: separate understanding, generation, and editing pipelines plus benchmark-neutral cross-task evaluation.
3. Core functionality: lazy registries for model, dataset, benchmark, and post-training plugins; typed requests and results; declared model capabilities.
4. Infrastructure: abstract interfaces, runtime/distributed context, metrics and atomic I/O, and adapters over model backbones.

The architecture and extension contracts are specified in docs/architecture.md. v0.2 validates the entire platform with dependency-light synthetic data; it does not claim clinical model quality.

Install

MedUMM requires Python 3.10 or newer.

python -m venv .venv
source .venv/bin/activate
pip install -e ".[test,baseline]"

Install the medical extra only for heavyweight Transformers backbones:

pip install -e ".[medical]"

LLaVA-Med uses the upstream implementation's older Transformers compatibility window. The legacy helper remains available, but the common per-model command is preferred and never modifies the parent Conda environment:

bash scripts/setup_model_env.sh llava_med_v1_5_7b

Docker

The root Dockerfile builds the common MedUMM CUDA platform without embedding model weights, evaluation data, credentials, or outputs. Its default dependency set supports the platform CLI, common Transformers and contrastive adapters, dataset preparation, serving, and distributed execution.

# Build the runtime image. Pass the current commit into the OCI metadata.
docker build \
  --build-arg VCS_REF="$(git rev-parse HEAD)" \
  -t medumm:1.7.0 .

# Check the image without downloading or loading model weights.
docker build --target smoke -t medumm:smoke .

# Inspect the platform catalog.
docker run --rm medumm:1.7.0 catalog --json

# Run a GPU evaluation with host assets mounted read-only and outputs writable.
docker run --rm --gpus all \
  -v /srv/medumm/models:/models:ro \
  -v /srv/medumm/data:/data:ro \
  -v /srv/medumm/outputs:/outputs \
  -v "$PWD/configs:/workspace/configs:ro" \
  medumm:1.7.0 evaluate --config /workspace/configs/evaluation/example.yaml

The base image and major PyTorch versions are build arguments. For example, a platform-only image can omit heavyweight model libraries:

docker build \
  --build-arg MEDUMM_EXTRAS=baseline,data,serve,distributed \
  -t medumm:platform .

Do not combine incompatible model stacks in this common image. Models such as LLaVA-Med require older pinned Transformers releases; build their generated definitions under environments/models/<model>/Dockerfile instead. Those definitions and the root image share /models, /data, /outputs, and the Hugging Face cache conventions, so deployment mounts remain consistent.

Unified CLI

# Inspect plugins without loading model weights
medumm catalog

# Inspect and validate the medical resource catalog
medumm resources list --kind model
medumm resources list --kind dataset
medumm resources validate
medumm resources template vqa_rad --kind dataset

# Post-train, infer, and evaluate
medumm post-train --list-methods
medumm post-train --template reca > reca-medical.yaml
medumm post-train --config reca-medical.yaml --plan
medumm post-train --config configs/post_training/medical_sft_smoke.yaml
medumm infer --config configs/inference/medical_reference_workflow.yaml
medumm evaluate --config configs/evaluation/medical_vqa_linear_smoke.yaml

# Aggregate one or more score reports
medumm report \
  --scores outputs/evaluation/medical_vqa_linear/score.json \
  --output-directory outputs/reports/smoke

All execution commands accept repeatable dotted overrides such as --set evaluation.data.max_samples=2.

Unified YAML

Every new config declares schema_version: "1.0" and contains exactly one execution block:

schema_version: "1.0"
runtime:
  seed: 42
  device: auto
evaluation:
  benchmark: medical_vqa
  data:
    adapter: medical_vqa_jsonl
    path: examples/medical/tiny_eval.jsonl
    image_root: examples/medical/images
  model:
    backbone: medical_reference
    parameters:
      fixed_answer: A
  mode: full       # generate, score, or full
  batch_size: 2
  resume: true
  output_directory: outputs/evaluation/example

Flat v0.1 evaluation configs remain readable. New files should use the unified envelope above.

Python API

Use the high-level functions for config-driven work:

from medumm import evaluate, infer, post_train

results = infer({
    "schema_version": "1.0",
    "inference": {
        "backbone": "medical_reference",
        "requests": [{
            "task": "understanding",
            "prompt": "Describe the supplied image.",
            "images": ["examples/medical/images/synthetic_scan.pgm"],
        }],
    },
})
print(results[0].text)

Use InferencePipeline when managing a loaded model directly:

from medumm import InferencePipeline, InferenceRequest

with InferencePipeline("medical_reference", {}) as pipeline:
    result = pipeline.run(InferenceRequest(
        request_id="case-001",
        task="generation",
        prompt="Synthetic medical phantom for software testing",
    ))

InferenceResult, EvaluationResult, and TrainingResult have stable to_dict() representations with schema version 1.0.

Evaluation and reproducibility

Evaluation supports four explicit states:

• audit: validate dataset quality and governance without loading a model;
• generate: write fingerprinted predictions.jsonl only;
• score: score an existing compatible prediction file without loading a model;
• full: generate missing predictions, then score them.

Scoring writes results.jsonl, score.json, and metrics.csv. CLI runs also write run_manifest.json containing the resolved config, component identity, run ID, software environment, Git commit, and result. The cross_task benchmark composes any registered benchmarks behind the same result contract.

Distributed workers automatically emit rank-local artifacts. Merge them only after all ranks finish:

medumm merge-predictions \
  --shards outputs/evaluation/run/predictions.rank-*.jsonl \
  --output outputs/evaluation/run/predictions.jsonl \
  --expected-count 500

Built-in plugins

The v0.8 resource catalog adds 32 individually registered model resources and 34 individually registered dataset resources. Run medumm resources list for the current machine-readable inventory instead of maintaining a duplicate table here.

The reference and linear plugins prove platform behavior; they are not medical foundation models. Model and dataset expansion follows docs/roadmap.md.

Run the smoke workflow

bash scripts/run_reference_workflow.sh

For Slurm:

sbatch scripts/slurm_reference_workflow.sh

The real-model workflow separates network-bound asset preparation from GPU execution:

sbatch scripts/slurm_prepare_llava_med_assets.sh
sbatch scripts/slurm_llava_med_vqa_rad.sh
sbatch scripts/slurm_medical_tasks_v0.6.sh
sbatch scripts/slurm_prepare_runtime_slices_v0.9.sh
sbatch scripts/slurm_runtime_slices_v0.9.sh
bash scripts/setup_medical_cuda126_env.sh
MEDUMM_DYNAMIC_PROXY_TARGET=REACHABLE_LOGIN_HOST sbatch scripts/slurm_prepare_pathvqa_v1.0.sh
sbatch scripts/slurm_clinical_evaluation_v1.0.sh

Outputs are written below outputs/ and ignored by Git.

Repository layout

src/medumm/
  api.py            high-level Python API
  cli/              unified command line
  core/             contracts, registries, config, runtime, I/O
  inference/        understanding, generation, editing execution
  backbones/        model adapters
  medical/          medical schemas, datasets, metrics
  evaluation/       benchmark and cross-task runners
  post_training/    registered training methods
  reporting/        reports and leaderboards
configs/            schema-versioned execution recipes
examples/medical/   synthetic, non-clinical smoke data
scripts/            local and Slurm workflows
tests/              contract and end-to-end tests

License

Apache-2.0. Model weights and datasets retain their own licenses and terms.