ARR Radiology Code Submission

This repository contains an anonymized code artifact for radiology vision-language experiments. It includes the main components used to construct fine-grained radiology alignment data, train a radiology FG-CLIP model, and evaluate downstream retrieval, classification, and report-generation tasks.

Raw datasets, model checkpoints, and some third-party baseline packages are not provided in this repository.

Repository Layout

arr_radiology_code_submission/
├── configs/                         # DeepSpeed and classification prompt configs
├── docs/                            # Data schema and code maps
├── scripts/                         # Reproducible entry-point wrappers
├── src/
│   ├── dataset/                     # Data construction and manifest builders
│   ├── evaluation/                  # Retrieval and classification evaluators
│   ├── extract/                     # Feature extraction for report generation
│   ├── fgclip_sim/                  # FG-CLIP model and training code
│   ├── inference/                   # Report generation and GREEN evaluation
│   ├── report_gen_decoder/          # Encoder-decoder report-generation modules
│   └── train_report_generation.py
├── pyproject.toml
├── requirements.txt
└── README.md

Environment Preparation

Create a Python environment and install the package from the repository root:

cd /path/to/arr_radiology_code_submission
python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e .
python -m pip install -r requirements.txt

FG-CLIP training uses DeepSpeed, and report scoring uses GREEN. Install those optional packages in the same environment if you plan to run the corresponding stages:

python -m pip install deepspeed

For MedImageInsight-based training and feature extraction, point the code to a local MedImageInsight package and checkpoint directory:

export MEDIMAGEINSIGHT_PACKAGE_ROOT=/path/to/MedImageInsights
export MEDIMAGEINSIGHT_MODEL_DIR=/path/to/MedImageInsights/model
export MEDIMAGEINSIGHT_VISION_MODEL=vision_model.pth
export MEDIMAGEINSIGHT_LANGUAGE_MODEL=language_model.pth

Dataset Preparation

The FG-CLIP trainer expects a JSON manifest described in docs/dataset_schema.md. The expected high-level structure is:

{
  "caption": "study-level report text",
  "f_path": "relative/path/to/image.jpg",
  "observations": [
    {
      "obs_text": "Left basilar opacity is present.",
      "hard_negatives": ["Right basilar opacity is present."],
      "regions": [
        {"region_name": "Left Lung", "bbox": [0, 0, 128, 128]}
      ]
    }
  ]
}

A convenient local data layout is:

data/
├── raw/images/
├── intermediate/
│   ├── base_records.json
│   ├── all_disease_sentences.json
│   └── observation_sentences.json
├── processed/
│   └── region_coordinates.json
├── classification/
│   ├── train.json
│   └── test.json
└── report_generation/
    ├── pretrain_train.json
    ├── pretrain_val.json
    ├── train.json
    ├── validation.json
    └── test.json

The data-construction pipeline is:

1. Start from data/intermediate/base_records.json, a JSON list of image/report records.
2. Split or normalize report findings into observation-level sentences with src/dataset/split_sentences.py.
3. Map each observation sentence to relevant anatomy using src/dataset/select_regions.py.
4. Predict anatomy boxes with src/dataset/segment_regions.py, producing data/processed/region_coordinates.json.
5. Generate observation-level hard negatives with src/dataset/generate_hard_negatives.py.
6. Merge records, anatomy selections, boxes, and hard negatives with src/dataset/build_training_manifest.py.

split_sentences.py, select_regions.py, and generate_hard_negatives.py use google/medgemma-27b-text-it by default. If you use another medical LLM, keep the output JSON/JSONL schema compatible with the merge step.

Run anatomy segmentation separately:

python src/dataset/segment_regions.py \
  --image_root data/raw/images \
  --input_json data/intermediate/base_records.json \
  --output_json data/processed/region_coordinates.json

Then run the wrapper for the remaining preprocessing and manifest merge:

bash scripts/prepare_dataset.sh

The resulting FG-CLIP training manifest is written to data/processed/train_fgclip_manifest.json.

Train Radiology FG-CLIP

The FG-CLIP implementation lives in src/fgclip_sim/. The radiology training recipe extends CLIP-style image-text contrastive learning with:

• region-box supervision;
• observation-level hard negatives;
• pathology contrastive loss;
• MedImageInsight image and text encoders loaded into the FG-CLIP wrapper.

Train with:

bash scripts/train_fgclip.sh

The wrapper calls src/fgclip_sim/train/train_mem.py through DeepSpeed and writes checkpoints under outputs/fgclip_ours/. Edit the script or call the Python entry point directly if you need different batch sizes, epochs, data paths, or checkpoint locations.

Retrieval Evaluation

Retrieval evaluation is feature-based. It assumes one aligned image embedding and one aligned text embedding per example.

Place embeddings at:

outputs/retrieval/<backbone>_image_embeddings.pt
outputs/retrieval/<backbone>_text_embeddings.pt

Then run:

BACKBONE=ours bash scripts/eval_retrieval.sh
BACKBONE=pubmedclip bash scripts/eval_retrieval.sh

The evaluator is src/evaluation/retrieval_eval.py and reports image-to-text and text-to-image recall at configurable k values.

Classification Evaluation

src/evaluation/classification_eval.py supports linear probing and prompt-based zero-shot classification with FG-CLIP checkpoints. The default wrapper uses:

• data/classification/train.json
• data/classification/test.json
• data/raw/images
• outputs/fgclip_ours/checkpoint-last
• configs/classification_prompts_chexpert.json

Run:

bash scripts/eval_classification.sh

For zero-shot-only evaluation, call the Python entry point directly with --zero_shot --prompt_json configs/classification_prompts_chexpert.json.

Report Generation

The report-generation pipeline trains a lightweight decoder over precomputed image patch features. The relevant modules are:

• feature extraction: src/extract/
• decoder training: src/train_report_generation.py
• inference: src/inference/run_inference.py
• GREEN scoring: src/inference/evaluate_green.py

Supported feature-extraction backbones include medclip, pubmedclip, radclip, raddino, cxrclip, gloria, medimageinsight, and ours.

Extract features:

BACKBONE=medclip bash scripts/extract_report_features.sh
BACKBONE=ours bash scripts/extract_report_features.sh

Train the report decoder:

BACKBONE=ours ENCODER_TYPE=identity bash scripts/train_report_decoder.sh

Run inference:

MODEL_NAME=ours bash scripts/run_report_inference.sh

Evaluate generated reports with GREEN:

python src/inference/evaluate_green.py \
  --input-json outputs/inference/ours_test.json \
  --output-dir outputs/green_eval

For the ours backbone, the expected setup is to extract MedImageInsight patch features with the trained FG-CLIP visual weights loaded.

External Resources

The following resources must be supplied locally to run the full pipeline:

• radiology image-report datasets;
• MedImageInsight package and model weights;
• FG-CLIP checkpoints produced by this training code;
• baseline checkpoints for MedCLIP, PubMedCLIP, RadCLIP, Rad-DINO, CXR-CLIP, and GLoRIA as needed;
• an instruction-tuned medical LLM for dataset construction;
• a GREEN-compatible evaluation package and model weights for report scoring.

Suggested local paths are:

external/MedImageInsights/
external/cxr-clip/
external/gloria/
checkpoints/
outputs/

Notes for Reviewers

• The shell scripts are reproducible entry points with default paths rooted in this repository.
• For non-default layouts, either edit the script variables or call the Python entry points directly.
• The artifact intentionally excludes raw data, private checkpoints, logs, notebooks, and machine-specific launch files.
• Generated outputs are expected under outputs/; intermediate prepared data is expected under data/.