RadAlign

Official implementation of RadAlign: Concept-Based Alignment for Interpretable Radiology Report Generation.

Paper: https://arxiv.org/abs/2501.07525

Overview

RadAlign is a multimodal framework for chest X-ray analysis and radiology report generation. It introduces learnable concept tokens that align visual features with radiological concepts via cross-attention, enabling interpretable disease classification and retrieval-based report generation.

Key contributions:

• Concept-based vision-language alignment with 14 radiological concepts across 6 disease categories
• Contrastive learning between visual concept tokens and textual concept descriptions
• Retrieval-augmented report generation using learned concept tokens and LLM refinement

Installation

Clone the repository and set up the environment using conda:

git clone https://github.com/difeigu/RadAlign.git
cd RadAlign
conda env create -f environment.yml
conda activate multimodal
pip install -e .

Or install dependencies directly:

pip install -r requirements_run.txt

Datasets

RadAlign is evaluated on two chest X-ray datasets:

• MIMIC-CXR: Requires access via PhysioNet
• OpenI (Indiana University): Available at OpenI

Update the dataset paths in downstream_config_mimicbest.yaml or downstream_config_openi.yaml before running.

Training

cd src
CUDA_VISIBLE_DEVICES=0,1,2,3 python trainer_concept.py \
    --dataset universal \
    --unique_name radalign_run \
    --batch_size 16 \
    --gpu 0,1,2,3

Inference and Report Generation

After training, run the following steps in sequence:

1. Compute classification thresholds:

python notebooks/get_cls_threshold.py

2. Update threshold in the config YAML.

3. Generate reports:

python src/caption_gen_querying_token.py

Set OPENAI_API_KEY in your environment for LLM-based report refinement:

export OPENAI_API_KEY=your_key_here

Acknowledgements

This work builds on BioViL-T and CXR-BERT from Microsoft for the vision and language encoders.

Citation

If you use this code in your research, please cite:

@inproceedings{gu2025radalign,
  title={Radalign: Advancing radiology report generation with vision-language concept alignment},
  author={Gu, Difei and Gao, Yunhe and Zhou, Yang and Zhou, Mu and Metaxas, Dimitris},
  booktitle={International Conference on Medical Image Computing and Computer-Assisted Intervention},
  pages={484--494},
  year={2025},
  organization={Springer}
}