Libra: Leveraging Temporal Images for Biomedical Radiology Analysis

This repository hosts Libra, a tool designed to generate radiology reports by leveraging temporal information from chest X-rays taken at different time points.

🔥 News

• [4 Oct 2025] 🛠️ Updated the batch inference function libra_eval_batch— see Batch Inference Guide.
• [27 Sep 2025] 📷 CCD: Clinical Contrastive Decoding is out! A plug-and-play, training-free decoding framework to boost any SOTA radiology MLLM!
• [20 Jul 2025] ✅ Support for LLaVA-Rad is added. Compatible weights are now available.
• [15 Jul 2025] ✅ Support for MAIRA-2 is added. Compatible weights are provided for benchmarking, with 'use_maira_feature_norm: true' set to ensure compatibility with the default feature extraction strategy.
• [14 Jul 2025] 🩺 For evaluating AI-generated radiology text, we recommend using 👉 RadEval.
• [9 Jul 2025] 🚚 The test dataset is now available on Hugging Face — check out ./MIMIC-CXR-RRG. It includes findings, impression, indication, comparison, technique, history, and examination sections, processed according to the official MIMIC-CXR guidelines.
• [8 Jul 2025] 💻 Released data preparation scripts for Prior Image Retrieve.
• [18 Jun 2025] 🎤 Invited talk at HealTAC 2025 — topic: Towards Temporal-Aware Multimodal Large Language Models for Improved Radiology Report Generation
• [16 May 2025] 📝 A short blog: some musings on "What Does ‘Temporal’ Really Mean?” — thoughts behind Libra and temporal reasoning in radiology.
• [15 May 2025] 🥳 The paper has been accepted to ACL 2025!
• [09 May 2025] ✨ Now with full support for the Phi-4 family — compact language and reasoning models from Microsoft.
• [24 Mar 2025] 🏆 Libra was invited to the ReXrank Challenge — a leading leaderboard for Chest X-ray Report Generation.

Overview

Radiology report generation requires integrating temporal medical images and creating accurate reports. Traditional methods often overlook crucial temporal information. We introduce Libra, a temporal-aware MLLM for chest X-ray report generation. Libra combines a radiology-specific image encoder with a novel Temporal Alignment Connector (TAC), designed to accurately capture and integrate temporal differences between paired current and prior images. Experiments show that Libra sets new performance benchmarks on the MIMIC-CXR dataset for the RRG task.

Contents

• Install
• Model Weights
  • Libra-v1.0
  • Libra-v0.5
  • Compatible weights
  • Projector weights
• Quick Start
  • Gradio Web UI
  • CLI Inference
  • Script Inference
  • ✨Batch Inference
• Dataset
  • Prepare Data
  • Preprocess Data
  • Data Download
• Train
  • Hyperparameters
  • Stage 1: visual feature alignment
  • Stage 2: RRG downstream task fine-tuning
  • ✨New Options to Note
• Evaluation
  • Generate model responses
  • Evaluate the generated report
  • Metrics

Install

[!TIP] We strongly recommend that you create an environment from scratch as follows:

Option 1:

Install the most up-to-date version directly from GitHub for quick use:

pip install git+https://github.com/X-iZhang/Libra.git

Option 2:

If you want to modify the code, you can clone the repository and install it in editable mode:

1. Clone this repository and navigate to Libra folder

git clone https://github.com/X-iZhang/Libra.git
cd Libra

2. Install Package

conda create -n libra python=3.10 -y
conda activate libra
pip install --upgrade pip  # enable PEP 660 support
pip install -e .

3. Install additional packages for Training and Evaluation cases

pip install -e ".[train,eval]"
pip install flash-attn --no-build-isolation

git pull
pip install -e .

Model Weights

Libra-v1.0 achieves SoTA performance.

Libra-v1.0

Libra-v0.5

[!NOTE] These two models are fine-tuned for Findings and Impression section generation. For more information on training strategies and dataset collection, please refer to Med-CXRGen (Gla-AI4BioMed at RRG24)

Compatible weights

[!NOTE]

• To use Llava-Med-v1.5, set conv_mode = llava_med_v1.5_mistral_7b
• To use LLaVA-Rad, set conv_mode = v1
• To use MAIRA-2, set conv_mode = maira_2

# ✅ With clinical instruction
prompt_with_clinical = (
    "Provide a description of the findings in the radiology study in comparison to the prior frontal image. "
    "INDICATION: Dyspnea. TECHNIQUE: PA and lateral views of the chest. COMPARISON: None."
)

# ✅ Without clinical instruction — all placeholders (INDICATION, TECHNIQUE, COMPARISON) and default prompt must still be present
prompt_minimal = (
    "Provide a description of the findings in the radiology study in comparison to the prior frontal image. "
    "INDICATION: None. TECHNIQUE: None. COMPARISON: None."
)

# 🧪 Example usage (following official MAIRA-2 format)
from libra.eval import libra_eval

frontal_image_url = "https://openi.nlm.nih.gov/imgs/512/145/145/CXR145_IM-0290-1001.png"
model_path = "X-iZhang/libra-maira-2"

answer = libra_eval(
    model_path=model_path,
    image_file=[frontal_image_url],
    query=prompt_with_clinical,
    conv_mode="maira_2",
    temperature=0.0,         # Use greedy decoding
    max_new_tokens=300,
)

# ✅ Expected output
print(answer)
# > There is a large right pleural effusion. No pneumothorax is identified. 
# > There is no left pleural effusion. There is no focal consolidation. 
# > The cardiomediastinal silhouette is within normal limits.

Projector weights

These projector weights were pre-trained for visual instruction tuning on chest X-ray to text generation tasks. They can be directly used to initialise your model for multimodal fine-tuning in similar clinical domains.

⚠️ Important Note: For compatibility, please ensure that the projector type, base LLM, conv_mode, and vision encoder exactly match those used in our projector pretraining setup. Please also ensure the following settings are correctly configured during instruction tuning:

--mm_projector_type TAC \ # or mlp2x_gelu
--mm_vision_select_layer all \ # or -2
--mm_vision_select_feature patch \
--mm_use_im_start_end False \
--mm_use_im_patch_token False \

Quick Start

Gradio Web UI

Launch a local or online web demo by running:

python -m libra.serve.app

python -m libra.serve.app --model-path /path/to/your/model

You just launched the Gradio web interface. Now, you can open the web interface with the URL printed on the screen. You will notice that both the default libra-v1.0 model and your model are available in the model list, and you can choose to switch between them.

CLI Inference

We support running inference using the CLI. To use our model, run:

python -m libra.serve.cli \
    --model-path X-iZhang/libra-v1.0-7b \
    --image-file "./path/to/current_image.jpg" "./path/to/previous_image.jpg"
    # If there is no previous image, only one path is needed.

Script Inference

You can use the libra_eval function in libra/eval/run_libra.py to easily launch a model trained by yourself or us on local machine or in Google Colab, after installing this repository.

from libra.eval import libra_eval

# Define the model path, which can be a pre-trained model or your own fine-tuned model.
model_path = "X-iZhang/libra-v1.0-7b"  # Or your own model

# Define the paths to the images. The second image is optional for temporal comparisons.
image_files = [
    "./path/to/current/image.jpg", 
    "./path/to/previous/image.jpg"  # Optional: Only include if a reference image is available
]

# Define the prompt to guide the model's response. Add clinical instructions if needed.
prompt = (
    "Provide a detailed description of the findings in the radiology image. "
    "Following clinical context: ..."
)

# Specify the conversational mode, matching the PROMPT_VERSION used during training.
conv_mode = "libra_v1"

# Call the libra_eval function.
libra_eval(
    model_path=model_path,
    image_file=image_files,
    query=prompt,
    temperature=0.9,
    top_p=0.8,
    conv_mode=conv_mode,
    max_new_tokens=512
)

libra_eval(
    model_path=model_path,
    image_file=image_files,
    query=prompt,
    num_beams=5, 
    length_penalty=2,
    num_return_sequences=2,
    conv_mode=conv_mode,
    max_new_tokens=512
)

libra_eval(
    model_path="./path/to/lora_weights",  # path to LoRA weights
    model_base="./path/to/base_model",  # path to base Libra model
    image_file=image_files,
    query=prompt,
    num_beams=5, 
    length_penalty=2,
    num_return_sequences=2,
    conv_mode=conv_mode,
    max_new_tokens=512
)

✨ Batch Inference

[!IMPORTANT] Now, we provide a simple batch inference function libra_eval_batch to facilitate the evaluation of multiple samples. This function is particularly useful for assessing model performance on datasets like MIMIC-CXR. Below is an example of how to use it:

# --- Import necessary libraries ---
from libra.eval.run_libra import load_model
from libra.eval import libra_eval_batch
from datasets import load_dataset

# --- Set up the model ---
model_path = "X-iZhang/libra-v1.0-7b"
# model_path = "X-iZhang/libra-v1.0-3b"
# model_path = "X-iZhang/Med-CXRGen-F"
# model_path = "X-iZhang/Med-CXRGen-I"
# model_path = "X-iZhang/libra-llava-med-v1.5-mistral-7b"
# model_path = "X-iZhang/libra-maira-2"
# model_path = "X-iZhang/libra-llava-rad"
reuse_model = load_model(model_path)
print('Load model success')

# --- load dataset and prepare images and queries ---
# Load subset and take first 4 samples
ds = load_dataset("X-iZhang/MIMIC-CXR-RRG", name="findings_section", split="test[:4]")

# Extract current images and prior images
images = [
    [ex["main_image"].convert("RGB"), ex["prior_image"].convert("RGB")]
    for ex in ds
]

# Or just current images if no prior images are available
# images = [ex["main_image"].convert("RGB") for ex in ds]

# Extract queries
queries = [ex["default_prompt"] for ex in ds]

# --- Set generation parameters ---
libra_eval_batch(
            libra_model=reuse_model,
            images=images,         # Dummy previous image for libra models
            queries=queries,
            max_new_tokens=128,
            temperature=0.0        # Greedy decoding
        )

['The patient is status post right upper lobe resection. The right hemidiaphragm is elevated. The lungs are clear. The heart and mediastinal structures are unremarkable. The bony thorax is grossly intact.',
 'The patient is status post right upper lobe resection. The right hemidiaphragm is elevated and there is persistent volume loss in the right hemithorax. The left lung is clear. There is no evidence of pneumothorax or pleural effusion.',
 'The patient is status post right upper lobe resection. The right hemidiaphragm is elevated and there is persistent volume loss in the right hemithorax. The left lung is clear. The heart is normal in size. Mediastinal structures are otherwise unremarkable. The bony thorax is grossly intact.',
 'The patient is status post right upper lobe resection. There is persistent volume loss in the right hemithorax with rightward shift of mediastinal structures. The right upper hilar mass is again demonstrated as well as a right apical pleural cap. The left lung is clear. The heart is normal in size. There are no pleural effusions.']

Dataset

Prepare Data

All the data we use comes from MIMIC-CXR and its two variants, and we strictly follow the official split for train/valid/test division.

• Image Data

All images used for Libra come from the MIMIC-CXR-JPG dataset in .jpg format. DICOM format is also supported and can be found in the MIMIC-CXR.

After downloading the images, they will be automatically organized into the following structure in ./path/to/playground/data:

./data/physionet.org/files/mimic-cxr-jpg/2.0.0
└──files
    ├── p10
    │   └── p10000032
    │       └── s50414267
    │           ├── image1.jpg
    │           └── image2.jpg
    ├── p11
    ├── p12
    ├── ...
    └── p19

• Annotation Data

All annotations used for Libra come from the MIMIC-CXR and its two variants. This includes Radiology Reports and other relevant Visual Question Answering.

Please download the following datasets from the official website: mimic-cxr-reports.zip from MIMIC-CXR, MIMIC-Diff-VQA, and MIMIC-Ext-MIMIC-CXR-VQA.

• Prior Image Retrieve

[!NOTE] You can obtain metadata labels from mimic-cxr-2.0.0-metadata.csv.gz

from tqdm import tqdm
# Sort the test set by subject_id, StudyDate, and StudyTime to ensure chronological order
train_df_pair = train_df_pair.sort_values(by=['subject_id', 'StudyDate', 'StudyTime'])
train_df_pair = train_df_pair.reset_index(drop=True)

# Initialize a new column to store the prior image ID for each record
train_df_pair['prior_image'] = None

# Get the list of unique subject IDs (i.e., different patients)
unique_subject_ids = train_df_pair['subject_id'].unique()

# Use tqdm to create a progress bar while processing subjects
for subject_id in tqdm(unique_subject_ids, desc='Processing subjects'):
    # Extract all records for the current subject
    group = train_df_pair[train_df_pair['subject_id'] == subject_id]
    prior_image = None  # Initialize the prior image as None
    last_date = None    # Keep track of the previous study date

    # Iterate through each image for the current subject
    for idx, row in group.iterrows():
        # If the current study date is later than the last recorded date, update the prior image
        if last_date is None or row['StudyDate'] > last_date:
            train_df_pair.at[idx, 'prior_image'] = prior_image
            last_date = row['StudyDate']
        else:  # Records from the same study date
            # Find the first record on the same date for this subject
            # All records from the same day share the same prior image
            same_day_first = group[(group['StudyDate'] == row['StudyDate'])].index[0]
            train_df_pair.at[idx, 'prior_image'] = train_df_pair.at[same_day_first, 'prior_image']

        # Update the prior_image to be the current image for the next iteration
        prior_image = row['image']

Preprocess Data

• Radiology Report Sections

For free-text radiology report, we extract the Findings, Impression, Indication, History, Comparison, and Technique sections using the official mimic-cxr repository.

[!NOTE] To enable more structured and accurate extraction of the Indication, History, Comparison, and Technique sections—beyond what the original scripts provide—we replace the official .py with our customised versions located in Libra/scripts/mimic-cxr/.

• Visual Question Answering for Chest X-ray

In Medical-Diff-VQA, the main image is used as the current image, and the reference image is used as the prior image. In MIMIC-Ext-MIMIC-CXR-VQA, all cases use a dummy prior image.

Data Download

If you want to train or evaluate your own tasks or datasets, please refer to Custom_Data.md.

Train

Libra adopt a two-stage training strategy: (1) visual feature alignment: the visual encoder and LLM weights are frozen, and the Temporal Alignment Connector is trained; (2) RRG downstream task fine-tuning: apply LoRA to fine-tune the pre-trained LLM on the Findings section generation task.

Libra is trained on 1 A6000 GPU with 48GB memory. To train on multiple GPUs, you can set the per_device_train_batch_size and the gradient_accumulation_steps accordingly. Always keep the global batch size the same: per_device_train_batch_size x gradient_accumulation_steps x num_gpus.

Hyperparameters

We set reasonable hyperparameters based on our device. The hyperparameters used in both pretraining and LoRA finetuning are provided below.

1. Pretraining

2. LoRA finetuning

Download Meditron checkpoints (automatically)

Our base LLM model, Meditron-7B, adapted to the medical domain from the Llama-2-7B model, will be downloaded automatically when you run our provided training scripts. No action is needed on your part.

Stage 1: visual feature alignment

Pretraining takes approximately 385 hours for Libra-v1.0-7b-pretrain on a single A6000 GPU (48GB) due to device limitations.

For detailed training scripts and guidelines, please refer to the following: pretrain.sh and pretrain_xformers.sh for memory-efficient attention implemented in xFormers.

• --mm_projector_type TAC: the Temporal Alignment Connector.
• --vision_tower microsoft/rad-dino: RAD-DINO is a vision transformer for encoding chest X-rays using DINOv2.
• --mm_vision_select_layer all: Use all image features from the encoder for the Layerwise Feature Extractor.
• --tune_mm_mlp_adapter True
• --freeze_mm_mlp_adapter False

Stage 2: RRG downstream task fine-tuning

You may download our pretrained projectors from the mm_tac_projector.bin file. It takes around 213 hours for Libra-v1.0-7b on a single A6000 GPU (48GB) due to device limitations.

For detailed training scripts and guidelines, please refer to: finetune_lora.sh.

• --tune_mm_mlp_adapter False
• --freeze_mm_mlp_adapter True

If you have enough GPU memory: Use finetune.sh to fine-tune the entire model. Alternatively, you can replace zero3.json with zero3_offload.json to offload some parameters to CPU RAM, though this will slow down the training speed.

If you are interested in continue finetuning Libra model to your own task/data, please check out Custom_Data.md.

✨ New Options to Note

[!NOTE]

• --mm_projector_type TAC: Specifies the Temporal Alignment Connector for Libra.
• --vision_tower microsoft/rad-dino: Uses RAD-DINO as the chest X-rays encoder.
• --mm_vision_select_layer all: Selects specific vision layers (e.g., -1, -2) or "all" for all layers.
• --validation_data_path ./path/: Path to the validation data.
• --compute_metrics True: Optionally computes metrics during validation. Note that this can consume significant memory. If GPU memory is insufficient, it is recommended to either disable this option or use a smaller validation dataset.

Evaluation

In Libra-v1.0, we evaluate models on the MIMIC-CXR test split for the findings section generation task. You can download the evaluation data here. To ensure reproducibility and output quality, we evaluate our model using the beam search strategy.

1. Generate Libra responses.

python -m libra.eval.eval_vqa_libra \
    --model-path X-iZhang/libra-v1.0-7b \
    --question-file libra_findings_section_eval.jsonl \
    --image-folder ./physionet.org/files/mimic-cxr-jpg/2.0.0 \
    --answers-file /path/to/answer-file.jsonl \
    --num_beams 10 \
    --length_penalty 2 \
    --max_new_tokens 1024 \
    --conv-mode libra_v1

You can evaluate Libra on your custom datasets by converting your dataset to the JSONL format and evaluating using eval_vqa_libra.py.

Additionally, you can execute the evaluation using the command line. For detailed instructions, see libra_eval.sh.

bash ./scripts/eval/libra_eval.sh beam

2. Evaluate the generated report.

In our case, you can directly use libra_findings_section_eval.jsonl and answer-file.jsonl for basic evaluation, using radiology_report.py.

from libra.eval import evaluate_report

references = "libra_findings_section_eval.jsonl"
predictions = "answer-file.jsonl"

resul = evaluate_report(references=references, predictions=predictions)

# Evaluation scores
resul
{'BLEU1': 51.25,
 'BLEU2': 37.48,
 'BLEU3': 29.56,
 'BLEU4': 24.54,
 'METEOR': 48.90,
 'ROUGE-L': 36.66,
 'Bert_score': 62.50,
 'Temporal_entity_score': 35.34}

Or use the command line to evaluate multiple references and store the results in a .csv file. For detailed instructions, see get_eval_scores.sh.

bash ./scripts/eval/get_eval_scores.sh

Metrics

• Temporal Entity F1

The $F1_{temp}$ score includes common radiology-related keywords associated with temporal changes. You can use temporal_f1.py as follows:

from libra.eval import temporal_f1_score

predictions = [
    "The pleural effusion has progressively worsened since previous scan.",
    "The pleural effusion is noted again on the current scan."
]
references = [
    "Compare with prior scan, pleural effusion has worsened.",
    "Pleural effusion has worsened."
]

tem_f1_score = temporal_f1_score(
    predictions=predictions,
    references=references
)

# Temporal Entity F1 score
tem_f1_score
{'f1': 0.500000000075,
 'prediction_entities': [{'worsened'}, set()],
 'reference_entities': [{'worsened'}, {'worsened'}]}

• Radiology-specific Metrics

Some specific metrics may require configurations that could conflict with Libra. It is recommended to follow the official guidelines and use separate environments for evaluation: RG_ER, CheXpert-F1, RadGraph-F1, RadCliQ, CheXbert vector.

[!NOTE] For evaluation, we recommend using RadEval — a unified framework for radiology text evaluation that integrates all the above metrics.

Acknowledgements 🙏

We sincerely thank the following projects for their contributions to Libra:

• LLaVA: A Large Language and Vision Assistant, laying the groundwork for multimodal understanding.
• FastChat: An Open Platform for Training, Serving, and Evaluating Large Language Model based Chatbots.
• LLaMA: Open and efficient foundation language models that inspired our core language processing capabilities.
• MEDITRON: Open and efficient medical Large language models.
• RAD-DINO: An open and efficient biomedical image encoder, enabling robust radiological analysis.

Citation ✒️

If you find our paper and code useful in your research and applications, please cite using this BibTeX:

@inproceedings{zhang2025libra,
  title={Libra: Leveraging temporal images for biomedical radiology analysis},
  author={Zhang, Xi and Meng, Zaiqiao and Lever, Jake and Ho, Edmond SL},
  booktitle={Findings of the Association for Computational Linguistics: ACL 2025},
  pages={17275--17303},
  year={2025}
}

or

@inproceedings{zhang-etal-2025-libra,
    title = "Libra: Leveraging Temporal Images for Biomedical Radiology Analysis",
    author = "Zhang, Xi  and
      Meng, Zaiqiao  and
      Lever, Jake  and
      Ho, Edmond S. L.",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.findings-acl.888/",
    pages = "17275--17303",
    ISBN = "979-8-89176-256-5",
    abstract = "Radiology report generation (RRG) requires advanced medical image analysis, effective temporal reasoning, and accurate text generation. While multimodal large language models (MLLMs) align with pre-trained vision encoders to enhance visual-language understanding, most existing methods rely on single-image analysis or rule-based heuristics to process multiple images, failing to fully leverage temporal information in multi-modal medical datasets. In this paper, we introduce **Libra**, a temporal-aware MLLM tailored for chest X-ray report generation. Libra combines a radiology-specific image encoder with a novel Temporal Alignment Connector (**TAC**), designed to accurately capture and integrate temporal differences between paired current and prior images. Extensive experiments on the MIMIC-CXR dataset demonstrate that Libra establishes a new state-of-the-art benchmark among similarly scaled MLLMs, setting new standards in both clinical relevance and lexical accuracy. All source code and data are publicly available at: https://github.com/X-iZhang/Libra."
}

Intended Use 🧰

Libra is primarily designed to assist clinical practitioners, researchers, and medical students in generating chest X-ray reports. Key applications include:

• Clinical Decision Support: Providing draft findings that can be refined by a radiologist.
• Educational Tool: Demonstrating example interpretations and temporal changes for training radiology residents.
• Research: Facilitating studies on automated report generation and temporal feature learning in medical imaging.

Important: Outputs should be reviewed by qualified radiologists or medical professionals before final clinical decisions are made.