LAnA

Layer-Wise Anatomical Attention model

Best current model in this collection: manu02/LAnA-Arxiv

Overview

LAnA is a medical report-generation project for chest X-ray images. The completed project is intended to generate radiology reports with a vision-language model guided by layer-wise anatomical attention built from predicted anatomical masks.

The architecture combines a DINOv3 vision encoder, lung and heart segmentation heads, and a GPT-2 decoder modified so each transformer layer receives a different anatomical attention bias derived from the segmentation mask.

Intended Use

• Input: a chest X-ray image resized to 512x512 and normalized with ImageNet mean/std.
• Output: a generated radiology report.
• Best fit: research use, report-generation experiments, and anatomical-attention ablations.

How to Run

New users should prefer the standard Hugging Face flow below. The legacy snapshot/manual implementation lives on the snapshot-legacy branch for backward compatibility.

Implementation 1: Standard Hugging Face loading

import torch
from PIL import Image
from transformers import AutoModel, AutoProcessor

repo_id = "manu02/LAnA"
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

processor = AutoProcessor.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModel.from_pretrained(repo_id, trust_remote_code=True)
model.move_non_quantized_modules(device)
model.eval()

image = Image.open("example.png").convert("RGB")
inputs = processor(images=image, return_tensors="pt")
inputs = {name: tensor.to(device) for name, tensor in inputs.items()}

with torch.inference_mode():
    generated = model.generate(**inputs, max_new_tokens=150)

report = processor.batch_decode(generated, skip_special_tokens=True)[0]
print(report)

Batched inference uses the same path:

batch = processor(images=[image_a, image_b], return_tensors="pt")
batch = {name: tensor.to(device) for name, tensor in batch.items()}
generated = model.generate(**batch, max_new_tokens=150)
reports = processor.batch_decode(generated, skip_special_tokens=True)

HF_TOKEN is optional for this public standard-loading path. If you do not set one, the model still loads, but Hugging Face may show lower-rate-limit warnings.

Legacy snapshot branch

Use the snapshot/manual branch only if you specifically need the older import-based workflow:

• Branch: snapshot-legacy
• Download example: snapshot_download("manu02/LAnA", revision="snapshot-legacy")

Licensing and Redistribution Notice

This checkpoint bundles or derives from Meta DINOv3 model materials. Redistribution of those components must follow the DINOv3 license terms included in this repository. The project code remains available under the repository's own license, but the full packaged checkpoint should not be treated as MIT-only.

Research and Safety Disclaimer

This model is intended for research and educational use only. It is not a medical device, has not been validated for clinical deployment, and should not be used as a substitute for professional radiology review.

MIMIC Test Results

These comparison tables are refreshed across the full LAnA collection whenever any collection model is evaluated.

Cross-Model Comparison: All Frontal Test Studies (3041 studies)

Cross-Model Comparison: Findings-Only Frontal Test Studies (2210 studies)

Data

• Full project datasets: CheXpert and MIMIC-CXR.
• Intended project scope: train on curated chest X-ray/report data from both datasets and evaluate on MIMIC-CXR test studies.
• Current released checkpoint datasets: MIMIC-CXR (findings-only) for training and MIMIC-CXR (findings-only) for validation.
• Current published evaluation: MIMIC-CXR test split, frontal-only (PA/AP) studies.

Evaluation

• Medical report metrics implemented in the repository include RadGraph F1 and CheXpert F1 (14-micro, 5-micro, 14-macro, 5-macro).

Experiment Model Descriptions

• LAnA-MIMIC-CHEXPERT: This variant was trained on a combined dataset of CheXpert and MIMIC-CXR using LoRA fine-tuning with the AdamW optimizer.
• LAnA-MIMIC: This model was trained on the MIMIC-CXR (findings-only) dataset using LoRA fine-tuning with the AdamW optimizer.
• LAnA: This model was trained on the MIMIC-CXR (findings-only) dataset using full-model optimization with AdamW instead of LoRA.
• LAnA-v2: This version keeps the same training setup as LAnA, but increases the effective global batch size from 16 to 128.
• LAnA-v3: This version keeps the same training setup as LAnA, including the effective global batch size of 16, but changes how EOS is handled so training and generation follow the same behavior. The model no longer uses the EOS token during training, and generation remained greedy without stopping when an EOS token was produced. In the previous setup, decoding was also greedy, stopped at EOS, and used a maximum of 128 new tokens.
• LAnA-v4: This version keeps the same decoding behavior as LAnA-v3, but increases the effective global batch size from 16 to 128.
• LAnA-v5: This version uses the training recipe from the original LAnA paper, while switching to the legacy CXR-Findings-AI generation behavior.
• LAnA-Arxiv: This model is the report-generation model created in the arXiv paper, packaged locally with its original legacy generation code.

Training Snapshot

• Run: LAnA
• This section describes the current public checkpoint, not the final completed project.
• Method: full_adamw
• Vision encoder: facebook/dinov3-vits16-pretrain-lvd1689m
• Text decoder: gpt2
• Visual projection: mlp4
• Segmentation encoder: facebook/dinov3-convnext-small-pretrain-lvd1689m
• Image size: 512
• Local batch size: 1
• Effective global batch size: 16
• Scheduler: cosine
• Warmup steps: 1318
• Weight decay: 0.01
• Steps completed: 3127
• Planned total steps: 26358
• Images seen: 50046
• Total training time: 1.0000 hours
• Hardware: NVIDIA GeForce RTX 5070
• Final train loss: 2.9207
• Validation loss: 2.6414

Status

• Project status: Training in progress
• Release status: Research preview checkpoint
• Current checkpoint status: Not final
• Training completion toward planned run: 11.87% (0 / 3 epochs)
• Current published metrics are intermediate and will change as training continues.

Notes

• Set HF_TOKEN with permission to access the DINOv3 repositories required by this model before downloading or running inference.
• segmenters/ contains the lung and heart segmentation checkpoints used to build anatomical attention masks.
• evaluations/mimic_test_metrics.json contains the latest saved MIMIC test metrics.