ACT volume-report model (clip_3d_ctrate_merlin_v1)

Weights for ACT (Auditable CT phenotyping through report-derived radiological observations), the native-resolution 3D vision-language model described in the manuscript under submission. Code: github.com/peterhan91/ACT.

Architecture

• Slice encoder: DINOv2 ViT-B/14 with register tokens (fetched at load time via torch.hub), applied to 160 axial slices at 224 x 224.
• Slice fusion: two-layer pre-norm Transformer (12 heads, feed-forward multiplier 2, rotary embeddings), class token appended to the end of the slice sequence.
• Projection: linear to a 768-d volume embedding.
• Text tower: CLIP-style causal Transformer, width 512, 12 layers, 8 heads, 77-token context, byte-pair encoding.

Training data

Contrastive volume-report pretraining on CT-RATE (chest CT) and Merlin (abdominal CT): 38,317 patients. The CT-RATE validation split was used only for checkpoint selection. No evaluation scan (CT-RATE test, PMBB, RSNA-2023, INSPECT) was seen in pretraining.

Usage

from load_pretrained import load_act  # model/load_pretrained.py in the ACT repo

model, preprocess_text = load_act()

The loader downloads clip_3d_ctrate_merlin_v1/best_model.pt from this repository and instantiates the architecture above. Volumes must be preprocessed with the ACT pipeline (RAS reorientation, HU clip [-1000, 1000], aspect-preserving resize and pad to 160 x 224 x 224, uint8).

Intended use and limitations

Released for academic research. Not a medical device; not for clinical decision-making. Performance is characterized in the manuscript on CT-RATE, PMBB, RSNA-2023, and INSPECT. Behaviour outside chest and abdominal CT of adults is uncharacterized. The source datasets' licenses and data-use terms apply to any data you use with the model.

Files

• clip_3d_ctrate_merlin_v1/best_model.pt: the released checkpoint, selected by mean AUROC over the 18 CT-RATE validation labels (0.756 at selection).
• Redistributable observation-bank artefacts will be added alongside the paper's publication.

Citation

See the ACT repository CITATION.cff. A journal reference will be added upon publication.