ALPaCA

Training, evaluation and demonstration code for ALPaCA (Adapting Llama for Pathology Context Analysis), a slide-level vision-language model for whole-slide-image question answering.

Scope and licensing

The author-developed ALPaCA code in this repository is released under the MIT License. The bundled third-party Prov-GigaPath and TorchScale source components remain subject to their respective licences; see Third-party notices.

The fine-tuned checkpoints, Llama-slideQA.bin, Llama-slideQA-BRCA.bin and Llama-slideQA-Morphology.bin, are available separately at CNX-PathLLM/Llama-slideQA. They are subject to the Llama 3.1 Community License, including its applicable use and redistribution conditions. Users must separately obtain access to the meta-llama/Llama-3.1-8B-Instruct base model under Meta's applicable terms.

ALPaCA is for research use only and is not a clinical diagnostic tool.

Installation

Create a Python 3.10 environment and install PyTorch appropriate to the local CUDA configuration. Then install the remaining dependencies:

pip install -r requirements.txt

The required Prov-GigaPath and TorchScale source components are included in this repository. Please follow their respective licence terms.

Obtain access to meta-llama/Llama-3.1-8B-Instruct before running training or evaluation. Prepare three-scale CONCH features for each WSI (0_1024, 1_512 and 1_1024). The included GMM prototype files are split into GTEx_TCGA_merged_output_less/ (32/16/8 clusters) and GTEx_TCGA_merged_output_more/ (256/128/64 clusters); use the directory that matches the selected --version setting.

Data and model assets

The following resources are hosted separately from this code repository:

The raw TCGA material is available through the NCI Genomic Data Commons, and GTEx source material is available through the GTEx Portal. Access to the original source data is subject to the relevant data-provider terms and any applicable institutional restrictions. This repository does not redistribute raw whole-slide images, pathology reports or notes, curated caption/QA datasets, or model checkpoints.

For the Hybrid adaptor, provide the local GMM prototype directory with --fea_root or --proto-root as appropriate. Patch features must be prepared separately with the CONCH encoder and can be extracted with the preprocessing utilities in SMMILe.

Aggregation strategies

--agg_strategy selects how patch features are aggregated into slide-level tokens:

Strategies are composable, for example abmil,gmm or random,gmm. For a composite strategy, supply a comma-separated --fea_root list in the same order as --agg_strategy. The random branch still requires a valid feature path, although it discards the features.

Key configuration flags

Training

Each stage has a ready-to-run shell script with encoder-specific blocks. Update feature paths, output paths, dataset names and hardware settings before execution.

Stage 1: slide-level description pretraining

accelerate launch --config_file=./accelerate_configs/deepspeed_zero2.yaml run_wsi.py \
    --max_steps 20000 --warmup_steps 1000 \
    --gpu 2 --train_batch_size 4 --eval_batch_size 2 --max_seq_length 512 \
    --agg_strategy longnet,gmm --embed_dim 512 --att_loss_weight 1 \
    --n_heads 32,16,8 --hierachical_token True --hierachical_adaptor True --gmm_need_query True \
    --llm_name meta-llama/Llama-3.1-8B-Instruct --venc_name Conch \
    --dataset_name_list CNX-PathLLM/TCGA-WSI-Description-4o,CNX-PathLLM/TCGA-WSI-Description-4omini,CNX-PathLLM/GTEx-WSI-Description \
    --data_cache_dir ~/.cache \
    --fea_root /path/to/Conch,/path/to/GTEx-TCGA-GMM_Conch/more \
    --output_dir /path/to/output/stage1 \
    --llm_requires_grad False --resume_from_checkpoint False

See run_wsi_stage1.sh for the full configuration.

Stage 2: general slide-QA instruction fine-tuning

accelerate launch --config_file=./accelerate_configs/deepspeed_zero2.yaml run_wsi.py \
    --max_steps 20000 --warmup_steps 100 --save_steps 300 \
    --gpu 2 --train_batch_size 8 --eval_batch_size 2 --max_seq_length 256 \
    --agg_strategy longnet,gmm --embed_dim 512 --att_loss_weight 1 \
    --n_heads 32,16,8 --hierachical_token True --hierachical_adaptor True --gmm_need_query True \
    --llm_name meta-llama/Llama-3.1-8B-Instruct --venc_name Conch \
    --dataset_name_list CNX-PathLLM/TCGA-WSI-CloseQA-Balanced,CNX-PathLLM/GTEx-WSI-CloseQA-Balanced,CNX-PathLLM/TCGA-WSI-OpenQA,CNX-PathLLM/GTEx-WSI-OpenQA \
    --data_cache_dir ~/.cache \
    --fea_root /path/to/Conch,/path/to/GTEx-TCGA-GMM_Conch/more \
    --output_dir /path/to/output/stage2 \
    --ckpt_path /path/to/output/stage1/ckpt20000.bin \
    --llm_requires_grad True --resume_from_checkpoint False

See run_wsi_stage2.sh for the full configuration.

Stage 3: disease-specific fine-tuning

Continue from the Stage 2 checkpoint with either the Breast dataset (CNX-PathLLM/TCGA-BRCA-Details-CloseQA,CNX-PathLLM/TCGA-BRCA-Details-OpenQA) or the Morphology dataset (CNX-PathLLM/PathChat_CloseQA_Balanced,CNX-PathLLM/PathChat_OpenQA). Use the Stage 2 command with the relevant --dataset_name_list, --ckpt_path and --output_dir; see run_wsi_stage3.sh.

The principal training entry point is run_wsi.py; the model implementation is in model/my_model.py.

Evaluation

General QA

python test_wsi.py \
    --max_seq_length 128 --batch_size 16 --select_data_num -1 --eval_sample_size -1 \
    --n_heads 32,16,8 --hierachical_token True --hierachical_adaptor True --gmm_need_query True \
    --llm_name meta-llama/Llama-3.1-8B-Instruct --venc_name Conch \
    --shuffle False --data_cache_dir ~/.cache \
    --dataset_name_list CNX-PathLLM/TCGA-WSI-CloseQA-Balanced,CNX-PathLLM/GTEx-WSI-CloseQA-Balanced,CNX-PathLLM/TCGA-WSI-OpenQA,CNX-PathLLM/GTEx-WSI-OpenQA \
    --agg_strategy longnet,gmm --embed_dim 512 \
    --fea_root /path/to/Conch,/path/to/GTEx-TCGA-GMM_Conch/more \
    --ckpt_path /path/to/Llama-slideQA.bin \
    --results_save_path /path/to/output.csv

See test_wsi_stage2.sh. For disease-specific evaluation, use the relevant Stage 3 dataset and checkpoint; see test_wsi_stage3.sh.

Single-WSI demo

Use CLAM_PreProcessing to extract CONCH features at the three scales used by ALPaCA:

<conch-dir>/
    <slide_id>_0_1024.npy  # level 0 (40x), 1024 px tiles
    <slide_id>_1_512.npy   # level 1 (20x), 512 px tiles
    <slide_id>_1_1024.npy  # level 1 (20x), 1024 px tiles

For a quick smoke test, pre-extracted features for 20 TCGA and 20 GTEx slides are available at CNX-PathLLM/Llama-slideQA-Sample-Features. Slide identifiers, questions, answers and demo modes are listed in demo/slides_sample.csv.

python demo/single_wsi_demo.py \
    --conch-dir /path/to/conch_features \
    --slide-id TCGA-XX-XXXX-01Z-00-DX1.<uuid> \
    --out-dir ./demo_out \
    --proto-root GMM_feature_extraction/GTEx-TCGA-prototypes/GTEx_TCGA_merged_output_more \
    --vlm-ckpt /path/to/Llama-slideQA.bin \
    --llm-name meta-llama/Llama-3.1-8B-Instruct \
    --question "Describe the morphology of this slide."

The script first tokenises the three feature scales against the selected GMM prototypes, then loads the model and produces a response. Use --skip-stage1 to reuse an existing GMM feature file. --mode selects the prompt format: open for free-text QA, close-mc for multiple choice and close-tf for True/False QA. The GMM feature-generation helper is GMM_feature_extraction/src/wsi_single_demo.py.

Acknowledgements

We thank the developers of CLAM and PANTHER for their open-source contributions to computational pathology.

Citation

If you use this code or the accompanying model checkpoint, please cite:

Gao, Z. et al. ALPaCA: Adapting Llama for Pathology Context Analysis to enable slide-level question answering. medRxiv 2025.04.22.25326190 (2025). https://doi.org/10.1101/2025.04.22.25326190

Preprint: https://www.medrxiv.org/content/10.1101/2025.04.22.25326190v1