E2E-ViT: Turning Pre-Trained Vision Transformers into End-to-End Histopathology Whole Slide Image Models for Survival Prediction

CVPR 2026 | Paper Link

Install

pip install torch torchvision timm scikit-survival pillow tqdm tensorboard

Patch Extraction

E2E-VIT takes pre-cropped patches as input. You can use any WSI preprocessing tool to extract 224×224 patches from whole-slide images, such as CLAM, TRIDENT and PIANO.

After extraction, organise the output as:

<patch_root>/
└── <dataset_name>/
    └── <slide_id>/
        ├── <prefix>_<x>x_<y>y.jpg
        ├── <prefix>_<x>x_<y>y.jpg
        └── ...

Example:

os_patches/
└── os_cptac_ccrcc/
    ├── C3L-00026-21/
    │   ├── no000000_000004096x_000013312y.jpg
    │   ├── no000001_000005120x_000011264y.jpg
    │   └── ...
    └── C3L-00418-21/
        └── ...

Filename convention: the last two _-separated fields before .jpg must be <x_coord>x and <y_coord>y (e.g. no000052_008192x_015360y.jpg).

Quick Start

bash run.sh

Or a single fold (use vit_small as an example):

python train.py \
    --data_json data_jsons/os_cptac_ccrcc_patchseq_dataset.json \
    --data_root /path/to/your/patch_root \
    --pfm_name vit_small \
    --fold_idx 0 \
    --merge_type mean \
    --merge_ratio 1 \
    --enable_alibi yes \
    --gpu_id 0

---

For data format, see data_jsons/example_dataset.json.

{
  "fold_0": {
    "train": [{"patchseq_dir": "os_patches/os_cptac_ccrcc/SLIDE-001", "events": 1, "days": 480, "label": 0}],
    // events: 1=event observed, 0=censored; days: follow-up time; label: discretised survival interval, for more details, see: https://huggingface.co/datasets/MahmoodLab/Patho-Bench
    "valid": [...]
  }
}

---

For CONCH model, extract the vision tower weights first:

python extract_conch_visual.py --output conch_visual.pth

Then pass conch_weights_path="conch_visual.pth" when calling E2E_VIT.from_pretrained("conch_v1", ...).

---

If your find our work useful, please consider citing:

@inproceedings{li2026turning,
  title={Turning Pre-Trained Vision Transformers into End-to-End Histopathology Whole Slide Image Models for Survival Prediction},
  author={Li, Jiawen and Hu, Jiali and Ling, Xitong and Yan, Renao and Chen, Yuxuan and Guan, Tian and He, Yonghong},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={21046--21056},
  year={2026}
}