👁️ AEGIS

Documentation | License

AEGIS is a toolkit for large-scale whole-slide image processing. This project was developed by Siddhesh Thakur.

[!NOTE] Contributions are welcome! Please report any issues. You may also contribute by opening a pull request.

Key Features:

• Tissue Segmentation: Extract tissue from background (H&E, IHC, etc.).
• Patch Extraction: Extract tissue patches of any size and magnification.
• Patch Feature Extraction: Extract patch embeddings from 20+ foundation models, including UNI, Virchow, H-Optimus-0 and more...
• Slide Feature Extraction: Extract slide embeddings from 5+ slide foundation models, including Threads (coming soon!), Titan, and GigaPath.

🔨 1. Installation:

Local Installation

• Create an environment: conda create -n "aegis" python=3.10, and activate it conda activate aegis.
• Cloning: git clone https://github.com/siddhesh-thakur/aegis.git && cd aegis.
• Local installation: pip install -e ..

Additional packages may be required to load some pretrained models. Follow error messages for instructions.

Docker Installation

• Build the Docker image: docker build -t aegis .
• Run the Docker container: docker run -it --gpus all -v /path/to/your/data:/data aegis /bin/bash This will mount your local data directory into the container at /data.

🔨 2. Running AEGIS:

Already familiar with WSI processing? Perform segmentation, patching, and UNI feature extraction from a directory of WSIs with:

aegis --task all --wsi_dir ./wsis --job_dir ./aegis_processed --patch_encoder uni_v1 --mag 20 --patch_size 256

Feeling cautious?

Run this command to perform all processing steps for a single slide:

aegis --slide_path ./wsis/xxxx.svs --job_dir ./aegis_processed --patch_encoder uni_v1 --mag 20 --patch_size 256

Or follow step-by-step instructions:

Step 1: Tissue Segmentation: Segments tissue vs. background from a dir of WSIs

• Command:

  aegis --task seg --wsi_dir ./wsis --job_dir ./aegis_processed --gpu 0 --segmenter hest
  

  • --task seg: Specifies that you want to do tissue segmentation.
  • --wsi_dir ./wsis: Path to dir with your WSIs.
  • --job_dir ./aegis_processed: Output dir for processed results.
  • --gpu 0: Uses GPU with index 0.
  • --segmenter: Segmentation model. Defaults to hest. Switch to grandqc for fast H&E segmentation. Add the option --remove_artifacts for additional artifact clean up.
• Outputs:
  • WSI thumbnails in ./aegis_processed/thumbnails.
  • WSI thumbnails with tissue contours in ./aegis_processed/contours.
  • GeoJSON files containing tissue contours in ./aegis_processed/contours_geojson. These can be opened in QuPath for editing/quality control, if necessary.

Step 2: Tissue Patching: Extracts patches from segmented tissue regions at a specific magnification.

• Command:

  aegis --task coords --wsi_dir ./wsis --job_dir ./aegis_processed --mag 20 --patch_size 256 --overlap 0
  

  • --task coords: Specifies that you want to do patching.
  • --wsi_dir wsis: Path to the dir with your WSIs.
  • --job_dir ./aegis_processed: Output dir for processed results.
  • --mag 20: Extracts patches at 20x magnification.
  • --patch_size 256: Each patch is 256x256 pixels.
  • --overlap 0: Patches overlap by 0 pixels (always an absolute number in pixels, e.g., --overlap 128 for 50% overlap for 256x256 patches.
• Outputs:
  • Patch coordinates as h5 files in ./aegis_processed/20x_256px/patches.
  • WSI thumbnails annotated with patch borders in ./aegis_processed/20x_256px/visualization.

Step 3a: Patch Feature Extraction: Extracts features from tissue patches using a specified encoder

• Command:

  aegis --task feat --wsi_dir ./wsis --job_dir ./aegis_processed --patch_encoder uni_v1 --mag 20 --patch_size 256
  

  • --task feat: Specifies that you want to do feature extraction.
  • --wsi_dir wsis: Path to the dir with your WSIs.
  • --job_dir ./aegis_processed: Output dir for processed results.
  • --patch_encoder uni_v1: Uses the UNI patch encoder. See below for list of supported models.
  • --mag 20: Features are extracted from patches at 20x magnification.
  • --patch_size 256: Patches are 256x256 pixels in size.
• Outputs:
  • Features are saved as h5 files in ./aegis_processed/20x_256px/features_uni_v1. (Shape: (n_patches, feature_dim))

AEGIS supports 21 patch encoders, loaded via a patch encoder_factory. Models requiring specific installations will return error messages with additional instructions. Gated models on HuggingFace require access requests.

Step 3b: Slide Feature Extraction: Extracts slide embeddings using a slide encoder. Will also automatically extract the right patch embeddings.

• Command:

  aegis --task feat --wsi_dir ./wsis --job_dir ./aegis_processed --slide_encoder titan --mag 20 --patch_size 512
  

  • --task feat: Specifies that you want to do feature extraction.
  • --wsi_dir wsis: Path to the dir containing WSIs.
  • --job_dir ./aegis_processed: Output dir for processed results.
  • --slide_encoder titan: Uses the Titan slide encoder. See below for supported models.
  • --mag 20: Features are extracted from patches at 20x magnification.
  • --patch_size 512: Patches are 512x512 pixels in size.
• Outputs:
  • Features are saved as h5 files in ./aegis_processed/20x_256px/slide_features_titan. (Shape: (feature_dim))

AEGIS supports 5 slide encoders, loaded via a slide-level encoder_factory. Models requiring specific installations will return error messages with additional instructions. Gated models on HuggingFace require access requests.

[!NOTE] If your task includes multiple slides per patient, you can generate patient-level embeddings by: (1) processing each slide independently and taking their average slide embedding (late fusion) or (2) pooling all patches together and processing that as a single "pseudo-slide" (early fusion).

Please see our tutorials for more support.

🙋 FAQ

• Q: How do I extract patch embeddings from legacy patch coordinates extracted with CLAM?

  • A:

    aegis --task feat --wsi_dir ..wsis --job_dir legacy_dir --patch_encoder uni_v1 --mag 20 --patch_size 256 --coords_dir extracted_mag20x_patch256_fp/
    

• Q: How do I keep patches corresponding to holes in the tissue?

  • A: In aegis, this behavior is default. Set --remove_holes to exclude patches on top of holes.

• Q: I see weird messages when building models using timm. What is happening?

  • A: Make sure timm==0.9.16 is installed. timm==1.X.X creates issues with most models.

• Q: How can I use aegis in other repos with minimal work?

  • A: Make sure aegis is installed using pip install -e .. Then, the aegis command will be available in your path.

• Q: I am not satisfied with the tissue vs background segmentation. What can I do?

  • A: AEGIS uses GeoJSON to store and load segmentations. This format is natively supported by QuPath. You can load the AEGIS segmentation into QuPath, modify it using QuPath's annotation tools, and save the updated segmentation back to GeoJSON.
  • A: You can try another segmentation model by specifying segmenter --grandqc.

• Q: I want to process a custom list of WSIs. Can I do it? Also, most of my WSIs don't have the micron per pixel (mpp) stored. Can I pass it?

  • A: Yes using the --custom_list_of_wsis argument. Provide a list of WSI names in a CSV (with slide extension, wsi). Optionally, provide the mpp (field mpp)

• Q: Do I need to install any additional packages to use AEGIS?

  • A: Most pretrained models require additional dependencies (e.g., the CTransPath patch encoder requires pip install timm_ctp). When you load a model using AEGIS, it will tell you what dependencies are missing and how to install them.

License and Terms of Use

ⓒ Siddhesh Thakur. This repository is released under the CC-BY-NC-ND 4.0 license and may only be used for non-commercial, academic research purposes with proper attribution. Any commercial use, sale, or other monetization of this repository is prohibited and requires prior approval. By downloading any pretrained encoder, you agree to follow the model's respective license.

Acknowledgements

The project was built on top of amazing repositories such as Timm, HuggingFace, and open-source contributions from the community. We thank the authors and developers for their contribution.

Issues

• The preferred mode of communication is via GitHub issues.
• If GitHub issues are inappropriate, email Siddhesh Thakur.
• Immediate response to minor issues may not be available.

How to cite

If you find our work useful in your research or if you use parts of this code, please consider citing our work.

(Citation placeholder)