Cryo-Bench 🧊

A Benchmark for Evaluating Geospatial Foundation Models on Cryosphere Applications

Cryo-Bench is a community benchmark that evaluates geospatial foundation models (GFMs) on six cryosphere segmentation datasets spanning five components: supraglacial debris, glacial lakes (two sensing configurations), sea ice, calving fronts, and Antarctic ice-shelf extent. It is built on top of the PANGAEA evaluation protocol using multi-sensor satellite imagery from Sentinel-1/2, Landsat-8, WorldView-2, and historical SAR missions.

---

📋 Tasks & Datasets

Cryo-Bench includes six benchmark tasks covering five components of the cryosphere:

---

🚀 Installation & Quick Start

git clone https://github.com/Sk-2103/Cryo-Bench.git
cd Cryo-Bench
conda env create -f environment.yml
conda activate pangaea-bench

Download the dataset(s) you need from the links in the table above and note the local path you extract them to (<data_root>). pangaea/run.py is the entry point and takes a Hydra config for every axis (dataset, encoder, decoder, ...), so a full run looks like this:

python -m torch.distributed.run --standalone --nproc_per_node=1 pangaea/run.py \
  task=segmentation preprocessing=seg_default criterion=cross_entropy \
  dataset=sicd_rgb encoder=unet_encoder decoder=seg_unet \
  dataset.root_path=<data_root>/SICD work_dir=./work_dir/unet_sicd \
  finetune=true limited_label_train=1

That reproduces the U-Net/SICD cell of the table below (27.17 mIoU). A few things to swap out:

• encoder is any file under configs/encoder/: dofa, terramind_large, croma_joint, vit_scratch, unet_encoder, ... For anything other than the U-Net baseline, use decoder=seg_upernet.
• dataset is any file under configs/dataset/: gssd, glid, glb_random, sicd_sar, sicd_rgb, zone_mapping_sar, zone_mapping_optical, shelf_bench.
• finetune=false freezes the encoder, the default evaluation regime for every GFM (the two from-scratch baselines always need finetune=true, since they have no pretrained weights). limited_label_train=0.1 runs the few-shot regime instead of the full training set.

See the paper for the full protocol (crop sizes, learning-rate schedule, normalization per dataset, evaluation regimes).

---

📁 Repository Structure

Cryo-Bench/
├── pangaea/            # evaluation framework
│   ├── datasets/       # one module per dataset (gssd, glid, glb, sicd_tiled, zone_mapping_*, shelf_bench)
│   ├── encoders/       # GFM wrappers (CROMA, DOFA, TerraMind, Prithvi, SatlasNet, Scale-MAE, SpectralGPT, SSL4EO-S12, ...) and the U-Net/ViT baselines
│   ├── decoders/       # UPerNet (GFMs) and U-Net (baseline) segmentation heads
│   ├── engine/         # trainer / evaluator
│   └── utils/
├── configs/             # Hydra configs, one subdirectory per axis
│   ├── dataset/         # 8 configs, one per dataset/modality variant
│   ├── encoder/         # 20 configs: 13 GFMs plus dataset-specific modality variants, and 2 baselines
│   ├── decoder/         # seg_unet.yaml, seg_upernet.yaml
│   └── preprocessing/, task/, criterion/, optimizer/, lr_scheduler/
└── assets/              # README images

---

🏆 Benchmark Results

The table reports mIoU (↑) with frozen encoders and 100% training data, using the UPerNet decoder across all six datasets. Rows are ordered by six-dataset average mIoU. U-Net and ViT are trained from scratch. Avg Rank is the mean of a model's rank (1-15) on each of the six datasets individually, and can read differently from the ordering by Avg mIoU (e.g. DOFA has a better Avg Rank than RemoteCLIP despite a lower Avg mIoU).

Bold = best performance · Italic = second best

📜 License

This project is licensed under the MIT License.

---

🙏 Acknowledgements

Cryo-Bench builds on the PANGAEA benchmark. We thank the developers of DOFA, TerraMind, Prithvi, SatlasNet, and all other foundation models included in this benchmark. We also thank the dataset authors of GSDD, GLID, GLB, SICD, CaFFe, and Shelf-Bench for making their data publicly available.