Project SEG4SEG

Public pytorch implementation for our paper SEG4SEG: Identifying Systematic Failure Modes in Segmentation by Subgroup Discovery Methods.

If you find this code helpful in your research, please cite the following paper:

@inproceedings{weng2026seg4seg,
        title={SEG4SEG: Identifying Systematic Failure Modes in Segmentation by Subgroup Discovery Methods},
        author={Weng, Nina and Petersen, Eike and Bissoto, Alceu and Sun, Susu and Koch, Lisa M and Feragen, Aasa and Bigdeli, Siavash and Baumgartner, Christian F},
        booktitle={Medical Imaging with Deep Learning},
        year={2026}
}

This work has been accepted to MIDL 2026 as Spotlight Presentation.

---

Getting Started

Step 0: Virtual Environment Setup and Update global_config.py

The code is implemented in Python 3.9 using the Pytorch library. We only tested the code with Python 3.9.

Install dependencies:

pip install -r requirements.txt

requirements.txt contains the minimal direct dependencies for this repo.

Step 1: Run SDM With Trained Segmentation Models

Run one SDM sweep with a trained segmentation model:

python3 -m sdm.run_once --ds_name isic2018_d --pre_trained_seg_model_config_name 250527090603_unet_isic_bce

What it does:

• loads test embeddings and segmentation performance
• runs clustering (gmm or hdbscan)
• saves csv results to ./records/{save_to}_{cluster_method}/{ds_name}_{config_name}/

For full argument and hyperparameter details, see sdm/run_once.py.

Step 2: SDM Result Analysis

After running SDM, you will get one CSV per hyperparameter setting under:

./records/{save_to}_{cluster_method}/{ds_name}_{config_name}/

Each CSV contains per-image cluster assignment and model performance, e.g.:

We provide 4 analysis notebooks in ./sdm/, one per case:

• Case A (Ultrasound calipers): sdm/sdm_analysis_caseA_updated.ipynb
• Case B (ISIC central crop): sdm/sdm_analysis_caseB_updated.ipynb
• Case C (ISIC annotation style): sdm/sdm_analysis_caseC_updated.ipynb
• Case D (FIVES image quality): sdm/sdm_analysis_caseD_updated.ipynb

All notebooks follow the same 3-part analysis:

1. Validate whether the target attribute is associated with performance differences (group statistics + significance test).
2. Plot SDM trend across lambda (performance gap and subgroup purity).
3. Run rank-based significance analysis across lambda and random seeds.

Notebook Usage

1. Run Section 1 in the notebook (Analysis config) and set:
   • dataset-specific options (ds_name_base, bias_levels, ds_config_map)
   • SDM options (cluster_method, sdm_ks, sdm_lambdas, random_seeds)
   • file paths (BASE_DIR, dataset metadata paths)
2. Run the metadata-loading cell and confirm no missing-file warnings.
3. Run all remaining cells top-to-bottom to generate tables and figures.

Synthetic vs Real-World Modes

• Case A and Case B are synthetic benchmark analyses with multiple bias levels.
• Case C and Case D support both synthetic and real-world analysis in one notebook.
  • Set MODE = 'synthetic' for shortcut datasets.
  • Set MODE = 'real' for original real-world datasets.

Note: make sure corresponding SDM CSV files and segmentation Dice files (records/seg_model/.../dice_test.pt) exist before running the notebooks.

If you want to use our pretrained models for analysis, you can download them from this link and follow the instructions provided in that link.

Step 3: Create Synthetic Datasets

We provide 4 scripts under prepare_dataset/.

First, set paths in global_config.py:

• FIVES_DATASET_DIR
• HC18_DATASET_DIR
• ISIC2018_DATASET_DIR
• ISIC2018_DATAFILES_DIR

Then run:

1. FIVES image quality (dark + blur)

python ./prepare_dataset/fives/mkdataset.py --splits train,test --percents 2,5,10,20

2. HC18 caliper artifacts

python ./prepare_dataset/hc18/mkdataset.py --bias-proportions 20,40,60,80

3. ISIC2018 annotation style (f -> p)

python ./prepare_dataset/isic2018_annotationstyle/mkdataset.py --percents 10,20,40,60,80,90

4. ISIC2018 central cropping

python ./prepare_dataset/isic2018_central_cropping/mkdataset.py --percents 20,40,60,80 --seed 42

Outputs:

• generated images/masks in dataset folders
• metadata CSV files for training/analysis

Step 4: Train a Segmentation Model With Synthetic Datasets

Command:

python3 -m train.train_seg --ds_name <DATASET_NAME> --model unet --loss_type bce --batch_size 32 --lr 1e-4 --num_epochs 100

Datasets:

• fives_p2, fives_p5, fives_p10, fives_p20
• hc18_caliper20, hc18_caliper40, hc18_caliper60, hc18_caliper80, hc18_caliper95
• isic2018_p20, isic2018_p40, isic2018_p60, isic2018_p80
• isic2018_as_p10, isic2018_as_p20, isic2018_as_p40, isic2018_as_p60, isic2018_as_p80, isic2018_as_p90

Example:

python3 -m train.train_seg --ds_name isic2018_as_p95 --model unet --loss_type bce --batch_size 32 --lr 1e-5 --num_epochs 100

What it does:

• train U-Net (BCE or Dice)
• evaluate on val/test each epoch
• log to W&B
• save best model + config under records/seg_model/...

Notes:

• For fives*, hc18*, and isic2018_as_p*, validation is created by splitting train (80/20).
• For isic, isic2018, isic2018_d, isic2018_p*, and isic2018_v2_p*, validation is loaded from the dataset val split.

Step 5: Dataset Links

The datasets used in this project could be found by the following link:

• HC18
• ISIC2018
• FIVES

The annotation-style metadata for ISIC2018 is provided in:

• datafiles/ISIC2018/isic2018_anno_styles.csv which is annotated by Nina Weng using LabelMe.

If you found our annotation useful, please cite our paper.