lartpang/PySODMetrics

PySODMetrics: A Simple and Efficient Implementation of Grayscale/Binary Segmentation Metrics

Python

449

87 commits

updated Aug 23, 2026

lartpang.github.io/PySODMetrics

See the code

README

PySODMetrics: A simple and efficient implementation of SOD metrics

[!important]

Our exploration in this field continues with PyIRSTDMetrics, a project born from the same core motivation. Think of them as twin initiatives: this project maps the landscape of current evaluation, while its sibling takes the next step to expand upon and rethink it. We'd love your star! 🌟

Introduction

A simple and efficient implementation of SOD metrics.

Based on numpy and scipy
Verification based on Fan's matlab code https://github.com/DengPingFan/CODToolbox
The code structure is simple and easy to extend
The code is lightweight and fast

Your improvements and suggestions are welcome.

Citation

If you use this library in your research, please consider citing the following paper:

@misc{PySODMetrics,
      title={Beyond Pixel Overlap: A Framework for Decomposing Segmentation Evaluation Metrics}, 
      author={Youwei Pang and Xiaoqi Zhao},
      year={2026},
      eprint={2607.00886},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2607.00886}, 
}

PySODEvalToolkit: A Python-based Evaluation Toolbox for Salient Object Detection and Camouflaged Object Detection

Supported Metrics

Metric	Sample-based	Whole-based	Related Class
MAE	soft,si-soft		`MAE`
S-measure $S_{m}$	soft		`Smeasure`
weighted F-measure ($F^{\omega}_{\beta}$)	soft		`WeightedFmeasure`
Human Correction Effort Measure	soft		`HumanCorrectionEffortMeasure`
Context-Measure ($C_{\beta}$, $C^{\omega}_{\beta}$)	soft		`ContextMeasure`, `CamouflageContextMeasure`
Multi-Scale IoU	max,avg,adp,bin		`MSIoU`
E-measure ($E_{m}$)	max,avg,adp		`Emeasure`
F-measure (old) ($F_{\beta}$)	max,avg,adp		`Fmeasure` (Will be removed!)
F-measure (new) ($F_{\beta}$, $F_{1}$)	max,avg,adp,bin,si-max,si-avg,si-adp,si-bin	bin,si-max,si-avg,si-bin	`FmeasureV2`+`FmeasureHandler`
BER	max,avg,adp,bin,si-max,si-avg,si-adp,si-bin	bin,si-max,si-avg,si-bin	`FmeasureV2`+`BERHandler`
Dice	max,avg,adp,bin,si-max,si-avg,si-adp,si-bin	bin,si-max,si-avg,si-bin	`FmeasureV2`+`DICEHandler`
FPR	max,avg,adp,bin,si-max,si-avg,si-adp,si-bin	bin,si-max,si-avg,si-bin	`FmeasureV2`+`FPRHandler`
IoU	max,avg,adp,bin,si-max,si-avg,si-adp,si-bin	bin,si-max,si-avg,si-bin	`FmeasureV2`+`IOUHandler`
Kappa	max,avg,adp,bin,si-max,si-avg,si-adp,si-bin	bin,si-max,si-avg,si-bin	`FmeasureV2`+`KappaHandler`
Overall Accuracy	max,avg,adp,bin,si-max,si-avg,si-adp,si-bin	bin,si-max,si-avg,si-bin	`FmeasureV2`+`OverallAccuracyHandler`
Precision	max,avg,adp,bin,si-max,si-avg,si-adp,si-bin	bin,si-max,si-avg,si-bin	`FmeasureV2`+`PrecisionHandler`
Recall	max,avg,adp,bin,si-max,si-avg,si-adp,si-bin	bin,si-max,si-avg,si-bin	`FmeasureV2`+`RecallHandler`
Sensitivity	max,avg,adp,bin,si-max,si-avg,si-adp,si-bin	bin,si-max,si-avg,si-bin	`FmeasureV2`+`SensitivityHandler`
Specificity	max,avg,adp,bin,si-max,si-avg,si-adp,si-bin	bin,si-max,si-avg,si-bin	`FmeasureV2`+`SpecificityHandler`
TNR	max,avg,adp,bin,si-max,si-avg,si-adp,si-bin	bin,si-max,si-avg,si-bin	`FmeasureV2`+`TNRHandler`
TPR	max,avg,adp,bin,si-max,si-avg,si-adp,si-bin	bin,si-max,si-avg,si-bin	`FmeasureV2`+`TPRHandler`

NOTE:

Sample-based si- variants calculate the sample-specific mean/maximum based on the sample-based fm sequence with a shape of (num_targets, 256).
Whole-based si- variants calculate the mean/maximum based on the average fm sequence across all targets from all samples.
Because the *adp variants are specialized for sample-based computation, they do not support whole-based computation.

Usage

The core files are in the folder py_sod_metrics.

[Latest, but may be unstable] Install from the source code: pip install git+https://github.com/lartpang/PySODMetrics.git
[More stable] Install from PyPI: pip install pysodmetrics

Examples

Reference

Matlab Code by DengPingFan(https://github.com/DengPingFan): In our comparison (the test code can be seen under the test folder), the result is consistent with the code.
- The matlab code needs to change Bi_sal(sal>threshold)=1; to Bi_sal(sal>=threshold)=1; in https://github.com/DengPingFan/CODToolbox/blob/910358910c7824a4237b0ea689ac9d19d1958d11/Onekey_Evaluation_Code/OnekeyEvaluationCode/main.m#L102. For related discussion, please see the issue.
- 2021-12-20 (version 1.3.0): Due to the difference between numpy and matlab, in version 1.2.x, there are very slight differences on some metrics between the results of the matlab code and ours. The recent PR alleviated this problem. However, there are still very small differences on E-measure. The results in most papers are rounded off to three or four significant figures, so, there is no obvious difference between the new version and the version 1.2.x for them.
https://en.wikipedia.org/wiki/Precision_and_recall

@inproceedings{Fmeasure,
    title={Frequency-tuned salient region detection},
    author={Achanta, Radhakrishna and Hemami, Sheila and Estrada, Francisco and S{\"u}sstrunk, Sabine},
    booktitle=CVPR,
    number={CONF},
    pages={1597--1604},
    year={2009}
}
@inproceedings{Smeasure,
    title={Structure-measure: A new way to evaluate foreground maps},
    author={Fan, Deng-Ping and Cheng, Ming-Ming and Liu, Yun and Li, Tao and Borji, Ali},
    booktitle=ICCV,
    pages={4548--4557},
    year={2017}
}
@inproceedings{Emeasure,
    title="Enhanced-alignment Measure for Binary Foreground Map Evaluation",
    author="Deng-Ping {Fan} and Cheng {Gong} and Yang {Cao} and Bo {Ren} and Ming-Ming {Cheng} and Ali {Borji}",
    booktitle=IJCAI,
    pages="698--704",
    year={2018}
}
@inproceedings{wFmeasure,
    title={How to evaluate foreground maps?},
    author={Margolin, Ran and Zelnik-Manor, Lihi and Tal, Ayellet},
    booktitle=CVPR,
    pages={248--255},
    year={2014}
}
@inproceedings{MSIoU,
    title = {Multiscale IOU: A Metric for Evaluation of Salient Object Detection with Fine Structures},
    author = {Ahmadzadeh, Azim and Kempton, Dustin J. and Chen, Yang and Angryk, Rafal A.},
    booktitle = ICIP,
    year = {2021},
}
@inproceedings{SizeInvarianceVariants,
    title = {Size-invariance Matters: Rethinking Metrics and Losses for Imbalanced Multi-object Salient Object Detection},
    author = {Feiran Li and Qianqian Xu and Shilong Bao and Zhiyong Yang and Runmin Cong and Xiaochun Cao and Qingming Huang},
    booktitle = ICML,
    year = {2024}
}
@inproceedings{HumanCorrectionEffortMeasure,
    title = {Highly Accurate Dichotomous Image Segmentation},
    author = {Xuebin Qin and Hang Dai and Xiaobin Hu and Deng-Ping Fan and Ling Shao and Luc Van Gool},
    booktitle = ECCV,
    year = {2022}
}
@article{ContextMeasure,
    title={Context-measure: Contextualizing Metric for Camouflage},
    author={Wang, Chen-Yang and Ji, Gepeng and Shao, Song and Cheng, Ming-Ming and Fan, Deng-Ping},
    journal={arXiv preprint arXiv:2512.07076},
    year={2025}
}

metrics

metrics-evaluation

metrics-library

metrics-reported

python3

saliency

saliency-detection

saliency-map

saliency-maps

saliency-methods

saliency-model

saliency-prediction

salient-object-detection

salient-regions

Contributors

lartpang

86 commits

GewelsJI

1 commits

Similar repos

THE-3TC/cotescore

COTe score and Character Error Vector metrics for document understanding pipelines

rankseg/rankseg

Boost segmentation model mIoU/Dice instantly WITHOUT retraining. A plug-and-play, training-free…

ZhengPeng7/BiRefNet

[CAAI AIR'24] Bilateral Reference for High-Resolution Dichotomous Image Segmentation

4,215

xyfJASON/PerceptEval

An open-source toolkit for evaluating image perception models.

huggingface/evaluate

🤗 Evaluate: A library for easily evaluating machine learning models and datasets.

2,483

ZhengPeng7/BiRefNet-matting

Model

This repo holds the official weights of BiRefNet for general matting.

ZhengPeng7/BiRefNet_HR-matting

Model

> This BiRefNet was trained with images in `2048x2048` for higher resolution image matting with…

obss/sahi

Framework agnostic sliced/tiled inference + interactive ui + error analysis plots

5,513

Languages

Python

100.0%

lartpang/PySODMetrics

PySODMetrics: A Simple and Efficient Implementation of Grayscale/Binary Segmentation Metrics

Python

449

87 commits

updated Aug 23, 2026

lartpang.github.io/PySODMetrics

See the code

README

PySODMetrics: A simple and efficient implementation of SOD metrics

[!important]

Our exploration in this field continues with PyIRSTDMetrics, a project born from the same core motivation. Think of them as twin initiatives: this project maps the landscape of current evaluation, while its sibling takes the next step to expand upon and rethink it. We'd love your star! 🌟

Introduction

A simple and efficient implementation of SOD metrics.

Based on numpy and scipy
Verification based on Fan's matlab code https://github.com/DengPingFan/CODToolbox
The code structure is simple and easy to extend
The code is lightweight and fast

Your improvements and suggestions are welcome.

Citation

If you use this library in your research, please consider citing the following paper:

@misc{PySODMetrics,
      title={Beyond Pixel Overlap: A Framework for Decomposing Segmentation Evaluation Metrics}, 
      author={Youwei Pang and Xiaoqi Zhao},
      year={2026},
      eprint={2607.00886},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2607.00886}, 
}

PySODEvalToolkit: A Python-based Evaluation Toolbox for Salient Object Detection and Camouflaged Object Detection

Supported Metrics

Metric	Sample-based	Whole-based	Related Class
MAE	soft,si-soft		`MAE`
S-measure $S_{m}$	soft		`Smeasure`
weighted F-measure ($F^{\omega}_{\beta}$)	soft		`WeightedFmeasure`
Human Correction Effort Measure	soft		`HumanCorrectionEffortMeasure`
Context-Measure ($C_{\beta}$, $C^{\omega}_{\beta}$)	soft		`ContextMeasure`, `CamouflageContextMeasure`
Multi-Scale IoU	max,avg,adp,bin		`MSIoU`
E-measure ($E_{m}$)	max,avg,adp		`Emeasure`
F-measure (old) ($F_{\beta}$)	max,avg,adp		`Fmeasure` (Will be removed!)
F-measure (new) ($F_{\beta}$, $F_{1}$)	max,avg,adp,bin,si-max,si-avg,si-adp,si-bin	bin,si-max,si-avg,si-bin	`FmeasureV2`+`FmeasureHandler`
BER	max,avg,adp,bin,si-max,si-avg,si-adp,si-bin	bin,si-max,si-avg,si-bin	`FmeasureV2`+`BERHandler`
Dice	max,avg,adp,bin,si-max,si-avg,si-adp,si-bin	bin,si-max,si-avg,si-bin	`FmeasureV2`+`DICEHandler`
FPR	max,avg,adp,bin,si-max,si-avg,si-adp,si-bin	bin,si-max,si-avg,si-bin	`FmeasureV2`+`FPRHandler`
IoU	max,avg,adp,bin,si-max,si-avg,si-adp,si-bin	bin,si-max,si-avg,si-bin	`FmeasureV2`+`IOUHandler`
Kappa	max,avg,adp,bin,si-max,si-avg,si-adp,si-bin	bin,si-max,si-avg,si-bin	`FmeasureV2`+`KappaHandler`
Overall Accuracy	max,avg,adp,bin,si-max,si-avg,si-adp,si-bin	bin,si-max,si-avg,si-bin	`FmeasureV2`+`OverallAccuracyHandler`
Precision	max,avg,adp,bin,si-max,si-avg,si-adp,si-bin	bin,si-max,si-avg,si-bin	`FmeasureV2`+`PrecisionHandler`
Recall	max,avg,adp,bin,si-max,si-avg,si-adp,si-bin	bin,si-max,si-avg,si-bin	`FmeasureV2`+`RecallHandler`
Sensitivity	max,avg,adp,bin,si-max,si-avg,si-adp,si-bin	bin,si-max,si-avg,si-bin	`FmeasureV2`+`SensitivityHandler`
Specificity	max,avg,adp,bin,si-max,si-avg,si-adp,si-bin	bin,si-max,si-avg,si-bin	`FmeasureV2`+`SpecificityHandler`
TNR	max,avg,adp,bin,si-max,si-avg,si-adp,si-bin	bin,si-max,si-avg,si-bin	`FmeasureV2`+`TNRHandler`
TPR	max,avg,adp,bin,si-max,si-avg,si-adp,si-bin	bin,si-max,si-avg,si-bin	`FmeasureV2`+`TPRHandler`

NOTE:

Sample-based si- variants calculate the sample-specific mean/maximum based on the sample-based fm sequence with a shape of (num_targets, 256).
Whole-based si- variants calculate the mean/maximum based on the average fm sequence across all targets from all samples.
Because the *adp variants are specialized for sample-based computation, they do not support whole-based computation.

Usage

The core files are in the folder py_sod_metrics.

[Latest, but may be unstable] Install from the source code: pip install git+https://github.com/lartpang/PySODMetrics.git
[More stable] Install from PyPI: pip install pysodmetrics

Examples

Reference

Matlab Code by DengPingFan(https://github.com/DengPingFan): In our comparison (the test code can be seen under the test folder), the result is consistent with the code.
- The matlab code needs to change Bi_sal(sal>threshold)=1; to Bi_sal(sal>=threshold)=1; in https://github.com/DengPingFan/CODToolbox/blob/910358910c7824a4237b0ea689ac9d19d1958d11/Onekey_Evaluation_Code/OnekeyEvaluationCode/main.m#L102. For related discussion, please see the issue.
- 2021-12-20 (version 1.3.0): Due to the difference between numpy and matlab, in version 1.2.x, there are very slight differences on some metrics between the results of the matlab code and ours. The recent PR alleviated this problem. However, there are still very small differences on E-measure. The results in most papers are rounded off to three or four significant figures, so, there is no obvious difference between the new version and the version 1.2.x for them.
https://en.wikipedia.org/wiki/Precision_and_recall

@inproceedings{Fmeasure,
    title={Frequency-tuned salient region detection},
    author={Achanta, Radhakrishna and Hemami, Sheila and Estrada, Francisco and S{\"u}sstrunk, Sabine},
    booktitle=CVPR,
    number={CONF},
    pages={1597--1604},
    year={2009}
}
@inproceedings{Smeasure,
    title={Structure-measure: A new way to evaluate foreground maps},
    author={Fan, Deng-Ping and Cheng, Ming-Ming and Liu, Yun and Li, Tao and Borji, Ali},
    booktitle=ICCV,
    pages={4548--4557},
    year={2017}
}
@inproceedings{Emeasure,
    title="Enhanced-alignment Measure for Binary Foreground Map Evaluation",
    author="Deng-Ping {Fan} and Cheng {Gong} and Yang {Cao} and Bo {Ren} and Ming-Ming {Cheng} and Ali {Borji}",
    booktitle=IJCAI,
    pages="698--704",
    year={2018}
}
@inproceedings{wFmeasure,
    title={How to evaluate foreground maps?},
    author={Margolin, Ran and Zelnik-Manor, Lihi and Tal, Ayellet},
    booktitle=CVPR,
    pages={248--255},
    year={2014}
}
@inproceedings{MSIoU,
    title = {Multiscale IOU: A Metric for Evaluation of Salient Object Detection with Fine Structures},
    author = {Ahmadzadeh, Azim and Kempton, Dustin J. and Chen, Yang and Angryk, Rafal A.},
    booktitle = ICIP,
    year = {2021},
}
@inproceedings{SizeInvarianceVariants,
    title = {Size-invariance Matters: Rethinking Metrics and Losses for Imbalanced Multi-object Salient Object Detection},
    author = {Feiran Li and Qianqian Xu and Shilong Bao and Zhiyong Yang and Runmin Cong and Xiaochun Cao and Qingming Huang},
    booktitle = ICML,
    year = {2024}
}
@inproceedings{HumanCorrectionEffortMeasure,
    title = {Highly Accurate Dichotomous Image Segmentation},
    author = {Xuebin Qin and Hang Dai and Xiaobin Hu and Deng-Ping Fan and Ling Shao and Luc Van Gool},
    booktitle = ECCV,
    year = {2022}
}
@article{ContextMeasure,
    title={Context-measure: Contextualizing Metric for Camouflage},
    author={Wang, Chen-Yang and Ji, Gepeng and Shao, Song and Cheng, Ming-Ming and Fan, Deng-Ping},
    journal={arXiv preprint arXiv:2512.07076},
    year={2025}
}

metrics

metrics-evaluation

metrics-library

metrics-reported

python3

saliency

saliency-detection

saliency-map

saliency-maps

saliency-methods

saliency-model

saliency-prediction

salient-object-detection

salient-regions

Similar repos

THE-3TC/cotescore

COTe score and Character Error Vector metrics for document understanding pipelines

rankseg/rankseg

Boost segmentation model mIoU/Dice instantly WITHOUT retraining. A plug-and-play, training-free…

ZhengPeng7/BiRefNet

[CAAI AIR'24] Bilateral Reference for High-Resolution Dichotomous Image Segmentation

4,215

xyfJASON/PerceptEval

An open-source toolkit for evaluating image perception models.

huggingface/evaluate

🤗 Evaluate: A library for easily evaluating machine learning models and datasets.

2,483

ZhengPeng7/BiRefNet-matting

Model

This repo holds the official weights of BiRefNet for general matting.

ZhengPeng7/BiRefNet_HR-matting

Model

> This BiRefNet was trained with images in `2048x2048` for higher resolution image matting with…

obss/sahi

Framework agnostic sliced/tiled inference + interactive ui + error analysis plots

5,513

Contributors

lartpang

86 commits

GewelsJI

1 commits

Languages

Python

100.0%

lartpang/PySODMetrics

README

PySODMetrics: A simple and efficient implementation of SOD metrics

Introduction

Citation

Related Projects

Supported Metrics

Usage

Examples

Reference

Contributors

Similar repos

Languages

lartpang/PySODMetrics

README

PySODMetrics: A simple and efficient implementation of SOD metrics

Introduction

Citation

Related Projects

Supported Metrics

Usage

Examples

Reference

Similar repos

Contributors

Languages