CO-SPY: Combining Semantic and Pixel Features to Detect Synthetic Images by AI (CVPR 2025)

With the rapid advancement of generative AI, it is now possible to synthesize high-quality images in a few seconds. Despite the power of these technologies, they raise significant concerns regarding misuse. To address this, various synthetic image detectors have been proposed. However, many of them struggle to generalize across diverse generation parameters and emerging generative models. In this repository, we introduce Co-Spy-Bench, a comprehensive benchmark designed to evaluate the robustness and effectiveness of synthetic image detectors under a wide range of generative conditions.

Features

Co-Spy-Bench highlights its diversity and comprehensive coverage of the latest generative models:

• Captions are sourced from five real-world datasets: MSCOCO2017, CC3M, Flickr, TextCaps, and SBU.
• Includes synthetic images generated by 22 different models, spanning a broad spectrum of architectures.
• Varies generation parameters such as diffusion steps and guidance scales to enhance diversity.

Dataset Structure

├── cc3m
│   ├── CompVis@ldm-text2im-large-256
│   │   ├── images_with_metadata.tar.gz
│   ├── CompVis@stable-diffusion-v1-4
│   │   ├── images_with_metadata.tar.gz
│   ├── PixArt-alpha@PixArt-XL-2-1024-MS
│   │   ├── images_with_metadata.tar.gz
│   ├── PixArt-alpha@PixArt-XL-2-512x512
│   │   ├── images_with_metadata.tar.gz
│   ├── black-forest-labs@FLUX.1-dev
│   │   ├── images_with_metadata.tar.gz
│   ├── black-forest-labs@FLUX.1-schnell
│   │   ├── images_with_metadata.tar.gz
│   ├── latent-consistency@lcm-lora-sdv1-5
│   │   ├── images_with_metadata.tar.gz
│   ├── latent-consistency@lcm-lora-sdxl
│   │   ├── images_with_metadata.tar.gz
│   ├── playgroundai@playground-v2-1024px-aesthetic
│   │   ├── images_with_metadata.tar.gz
│   ├── playgroundai@playground-v2-256px-base
│   │   ├── images_with_metadata.tar.gz
│   ├── playgroundai@playground-v2-512px-base
│   │   ├── images_with_metadata.tar.gz
│   ├── playgroundai@playground-v2.5-1024px-aesthetic
│   │   ├── images_with_metadata.tar.gz
│   ├── runwayml@stable-diffusion-v1-5
│   │   ├── images_with_metadata.tar.gz
│   ├── segmind@SSD-1B
│   │   ├── images_with_metadata.tar.gz
│   ├── segmind@SegMoE-SD-4x2-v0
│   │   ├── images_with_metadata.tar.gz
│   ├── segmind@small-sd
│   │   ├── images_with_metadata.tar.gz
│   ├── segmind@tiny-sd
│   │   ├── images_with_metadata.tar.gz
│   ├── stabilityai@sdxl-turbo
│   │   ├── images_with_metadata.tar.gz
│   ├── stabilityai@stable-diffusion-2-1
│   │   ├── images_with_metadata.tar.gz
│   ├── stabilityai@stable-diffusion-2
│   │   ├── images_with_metadata.tar.gz
│   ├── stabilityai@stable-diffusion-3-medium-diffusers
│   │   ├── images_with_metadata.tar.gz
│   ├── stabilityai@stable-diffusion-xl-base-1.0
│   │   ├── images_with_metadata.tar.gz
├── flickr
│   ├── ...
├── mscoco
│   ├── ...
├── sbu
│   ├── ...
├── textcaps
│   ├── ...

Typically, each folder contains a file named images_with_metadata.tar.gz, which includes synthetic images and their corresponding metadata. These are generated by a specific generative model using captions from the root folder.

For example, cc3m/black-forest-labs@FLUX.1-dev/images_with_metadata.tar.gz indicates that the data in this compressed file were generated by FLUX.1-dev using captions from the CC3M dataset.

images_with_metadata.tar.gz typically contains 5,000 synthetic images, along with the metadata used during their generation.

BibTex

If you find this work helpful, please kindly like ❤️ our dataset repo and cite 📄 our paper.

@inproceedings{cheng2025co,
  title={CO-SPY: Combining Semantic and Pixel Features to Detect Synthetic Images by AI},
  author={Cheng, Siyuan and Lyu, Lingjuan and Wang, Zhenting and Zhang, Xiangyu and Sehwag, Vikash},
  booktitle={Proceedings of the Computer Vision and Pattern Recognition Conference},
  pages={13455--13465},
  year={2025}
}