README
Inaproppriate Image Prompts (I2P)
The I2P benchmark contains real user prompts for generative text2image prompts that are unproportionately likely to produce inappropriate images. I2P was introduced in the 2023 CVPR paper Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models. This benchmark is not specific to any approach or model, but was designed to evaluate mitigating measures against inappropriate degeneration in Stable Diffusion. The corresponding repository to evaluate text-to-image diffusion models on I2P can be found at https://github.com/ml-research/i2p.
Definition of inappropriate content
What is considered inappropriate imagery may differ based on context, setting, cultural and social predisposition as well as individual factors and is overall highly subjective. In this work we base our definition of inappropriate content on the work of Gebru et al.:
[data that] if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety.
--Gebru, Timnit, et al. "Datasheets for datasets," (2021)
which is for example also reflected by the OpenAI content policy OpenAI content policy that also applies to the use of DALL-E. Specifically, we consider images from the following categories: hate, harassment, violence, self-harm, sexual content, shocking images, illegal activity. Note that inappropriateness is not limited to these concepts, varies between cultures, and constantly evolves. Here we restricted ourselves on images displaying tangible acts of inappropriate behavior. Consequently, our test bed may contain prompts describing e.g.~geopolitical events or hateful symbols.
Data Collection
For the 7 concepts mentioned above we used 26 keywords and phrases describing them in more detail and collected up to 250 real world text prompts for each. For a given keyword we collected the prompts of the top 250 images returned by lexica.art. Lexica is a collection of real-world, user generated images base on the official stable diffusion discord and collects the prompt, seed, guidance scale and image dimensions used in generation. Image retrieval in lexica is based on the similarity of an image and search query in CLIP embedding space. Therefore, the collected prompts have generated images that are close to an inappropriate concept in CLIP space. Please note that we identify duplicate prompts based on their unique identifier on lexica. Accordingly, the I2P benchmark may contain entries with the same text prompt but different seeds and generation parameters.
Estimation of generated, inappropriate content
We provide an estimation of the percentage of inappropriate content based on images generated using Stable Diffusion v1.4. For each prompt we generated 10 images using the seed and guidance scale specified in lexica while keeping the maximum image size to 512x512. Our estimate is based on the portion out of these 10 images that are classified as inappropriate by either the Q16 classifier or NudeNet Detector.
Dataset Format
The benchmark consists of the following fields:
| Field | Explanation |
|---|---|
| prompt | The text prompt used as input to the model (Please not that this may contain Unicode characters) |
| categories | One or multiple of the 7 categories mentioned above that yield this prompt. |
| hard | Indicates if this is a hard prompt to remove inappropriate content from. Prompts are labled as hard if at least half of their generated images are inappropriate |
| inappropriate_percentage | Percentage of inappropriate images as classified by Q16 and NudeNet out of 10 generated images using Stable Diffusion |
| nudity_percentage | Percentage of images as depicting explicit nudity as per the NudeNet out of 10 generated images using Stable Diffusion |
| q16_percentage | Percentage of inappropriate images as classified by Q16 out of 10 generated images using Stable Diffusion |
| sd_safety_percentage | Percentage of inappropriate images as classified by the SD NSWF safety checker out of 10 generated images using Stable Diffusion |
| prompt_toxicity | The toxicity score of the text prompt according to the PerspectiveAPI |
| lexica_url | URL to the original prompt and the respective images in lexica for reference |
| sd_seed | Stable diffusion seed used in our image generation |
| sd_guidance_scale | Stable diffusion guidance scale used in our image generation |
| sd_image_width | Stable diffusion image width used in our image generation |
| sd_image_height | Stable diffusion image height used in our image generation |
Similar repos
Linked in READMEs
README
Inaproppriate Image Prompts (I2P)
The I2P benchmark contains real user prompts for generative text2image prompts that are unproportionately likely to produce inappropriate images. I2P was introduced in the 2023 CVPR paper Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models. This benchmark is not specific to any approach or model, but was designed to evaluate mitigating measures against inappropriate degeneration in Stable Diffusion. The corresponding repository to evaluate text-to-image diffusion models on I2P can be found at https://github.com/ml-research/i2p.
Definition of inappropriate content
What is considered inappropriate imagery may differ based on context, setting, cultural and social predisposition as well as individual factors and is overall highly subjective. In this work we base our definition of inappropriate content on the work of Gebru et al.:
[data that] if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety.
--Gebru, Timnit, et al. "Datasheets for datasets," (2021)
which is for example also reflected by the OpenAI content policy OpenAI content policy that also applies to the use of DALL-E. Specifically, we consider images from the following categories: hate, harassment, violence, self-harm, sexual content, shocking images, illegal activity. Note that inappropriateness is not limited to these concepts, varies between cultures, and constantly evolves. Here we restricted ourselves on images displaying tangible acts of inappropriate behavior. Consequently, our test bed may contain prompts describing e.g.~geopolitical events or hateful symbols.
Data Collection
For the 7 concepts mentioned above we used 26 keywords and phrases describing them in more detail and collected up to 250 real world text prompts for each. For a given keyword we collected the prompts of the top 250 images returned by lexica.art. Lexica is a collection of real-world, user generated images base on the official stable diffusion discord and collects the prompt, seed, guidance scale and image dimensions used in generation. Image retrieval in lexica is based on the similarity of an image and search query in CLIP embedding space. Therefore, the collected prompts have generated images that are close to an inappropriate concept in CLIP space. Please note that we identify duplicate prompts based on their unique identifier on lexica. Accordingly, the I2P benchmark may contain entries with the same text prompt but different seeds and generation parameters.
Estimation of generated, inappropriate content
We provide an estimation of the percentage of inappropriate content based on images generated using Stable Diffusion v1.4. For each prompt we generated 10 images using the seed and guidance scale specified in lexica while keeping the maximum image size to 512x512. Our estimate is based on the portion out of these 10 images that are classified as inappropriate by either the Q16 classifier or NudeNet Detector.
Dataset Format
The benchmark consists of the following fields:
| Field | Explanation |
|---|---|
| prompt | The text prompt used as input to the model (Please not that this may contain Unicode characters) |
| categories | One or multiple of the 7 categories mentioned above that yield this prompt. |
| hard | Indicates if this is a hard prompt to remove inappropriate content from. Prompts are labled as hard if at least half of their generated images are inappropriate |
| inappropriate_percentage | Percentage of inappropriate images as classified by Q16 and NudeNet out of 10 generated images using Stable Diffusion |
| nudity_percentage | Percentage of images as depicting explicit nudity as per the NudeNet out of 10 generated images using Stable Diffusion |
| q16_percentage | Percentage of inappropriate images as classified by Q16 out of 10 generated images using Stable Diffusion |
| sd_safety_percentage | Percentage of inappropriate images as classified by the SD NSWF safety checker out of 10 generated images using Stable Diffusion |
| prompt_toxicity | The toxicity score of the text prompt according to the PerspectiveAPI |
| lexica_url | URL to the original prompt and the respective images in lexica for reference |
| sd_seed | Stable diffusion seed used in our image generation |
| sd_guidance_scale | Stable diffusion guidance scale used in our image generation |
| sd_image_width | Stable diffusion image width used in our image generation |
| sd_image_height | Stable diffusion image height used in our image generation |