BackdoorDM: A Comprehensive Benchmark for Backdoor Learning in Diffusion Model

BackdoorDM is the first comprehensive benchmark designed for backdoor learning research in diffusion models (DMs), which owns the following features:

• SOTA Methods Integration: A comprehensive benchmark integrating 9 backdoor attack methods, 5 backdoor defense methods, and 3 analysis tools in DMs to facilitate the research.
  • 9 backdoor attack methods: BadDiffusion, TrojDiff, VillanDiffusion (uncondition)(condition), InviBackdoor, BiBadDiff, RickRolling(TPA)(TAA), BadT2I(Pixel-Backdoor)(Object-Backdoor)(Style-Backdoor), PaaS(TI)(DB), EvilEdit.
  • 5 backdoor defense methods: Text Perturbations Defense, Elijah, TERD, T2IShield, DAA.
  • 3 analysis tools: Assimilation Phenomenon, Activation Norm, Pre-Activation Distribution.
• Systematic Taxonomy: A systematic classification and precise formulation of various backdoor attack types and target types in DMs, clearly defining the research scope in this field.
• Novel Evaluation Method: Integration of the current evaluation metrics and further propose a unified backdoor evaluation method using MLLM, which provides detailed image-level evaluation on both model specificity and model utility.
• Key Findings: New findings from the SOTA backdoor attacks of different target types, aiming to inspire future research.

We hope that BackdoorDM can help address current challenges in the backdoor learning research and contribute to building a trustworthy DMs community.

---

Table of Contents

• Quick Start
  • Installation
  • Attack
  • Defense
  • Evaluation
  • Visualization analysis
• Pre-trained Backdoored Models
• Supported attacks
• Supported defenses
• Analysis tools
• Evaluation metrics for different target types

---

Quick Start

Installation

# Clone the code repository.
git clone https://github.com/linweiii/BackdoorDM.git
cd ./BackdoorDM
# (optional) Install the environment. 
bash install.sh

Attack

Since the backdoor targets are diverse and complex in DMs, in this project, we separate the training and evaluation, i.e., no evaluation is conducted during the attack process. All attacked results (including model weights and logs) are stored in ./results folder.

We provide an attack example for EvilEdit, which is classified as ObjectRep:

1. Set the config files under the corresponding folder. base_config.yaml contains the basic setting (e.g., model, result folder, training dataset...). bd_config_[target type].yaml contains the target-specific setting for each attack (e.g., trigger, target...). The config files are stored distributedly based on different attack forms:

   |-- attack
     |-- t2i_gen
       |-- configs
         |-- base_config.yaml
         |-- bd_config_objectRep.yaml
       |-- eviledit
         |-- eviledit.py
     |-- uncond_gen
   

   Note that the specified values from command line are prior and can overwrite the ones in config files.

2. Run the attack. After setting the configs, you can run one specific attack via command line or use the scripts as one-click run for all attacks with the same target.

   # Attack example
   python ./attack/t2i_gen/eviledit/eviledit.py \
     --base_config 'attack/t2i_gen/configs/base_config.yaml' \
     --bd_config 'attack/t2i_gen/configs/bd_config_objectRep.yaml' \
     --model_ver 'sd15' \
     --device 'cuda:0'
   

   # One-click run: run all the supported objectRep attacks.
   bash ./scripts/run_attack_objectRep.sh
   

3. Attack results are stored in the ./results folder, which can be further used in Evaluation or Defense. The attacked result folder is named as [attack method]_[model version]:

   |-- results
     |-- eviledit_sd15
       |-- train_logs       # Store the training logs.
       |-- eviledit_*.pt    # Attacked model checkpoint.
   

Defense

We classify the current defense methods into input-level and model-level. Here, we provide an model-level example T2Ishield, which is applicable for all text-to-image attacks.

1. Set backdoored model path. You are encouraged to set the path of the backdoored model first in backdoored_model_path_dict under ./evaluation/configs/bdmodel_path.py file, which can also be used for evaluation. For example,

   'eviledit': 'eviledit_trigger-beautifuldog_target-cat.pt'
   

2. Run the defense. You can run defense for one specific attack via command line or use the scripts as one-click run for all attacks.

   # Defense example. You need to run EvilEdit attack before running defense.
   # Following the default setting.
   python ./defense/model_level/t2ishield/t2ishield.py \
       --backdoor_method 'eviledit' \
       --device 'cuda:0'
   

   # One-click run: run all the supported objectRep attacks.
   bash ./scripts/run_defend_t2ishield.sh
   

3. Defense results are stored in defense folder under the specific attacked results. Similar to attack results, the evaluation need to be conducted explicitly.

   |-- results
     |-- eviledit_sd15
       |-- defense            
         |-- t2ishield         # For T2IShield only.
           |-- defense_logs    # Store the defense logs.
           |-- defended_model  # Defended model checkpoint.
   

Evaluation

Evaluation with traditional method. All metrics except for the GPT-related ones are evaluated in main_eval.py.

1. Set backdoored model path and config files. You are encouraged to set the path of the backdoored model first in backdoored_model_path_dict under ./evaluation/configs/bdmodel_path.py file, and also set the evaluation config file eval_config*.yaml. For example, setting the backdoored model path for EvilEdit:

   'eviledit': 'eviledit_trigger-beautifuldog_target-cat.pt'
   

2. Run the evaluation. You can evaluate one specific metric for a backdoored model via command line or use the scripts as one-click run for all related results.

   # Evaluation example for ACC and ASR using ViT
   python ./evaluation/main_eval.py \
     --metric 'ACCASR' \
     --backdoor_method 'eviledit' \
     --device 'cuda:0'
   

   # One-click run: run all supported evaluation on ACC and ASR.
   bash ./scripts/run_eval_t2i_ACCASR.sh
   

3. Evaluation results. All tradictional evaluation results and logs are saved in eval_results.csv and eval_logs under the attacked results folder.

Evaluation with MLLM (GPT-4o). $ASR_{GPT}$, $PSR_{GPT}$ for model specificity, and $ACC_{GPT}$ for model utility are evaluated in mllm_eval.py. The evaluation configs are the same as above.

# Evaluation example with MLLM
python ./evaluation/mllm_eval.py \
    --eval_mllm 'gpt4o' \
    --backdoor_method 'eviledit' \
    --model_ver 'sd15' \
    --device 'cuda:0'

# One-click run: run all supported mllm evaluation.
bash ./scripts/run_eval_mllm.sh

The mllm evaluation results are stored in eval_mllm under the attacked results folder.

Visualization analysis

We provide three visualization analysis tools Assimilation Phenomenon, Activation Norm, and Pre-Activation Distribution. Note that assimilation analysis can only be applied to T2I models.

1. Set backdoored model path and config files. You are encouraged to set the path of the backdoored model first in backdoored_model_path_dict under ./evaluation/configs/bdmodel_path.py file, and also set the evaluation config file eval_config*.yaml.

2. Run the analysis. For assimilation phenomenon, you need to specify the prompts you want to use to analyze the model. Otherwise, the prompts will be chosen from the dataset randomly. For example, for EvilEdit:

   # Assimilation visualization example
   python ./analysis/assimilation/assimilation.py \
       --backdoor_method 'eviledit' \
       --result_dir 'eviledit_sd15' \
       --clean_prompts 'a dog sitting on the sofa' \
       --bd_prompts 'a beaytiful dog sitting on the sofa' \
       --device 'cuda:0'
   

   Activation norm analysis can be applied to both unconditional models and T2I models. For unconditional models, we hook the convolutional layers by default (65 convolutional layers in total). For T2I models, we hook the FFN layers by default (16 FFN layers in total).

   For unconditional models, we take BadDiffusion for example:

   # Activation norm visualization example (hook the first three layers)
   python ./analysis/activations/activations.py \
       --backdoor_method 'baddiffusion' \
       --result_dir 'baddiffusion_DDPM-CIFAR10-32' \
       --timesteps 1000 \
       --plot_timesteps [199, 399, 599, 799, 999] \
       --selected_layers [0, 1, 2] \  
       --device 'cuda:0'
   

   For T2I models, we take EvilEdit for example:

   # Activation norm visualization example (hook the first two layers)
   python ./analysis/activations/activations.py \
       --backdoor_method 'eviledit' \
       --result_dir 'eviledit_sd15' \
       --clean_prompts 'a dog sitting on the sofa' \
       --bd_prompts 'a beaytiful dog sitting on the sofa' \
       --timesteps 51 \
       --plot_timesteps [9, 19, 29, 39, 49] \
       --selected_layers [0, 1] \  
       --device 'cuda:0'
   

3. Visualization results (figures) are stored in analysis folder under the specific attacked results.

Pre-trained Backdoored Models

We release the attacked model weights used to produce the evaluation results in our paper, hosted on HuggingFace Hub:

📦 Weilin0/BackdoorDM

The released weights are all 17 attack/version combinations (9 attack methods × SD v1.5, plus SD v2.0 where applicable). They mirror the ./results layout used by this codebase, so after download they can be used directly by Evaluation, Defense, and Visualization analysis — see evaluation/configs/bdmodel_path.py.

Download

# One-click download everything (~58GB) into ./results
bash scripts/download_results.sh --all

# Selective download (e.g. just EvilEdit SD15)
bash scripts/download_results.sh --method eviledit --version sd15

# Interactive: choose methods/versions at the prompt
bash scripts/download_results.sh

Requires huggingface_hub (pip install huggingface_hub) or huggingface-cli.

Reported metrics

The evaluation results (ASR, PSR, ACC) for these weights are reported in our paper BackdoorDM: A Comprehensive Benchmark for Backdoor Learning in Diffusion Model (NeurIPS 2025 D&B) — see Table 18 / 19 / 4 and Section 4.2. Please refer to the paper for all quantitative results.

• Missing methods: BiBadDiff has no SD2.0 release; the ObjectAdd attacks (eviledit_numAdd, badt2i_objectAdd) are implemented in the code but their weights are not included in this release.

Intended use & license

• For research only. These are backdoored (poisoned) models. They are intended solely for backdoor defense research, attack-benchmark reproduction, and security analysis of text-to-image diffusion models.
• Do not use them in production image-generation services, or for any purpose where generated content is exposed to untrusted end users.
• The code is released under the MIT license; the models are released for research purposes only.

Supported attacks

Note: The methods implemented in this project (including attacks, defenses, and visualization) are adapted from the official code repositories.

Supported defenses

Analysis tools

Evaluation metrics for different target types

We evaluate the attack performance in terms of model specificity, model utility, and attack efficiency. Each target type of the defined taxonomy can be evaluated and compared under the same criteria: