hotflip_corpus_poisoning

This is the code repository for our ECIR2025 reproducibility paper 《Reproducing HotFlip for Corpus Poisoning Attacks in Dense Retrieval 》.

Files structure

• datasets/ contains the datasets files used in the experiments.
• results/ contains the results of the experiments.
• scripts/ contains the scripts to run the experiments.
• utils/ contains the utility functions used in the experiments.

Datasets

The datasets used in the experiments are from the BEIR library. The datasets are stored in the datasets/ folder. These datasets will download automatically when you run the code.

Requirements

• Python ,PyTorch , numpy, pandas, beir,transformers, sentence_transformers, sklearn, wandb
• If you do not want to use wandb, you can comment out all code with wandb in the code.
• You need to install the beir library(https://github.com/beir-cellar/beir)

Introduction of the code

Things need for the experiments in RQ1:

• 1, embedding_index.py is used to index the embeddings of the corpus, and save the retrieval results by BEIR.
• 2.1, hotflip_raw.py is used to generate the adversarial examples by hotflip (Zhong et al., 2023).
• 2.2, hotflip_attack_ours.py is used to generate the adversarial examples by hotflip (Zhong et al., 2023) with our pipeline optimizing strategy (Mean embedding).
• 3, evaluate_attack.py is used to evaluate the attack performance of the adversarial examples. The retrieval results are saved in results/.
• 4, attack_results_statistics.py is used to calculate the statistics of the attack results.

Things need for the experiments in RQ2:

• 1, repeat the steps in RQ1 to generate the adversarial examples with all 7 retrievers.
• 2, evaluate_attack.py is used to evaluate the attack performance of the adversarial examples. The retrieval results are saved in results/.
• 3, transfer_attack_statical.py is used to calculate the statistics of the attack results of black-box attacks.

Things need for the experiments in RQ3:

• 1, hotflip_ours_attack_corpus.py is used for the corpus poisoning attack with our pipeline optimizing strategy (Mean embedding).

Steps to reproduce the results

RQ1

• 1, Run sbatch scripts/embedding_index.sh to get the retrieval results of the all datsaets with all retrievers. The retrieval results are saved in results/beir_result.
• 2, Run sbatch scripts/generate_hotflip_multi_raw.sh to generate the adversarial examples by hotflip (Zhong et al., 2023). The adversarial examples are saved in results/hotflip_raw-generate.
• 3, Run sbatch scripts/generate_hotflip_multi.sh to generate the adversarial examples by hotflip (Zhong et al., 2023) with our pipeline optimizing strategy (Mean embedding). The adversarial examples are saved in results/hotflip-generate.
• 4, Run sbatch scripts/evaluate_attack.sh to evaluate the attack performance of the adversarial examples. The results are saved in results/attack_results.
• 5, Run python attack_results_statistics.py to calculate the statistics of the attack results.

RQ2

• 1, Since we have already generated the adversarial examples in RQ1, we do not need to repeat the steps in RQ1.
• 2, Run sbatch scripts/transfer_attack.sh. The retrieval results are saved in results/attack_results/hotflip and results/attack_results/hotflip_raw.
• 3, Run python transfer_attack_statical.py --method hotflip_raw and python transfer_attack_statical.py --method hotflip to calculate the statistics of the attack results of black-box attacks. Remember to change the method to hotflip_raw and hotflip respectively. And change seed_list = [1999] only for k_list=[10] in the evaluation hotflip_raw method.

RQ3

• 1, Run sbatch scripts/attack_corpus_ous.sh to generate the adversarial passages for the corpus poisoning attack. The results are saved in results_corpus_attack/hotflip-generate.
• 2, When you finish the code, they will output the evaluation results directly. Just record the results.

Citation

If you find this code useful, I would greatly appreciate it if you could cite our paper:

@inproceedings{li2025reproducinghotflip,
  title={Reproducing HotFlip for Corpus Poisoning Attacks in Dense Retrieval},
  author={Yongkang Li and Panagiotis Eustratiadis and Evangelos Kanoulas},
  booktitle    = {Advances in Information Retrieval - 47th European Conference on Information
                  Retrieval, {ECIR} 2025, Lucca, Italy, April 6-10, 2025, Proceedings,
                  Part {IV}},
  series       = {Lecture Notes in Computer Science},
  volume       = {15575},
  pages        = {95--111},
  publisher    = {Springer},
  year         = {2025},
  url          = {https://doi.org/10.1007/978-3-031-88717-8\_8},
  doi          = {10.1007/978-3-031-88717-8\_8},
}