On the Adversarial Robustness of Visual-Language Chat Models

Introdution

This is the official repository of our ICMR 2025 paper "On the Adversarial Robustness of Visual-Language Chat Models"

Requirements

• Install required python packages:

python -m pip install -r requirements.py

Training

Training commands are as follows.

• VQA:

python scripts/image_attack_vqa_target.py \
    --eps <noise_budget> \
    --step_size <step_size> \
    --epoch <steps_of_PGD> \
    --model <model_name>

• Jailbreaking:

python scripts/image_attack_bad_prompts.py \
    --eps <noise_budget> \
    --step_size <step_size> \
    --epoch <steps_of_PGD> \
    --model <model_name>

• Information hiding:

python scripts/image_attack_information.py \
    --eps <noise_budget> \
    --step_size <step_size> \
    --epoch <steps_of_PGD> \
    --model <model_name>

The parameter choices for the above commands are as follows:

• Noise budget --eps: 4 , 8, 16, 32, ...
• Steps of PGD --epoch: 100, 200, 300, 400, ...
• Model name --model: minigpt4 , minigpt5, mplug, mplug2, ...

The trained checkpoints will be saved at scripts/experiments_<type>/<model>_eps=<args.eps>_epoch=<args.epoch>.

Acknowledgement

• Relevant VLMs:
  • MiniGPT-4,
  • MiniGPT-5,
  • mPlUG-owl,
  • mPlUG-owl2,
  • LLaVA-v1.5.

Cite the paper

@inproceedings{qin2025m3,
  author       = {Tianrui Qin and Xuan Wang and  Juanjuan Zhao and Kejiang Ye and Cheng-Zhong Xu and Xitong Gao},
  editor       = {Zhongfei (Mark) Zhang and Elisa Ricci and Yan Yan and Liqiang Nie and Vincent Oria and Lamberto Ballan},
  title        = {On the Adversarial Robustness of Visual-Language Chat Models},
  booktitle    = {Proceedings of the 2025 International Conference on Multimedia Retrieval, {ICMR} 2025, Chicago, IL, USA, 30 June 2025 - 3 July 2025},
  pages        = {1118--1127},
  publisher    = {{ACM}},
  year         = {2025},
  url          = {https://doi.org/10.1145/3731715.3733407},
  doi          = {10.1145/3731715.3733407},
  timestamp    = {Thu, 26 Jun 2025 14:33:29 +0200},
  biburl       = {https://dblp.org/rec/conf/mir/QinWZY0G25.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}