An Open Benchmark for Evaluating Jailbreak Guardrails in Large Language Models

Introduction

This repository provides instruction datasets in our SoK paper, SoK: Evaluating Jailbreak Guardrails for Large Language Models. The datasets are collected from various sources to evaluate the effectiveness of jailbreak guardrails in large language models (LLMs), including harmful prompts (i.e., JailbreakHub, JailbreakBench, MultiJail, and SafeMTData) and normal prompts (i.e., AlpacaEval and OR-Bench). The details (i.e., the number of prompts and jailbreak methods used for each dataset) of these sub datasets are summarized as follows:

Accessing the Datasets

Some of the contents of the datasets may be offensive to some readers

For JailbreakHub, MultiJail, AlpacaEval and OR-Bench, each entry in the dataset is a dictionary with the following keys:

• index: The index of the jailbreak prompt.
• goal: The original goal of the jailbreak prompt.
• jailbreak prompt: The jailbreak prompt used to attack LLMs. Although AlpacaEval and OR-Bench are in benign prompts, we still use jailbreak prompt as the key to keep the consistency of the datasets.

For JailbreakBench and SafeMTData, please refer to their respective Hugging Face pages for more details.

Sample Usage

To evaluate guardrails, you can refer to the scripts provided in the GitHub repository. For instance, you can run the following command to evaluate all the guardrails to protect Llama-3-8B-Instruct against the IJP attack:

sh ./scripts/ijp.sh

The results will then be saved in ./results/llama-3-8b/IJP/.

Usage

Please refer to our github repository SoK4JailbreakGuardrails for more details on how to use the datasets for evaluating jailbreak guardrails in LLMs. The repository provides scripts to evaluate various jailbreak guardrails using the datasets mentioned above.

Citation

If you find our benchmark useful, please consider citing it as follows:

@article{wang2025sok,
  title={SoK: Evaluating Jailbreak Guardrails for Large Language Models},
  author={Wang, Xunguang and Ji, Zhenlan and Wang, Wenxuan and Li, Zongjie and Wu, Daoyuan and Wang, Shuai},
  journal={arXiv preprint arXiv:2506.10597},
  year={2025}
}