README
A Platform for Prompt Injection Evaluation
PIArena is an easy-to-use toolbox and also a comprehensive benchmark for researching prompt injection attacks and defenses. It was introduced in the paper PIArena: A Platform for Prompt Injection Evaluation.
The platform provides:
- Plug-and-play Attacks & Defenses – Easily integrate state-of-the-art defenses into your workflow to protect your LLM system against prompt injection attacks. You can also play with existing attack strategies to perform a better research.
- Systematic Evaluation Benchmark – End-to-end evaluation pipeline enables you to easily evaluate attacks / defenses on various datasets.
- Add Your Own – You can also easily integrate your own attack or defense into our benchmark to systematically assess how well it perform.
Dataset Statistics
| Dataset | Task Type | Utility Metric | Avg Len | #Samples |
|---|---|---|---|---|
| SQuAD v2 (Rajpurkar et al., 2018) | Question Answering | LLM-as-a-Judge | 706 | 200 |
| Dolly (QA) (Conover et al., 2023) | Question Answering | LLM-as-a-Judge | 1,062 | 200 |
| Dolly (Info Extraction) (Conover et al., 2023) | Information Extraction | LLM-as-a-Judge | 1,086 | 200 |
| Dolly (Summarization) (Conover et al., 2023) | Summarization | LLM-as-a-Judge | 1,567 | 200 |
| NQ (Kwiatkowski et al., 2019) | RAG | LLM-as-a-Judge | 5,432 | 100 |
| MS-MARCO (Bajaj et al., 2016) | RAG | LLM-as-a-Judge | 5,089 | 100 |
| HotpotQA (Yang et al., 2018) | RAG | LLM-as-a-Judge | 3,519 | 100 |
| HotpotQA-Long (Yang et al., 2018) | Question Answering | F1-Score | 17,942 | 100 |
| Qasper (Dasigi et al., 2021) | Question Answering | F1-Score | 18,523 | 100 |
| GovReport (Huang et al., 2021) | Summarization | ROUGE-L | 16,581 | 100 |
| MultiNews (Fabbri et al., 2019) | Summarization | ROUGE-L | 8,907 | 100 |
| PassageRetrieval (Bai et al., 2023) | Information Retrieval | Retrieval Score | 19,777 | 100 |
| LCC (Guo et al., 2023) | Code Generation | Code Similarity | 12,247 | 100 |
| Total | 1,700 |
Data Fields
Each sample contains the following fields:
| Field | Description |
|---|---|
target_inst | The original user instruction (e.g., a question to answer or a summarization request) |
context | The clean context provided to the LLM (e.g., a document or passage) |
injected_task | The adversarial prompt injection task embedded by the attacker |
target_task_answer | The ground-truth answer for the original task (used for utility evaluation) |
injected_task_answer | The expected output if the injection succeeds (used for ASR evaluation) |
category | The category of the injection attack |
Usage
from datasets import load_dataset
# Load a specific subset
ds = load_dataset("sleeepeer/PIArena", "squad_v2")
# Available subsets:
# Basic QA: squad_v2, dolly_closed_qa, dolly_information_extraction, dolly_summarization
# RAG: nq_rag, hotpotqa_rag, msmarco_rag
# Long-context: hotpotqa_long, qasper_long, gov_report_long, multi_news_long, passage_retrieval_en_long, lcc_long
# Knowledge corruption: nq_rag_knowledge_corruption, hotpotqa_rag_knowledge_corruption, msmarco_rag_knowledge_corruption
Citation
@inproceedings{geng2026piarena,
title={{PIArena}: A platform for prompt injection evaluation},
author={Geng, Runpeng and Yin, Chenlong and Wang, Yanting and Chen, Ying and Jia, Jinyuan},
booktitle={Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
pages={33170--33192},
year={2026}
}
Similar repos
README
A Platform for Prompt Injection Evaluation
PIArena is an easy-to-use toolbox and also a comprehensive benchmark for researching prompt injection attacks and defenses. It was introduced in the paper PIArena: A Platform for Prompt Injection Evaluation.
The platform provides:
- Plug-and-play Attacks & Defenses – Easily integrate state-of-the-art defenses into your workflow to protect your LLM system against prompt injection attacks. You can also play with existing attack strategies to perform a better research.
- Systematic Evaluation Benchmark – End-to-end evaluation pipeline enables you to easily evaluate attacks / defenses on various datasets.
- Add Your Own – You can also easily integrate your own attack or defense into our benchmark to systematically assess how well it perform.
Dataset Statistics
| Dataset | Task Type | Utility Metric | Avg Len | #Samples |
|---|---|---|---|---|
| SQuAD v2 (Rajpurkar et al., 2018) | Question Answering | LLM-as-a-Judge | 706 | 200 |
| Dolly (QA) (Conover et al., 2023) | Question Answering | LLM-as-a-Judge | 1,062 | 200 |
| Dolly (Info Extraction) (Conover et al., 2023) | Information Extraction | LLM-as-a-Judge | 1,086 | 200 |
| Dolly (Summarization) (Conover et al., 2023) | Summarization | LLM-as-a-Judge | 1,567 | 200 |
| NQ (Kwiatkowski et al., 2019) | RAG | LLM-as-a-Judge | 5,432 | 100 |
| MS-MARCO (Bajaj et al., 2016) | RAG | LLM-as-a-Judge | 5,089 | 100 |
| HotpotQA (Yang et al., 2018) | RAG | LLM-as-a-Judge | 3,519 | 100 |
| HotpotQA-Long (Yang et al., 2018) | Question Answering | F1-Score | 17,942 | 100 |
| Qasper (Dasigi et al., 2021) | Question Answering | F1-Score | 18,523 | 100 |
| GovReport (Huang et al., 2021) | Summarization | ROUGE-L | 16,581 | 100 |
| MultiNews (Fabbri et al., 2019) | Summarization | ROUGE-L | 8,907 | 100 |
| PassageRetrieval (Bai et al., 2023) | Information Retrieval | Retrieval Score | 19,777 | 100 |
| LCC (Guo et al., 2023) | Code Generation | Code Similarity | 12,247 | 100 |
| Total | 1,700 |
Data Fields
Each sample contains the following fields:
| Field | Description |
|---|---|
target_inst | The original user instruction (e.g., a question to answer or a summarization request) |
context | The clean context provided to the LLM (e.g., a document or passage) |
injected_task | The adversarial prompt injection task embedded by the attacker |
target_task_answer | The ground-truth answer for the original task (used for utility evaluation) |
injected_task_answer | The expected output if the injection succeeds (used for ASR evaluation) |
category | The category of the injection attack |
Usage
from datasets import load_dataset
# Load a specific subset
ds = load_dataset("sleeepeer/PIArena", "squad_v2")
# Available subsets:
# Basic QA: squad_v2, dolly_closed_qa, dolly_information_extraction, dolly_summarization
# RAG: nq_rag, hotpotqa_rag, msmarco_rag
# Long-context: hotpotqa_long, qasper_long, gov_report_long, multi_news_long, passage_retrieval_en_long, lcc_long
# Knowledge corruption: nq_rag_knowledge_corruption, hotpotqa_rag_knowledge_corruption, msmarco_rag_knowledge_corruption
Citation
@inproceedings{geng2026piarena,
title={{PIArena}: A platform for prompt injection evaluation},
author={Geng, Runpeng and Yin, Chenlong and Wang, Yanting and Chen, Ying and Jia, Jinyuan},
booktitle={Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
pages={33170--33192},
year={2026}
}