Prompt Injection Evaluation Tool

A comprehensive evaluation tool for prompt injection detection models. This tool includes dataset merging capabilities and supports evaluation of multiple model types with unified CSV datasets.

Features

• Dataset Management: Merge multiple format datasets (CSV, TSV, JSON, Parquet) into unified format, defaults to 20265 samples from various sources.
• Short Dataset Creation: Create balanced smaller datasets for faster testing and development
• Model Support: Multiple model types (Llama Prompt Guard, ProtectAI DeBERTa, PreambleAI, Qualifire, SavantAI, etc.)
• Binary Classification: Standardized binary labels (0=Safe, 1=Threat) with original label preservation
• Comprehensive Metrics: Detailed evaluation metrics including confusion matrix analysis
• Organized Output: Results saved in structured directories under results/
• Command Line Interface: Flexible CLI for evaluation with sampling support

Prerequisites

Environment Setup

conda create -n model-eval python=3.12
conda activate model-eval
python -m pip install torch pandas transformers tqdm pyarrow -i https://pypi.tuna.tsinghua.edu.cn/simple

Project Structure

prompt-injection-eval/
├── merge_datasets.py         # Dataset merging script
├── model-eval-v2.0.py       # Main evaluation script v2.0
├── model-eval-v1.1.py       # Legacy evaluation script
├── datasets/                # Dataset directory
│   ├── create_short_dataset.py  # Short dataset creation script
│   ├── dataset-source/      # Original datasets in various formats
│   │   ├── *.csv            # CSV datasets
│   │   ├── *.tsv            # TSV datasets
│   │   ├── *.json           # JSON datasets
│   │   └── *.parquet        # Parquet datasets
│   ├── merged_datasets.csv  # Unified merged dataset
│   └── merged_datasets_short_*.csv  # Balanced short datasets
├── models/                  # Model directory
│   ├── deepset-deberta/
│   ├── preambleai/
│   ├── qualifire/
│   └── protectaiv1/
│   └── protectaiv2/
└── results/                 # Evaluation results
    └── evaluation_results_*/

Quick Start

Step 1: Merge Datasets (Optional)

First, merge all datasets into a unified CSV format:

cd datasets
python merge_datasets.py

This will:

• Scan datasets/dataset-source/ for all supported formats
• Standardize column names and labels
• Convert labels to binary format (0=Safe, 1=Threat)
• Save unified dataset to datasets/merged_datasets.csv

Step 2: Evaluate Models

Run model evaluation using the merged dataset:

# Basic evaluation
# Default dataset: datasets/merged_datasets.csv, contains 20265 samples
# Default model: ./models/deepset-deberta/
python model-eval-v2.0.py --model ./models/deepset-deberta/

# With specific dataset (using short dataset for faster evaluation)
python model-eval-v2.0.py --model ./models/deepset-deberta/ --dataset datasets/merged_datasets_short_1000.csv

# Sample evaluation (1000 samples)
python model-eval-v2.0.py --model ./models/deepset-deberta/ --sample 1000

Model Evaluation Results Comparison

Evaluation Results on Full Dataset (20,265 samples)

Evaluation Results on 4,000 Sample Dataset

Evaluation Results on 3,000 Sample Dataset

Evaluation Results on 2,000 Sample Dataset

Evaluation Results on 1,000 Sample Dataset

Key Findings

Top Performers (Full Dataset - 20,265 samples):

1. Qualifire - Best overall performance with 94.57% accuracy and excellent balance of metrics
2. Vijil mBERT - Strong second with 93.38% accuracy and high precision
3. PreambleAI - High precision (96.64%) with good overall performance

Consistency Across Dataset Sizes:

• Qualifire consistently ranks #1 across all dataset sizes
• SavantAI show strong performance on smaller datasets
• ProtectAI v1 consistently shows low recall but high precision
• Deepset DeBERTa has very high recall but poor precision (high false positive rate)

Trade-offs:

• High Precision, Lower Recall: PreambleAI, Llama Prompt Guard, ProtectAI v1
• High Recall, Lower Precision: Deepset DeBERTa
• Balanced Performance: Qualifire, Vijil mBERT, SavantAI

Dataset Management

Supported Input Formats

• CSV: Comma-separated values
• TSV: Tab-separated values
• JSON: JavaScript Object Notation
• Parquet: Apache Parquet format

Available Datasets

The project includes one full merged dataset and three pre-generated short datasets with balanced distribution:

Dataset Overview

Source Distribution

merged_datasets.csv actually has 20265 samples, 231 samples are duplicates and are removed.

Label Distribution by Source (merged_datasets.csv)

Short Dataset Creation

The create_short_dataset.py script allows you to create balanced smaller datasets from the full merged dataset for faster testing and development:

Features

• Balanced Sampling: Maintains proportional representation from each source dataset
• Label Balance: Preserves label distribution within each source when possible
• Configurable Size: Specify target number of samples
• Reproducible: Uses random seed for consistent results
• Auto-naming: Generates descriptive filenames based on actual sample count

Usage

cd datasets

# Basic usage - creates 1000 samples by default
python create_short_dataset.py

# Custom sample size

python create_short_dataset.py --samples=1036
#python create_short_dataset.py --samples=2340
#python create_short_dataset.py --samples=3840


# Specify input and output files
python create_short_dataset.py --input merged_datasets.csv --output my_dataset.csv --samples 500

# With custom random seed
python create_short_dataset.py --samples 1000 --seed 123

Command Line Options

Example Output

Creating balanced short dataset with target of 1000 samples...
Loading dataset from: merged_datasets.csv
Loaded dataset with 20265 rows

=== Original Dataset Distribution ===
Label distribution:
  0 (Safe): 11,847 (58.4%)
  1 (Threat): 8,418 (41.6%)

=== Sampling Strategy ===
Target samples: 1000
Total sources: 9
Base samples per source: 111
Remainder to distribute: 1

Final short dataset saved to: merged_datasets_short_1000.csv

Dataset Structure

All datasets follow the same structure after merging:

Label Mapping

The merging process standardizes all labels to binary format:

• 0 (Safe): Normal, benign prompts
• 1 (Threat): Prompt injection attempts, jailbreaks, malicious prompts

Model Configuration

Supported Models

• Llama Prompt Guard 2 (86M): ./models/Llama-Prompt-Guard-2-86M/
• ProtectAI DeBERTa v3: ./models/protectaiv2/
• ProtectAI DeBERTa v1: ./models/protectaiv1/
• PreambleAI: ./models/preambleai/
• Qualifire: ./models/qualifire/
• SavantAI: ./models/testsavantai-prompt-injection-defender-large-v0/
• Deepset DeBERTa: ./models/deepset-deberta/
• Vijil mBERT: ./models/vijil-mbert-prompt-injection/
• YOUR OWN MODEL ...

Model Downloads

# Example model downloads using huggingface-cli
huggingface-cli download PreambleAI/prompt-injection-defense --local-dir ./models/preambleai/
huggingface-cli download protectai/deberta-v3-base-prompt-injection-v2 --local-dir ./models/protectaiv2/

Output Files

Results are saved in timestamped directories under results/:

results/evaluation_results_{model_name}_{timestamp}/
├── eval_results_{timestamp}.json      # Detailed results with predictions
├── eval_summary_{timestamp}.csv       # Flattened results for analysis
├── evaluation_report_{timestamp}.txt  # Human-readable report
└── evaluation_metrics_{timestamp}.json # Structured metrics

Evaluation Metrics

The tool provides comprehensive evaluation metrics:

• Accuracy: Overall classification accuracy
• Precision: Threat detection precision
• Recall: Threat detection recall (sensitivity)
• F1 Score: Harmonic mean of precision and recall
• False Positive Rate (FPR): Rate of false threat detection
• Confusion Matrix: Detailed classification breakdown

Advanced Usage

Custom Model Integration

To add your own model, create a directory under models/ and ensure it follows the HuggingFace transformers format.

Batch Evaluation

# Evaluate multiple models with the same dataset
 for model in deepset-deberta testsavantai-prompt-injection-defender-large-v0 Llama-Prompt-Guard-2-86M protectaiv1 protectaiv2 preambleai qualifire; do
    python model-eval-v2.0.py --model ./models/$model/ --dataset datasets/merged_datasets_short_2000.csv
done

Sampling Strategies

# Random sampling from full dataset
python model-eval-v2.0.py --model ./models/deepset-deberta/ --sample 500

# Use pre-created balanced short dataset
python model-eval-v2.0.py --model ./models/deepset-deberta/ --dataset datasets/merged_datasets_short_1000.csv

Version History

• v2.0: Enhanced evaluation pipeline with better output formatting
• v1.1: Added sampling support and improved metrics
• v1.0: Initial release with basic model evaluation

Model Sources

3.1 testsavantai/prompt-injection-defender-large-v0

https://huggingface.co/testsavantai/prompt-injection-defender-large-v0

3.2 Qualifire

https://huggingface.co/qualifire/prompt-injection-sentinel

3.3 ProtectAI deberta-v3-base-prompt-injection-v2

https://huggingface.co/protectai/deberta-v3-base-prompt-injection-v2

3.4 Meta Llama Prompt-Guard-86M

https://huggingface.co/meta-llama/Prompt-Guard-86M/tree/main

3.5 vijil/mbert-prompt-injection

https://huggingface.co/vijil/mbert-prompt-injection

3.6 PreambleAI/prompt-injection-defense

https://huggingface.co/PreambleAI/prompt-injection-defense/tree/main Training dataset not specified

Dataset Sources

4.1 Deepset

Qualifire used this dataset for training: https://huggingface.co/datasets/deepset/prompt-injections

Clone of this dataset used by testsavantai/prompt-injection-defender-large-v0: https://huggingface.co/datasets/JasperLS/prompt-injections

4.2 allenai/wildjailbreak

Qualifire used this dataset for training: https://huggingface.co/datasets/allenai/wildjailbreak

4.3 jackhhao/jailbreak-classification

ProtectAI and Qualifire used this dataset for training: https://huggingface.co/datasets/jackhhao/jailbreak-classification

4.4 qualifire/prompt-injections-benchmark

https://huggingface.co/datasets/qualifire/prompt-injections-benchmark

4.5 xTRam1/safe-guard-prompt-injection

https://huggingface.co/datasets/xTRam1/safe-guard-prompt-injection

4.6 rubend18/ChatGPT-Jailbreak-Prompts

ProtectAI and testsavantai/prompt-injection-defender-large-v0 used this dataset for training: https://huggingface.co/datasets/rubend18/ChatGPT-Jailbreak-Prompts

4.7 NVIDIA nvidia/Aegis-AI-Content-Safety-Dataset-2.0

Prisma AIRS used this dataset for training: https://huggingface.co/datasets/nvidia/Aegis-AI-Content-Safety-Dataset-2.0

4.8 Purple Llama CyberSecEval

Llama Prompt Guard used this dataset for training: https://github.com/meta-llama/PurpleLlama/tree/23156b70efb596831c02c6461fc42da1f75988ec/CybersecurityBenchmarks

License

This project is open source. Please check individual model licenses for their respective usage terms.