Why REDACT?

REDACT is a multilingual benchmark for evaluating character-span detection and canonical PII type classification under controlled variation. Rather than treating PII detection as a single aggregate score, REDACT is designed to expose where detectors succeed and fail across language, entity type, disclosure form, and policy-sensitivity slices.

At a glance

Benchmark results

All five systems were evaluated on the same locked, language-stratified 1,000-record split with 40 records per language. Partial-overlap micro-F1 is the primary aggregate metric.

REDACT also reports two gold-stratified diagnostic views:

• F5: recall by disclosure form
• F6: recall by sensitivity tier

These slices reveal detector-specific failure structure that aggregate F1 can hide. Precision and F1 remain the primary metrics for aggregate, per-entity-type, and per-language evaluation.

[!NOTE] The obfuscated disclosure-form slice contains only 15 gold entities in the locked evaluation split, so those recall values should be interpreted descriptively.

The canonical camera-ready detector report is reports/11_stratified_f1_v2.json.

What is released?

The 1,000-record subset is the official evaluation split used for the five reported detector systems. The remaining 12,427 records are also released as part of the benchmark corpus.

The large JSON files are stored with Git LFS. The Hugging Face mirror is usually the easiest way to download the corpus.

What does a REDACT annotation contain?

The headline task is character-span detection plus classification into one of 51 canonical entity types.

Each gold entity additionally carries:

These metadata fields are gold analysis attributes. Systems are not required to predict them in the headline detector experiment.

Example entity annotation:

{
  "entity_type": "Full_Name",
  "entity_string": "María García",
  "start": 19,
  "end": 31,
  "mention_index": 0,
  "disclosed": true,
  "disclosure_form": "complete",
  "sensitivity_tier": "MEDIUM"
}

Sensitivity tiers are operational benchmark labels for controlled analysis. They should not be interpreted as universal re-identification-risk rankings or legal advice.

Construction pipeline

REDACT controls nine record-level axes: domain, format, difficulty, length, density, code-switching, language, adjacency, and co-occurrence pattern.

flowchart LR
    A["Sampling<br/>9 controlled axes"] --> B["Generation<br/>16 hard rules"]
    B --> C["Deterministic<br/>span alignment"]
    C --> D["Conditional<br/>verification"]
    D --> E["Repair and<br/>normalization"]
    E --> F["Near-duplicate<br/>filtering"]
    F --> G["Release<br/>audits"]
    G --> H["13,427<br/>records"]

A strength-2 covering array produces 5,510 configurations, and paired-sweep mode contributes 890 additional configurations, giving 6,400 planned configurations per run.

6,400 configurations/run x 3 planned runs
        = 19,200 planned generation slots
                    |
                    v
             14,285 loaded
                    |
        - 814 structurally invalid records
                    |
                    v
             13,471 records
                    |
        - 44 near duplicates
                    |
                    v
             13,427 released

Generation uses 16 hard rules. The prompt artifact contains 10 core few-shot anchors plus one paired-sweep-specific example. Historical generation artifacts are preserved for provenance; publication-facing definitions in this README and data/schema.md should be treated as the current benchmark documentation.

See generation/README.md for the reproducibility overview.

Quality assurance

The release combines deterministic checks, model-based assessment, and human validation.

The G-Eval quality panel uses three model families: GPT-5.2, Claude Sonnet 4.6, and Gemini 2.5 Pro.

Evaluation protocol

See evaluation/README.md for the full scoring and reproducibility protocol.

Three span-matching modes are supported:

Reported analyses include:

• aggregate micro-F1 and macro-F1
• per-entity-type F1
• per-language F1
• F5 recall by disclosure form
• F6 recall by sensitivity tier

Quick start

Clone with Git LFS

git lfs install
git clone https://github.com/guneeshvats/REDACT-PII-Benchmark.git
cd REDACT-PII-Benchmark
git lfs pull

Or download the corpus from Hugging Face after accepting the dataset terms.

Load the corpus

import json

with open("data/pii_benchmark_full.json", encoding="utf-8") as f:
    records = json.load(f)

print(len(records))  # 13427

record = records[0]
print(record["axes"]["language"])
print(record["entities"][0])

Languages

REDACT contains 25 language variants across 9 scripts:

AR · CS · DA · DE · EN · ES · FI · FR · FR_CA · HE · HI · HU · IT · JA · KO · NL · NO · PT_BR · PT_EU · RU · SV · TH · TR · ZH_CN · ZH_TW

Each language contributes 453-621 records to the released corpus.

Repository map

.
├── README.md
├── LICENSE
├── DATASET_TERMS.md
├── data/
│   ├── pii_benchmark_full.json
│   ├── pii_benchmark_sample1000.json
│   └── schema.md
├── catalog/
│   ├── PII_Pattern_Catalog_v4_MASTER.md
│   ├── pattern_seed_data_v4.json
│   └── co_to_v4_mapping.json
├── generation/
│   ├── README.md
│   ├── prompts/
│   ├── samplers/
│   ├── seed/
│   └── sygra_graph/
├── evaluation/
│   ├── README.md
│   ├── scripts/
│   ├── judge/
│   └── manual_validation/
├── docs/
│   ├── methodology.md
│   ├── metrics_reference.md
│   ├── generation_guide.md
│   ├── runbook.md
│   └── PROVENANCE.md
└── reports/
    ├── README.md
    ├── 06_stats.json
    └── 11_stratified_f1_v2.json

Licensing and intended use

The repository uses separate terms for code and data:

• Code and general repository documentation: MIT License, see LICENSE.
• Dataset artifacts and pattern catalog: REDACT Dataset Terms, see DATASET_TERMS.md.

The dataset is intended for research, evaluation, auditing, education, and defensive privacy work. The corpus contains synthetic identifiers rather than real personal data, but some formats are deliberately realistic because detector evaluation requires structural plausibility.

Citation

If you use REDACT, please cite:

@inproceedings{vats2026redact,
  title     = {REDACT: A Systematically Controlled Multilingual Benchmark for Personal Information Detection},
  author    = {Guneesh Vats and Anubha Agrawal and Shikha Singhal and Ajita Dash and Praison Selvaraj and Vidhan Jhawar and Ranga Prasad Chenna and Bharadwaj Y M G},
  booktitle = {Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track},
  year      = {2026},
  note      = {To appear}
}

Preprint: arXiv:2606.19881

Authors

Guneesh Vats · Anubha Agrawal · Shikha Singhal · Ajita Dash · Praison Selvaraj · Vidhan Jhawar · Ranga Prasad Chenna · Bharadwaj Y M G