This dataset was generated for RAT-Bench, a comprehensive, multilingual benchmark for evaluating text anonymization tools.

• Github repo containing evaluation code & instructions;
• Leaderboard with rankings for existing tools;
• Paper containing benchmark construction details, experimental setup, and extensive results.

Dataset contents

This repository contains benchmark data in three languages:

• english
• serbian
• spanish
• dutch

Each language directory contains .json files named in the format:

• level_X.json

where X denotes the difficulty level of the generated text (higher levels correspond to more challenging anonymization settings).

File format

Each level_X.json file contains a list of benchmark examples for the corresponding language and difficulty level. (Refer to the paper and the evaluation repo for the exact schema and how each field is used in scoring.)

Data Fields

Each record contains the following fields:

• id: Unique record identifier.
• profile: All attributes and values leaked in the corresponding text.
• direct_identifiers: Attributes and values of direct identifiers leaked in the text.
• indirect_identifiers: Attributes and values of indirect identifiers leaked in the text.
• features: Names of the leaked attributes (used when computing re-identification risk).
• difficulty: Difficulty level of the text.
• prompt: Prompt used to generate the text.
• scenario: Scenario of the text.
• text: Generated text containing leaked identifiers at the specified difficulty level. This is the text to be anonymized by anonymization tools.