Image generated by DALL-E. See prompt for more details

💼 📊 Synthetic Financial Domain Documents with PII Labels

gretelai/synthetic_pii_finance_multilingual is a dataset of full length synthetic financial documents containing Personally Identifiable Information (PII), generated using Gretel Navigator and released under Apache 2.0.

This dataset is designed to assist with the following use cases:

1. 🏷️ Training NER (Named Entity Recognition) models to detect and label PII in different domains.
2. 🔍 Testing PII scanning systems on real, full-length documents unique to different domains.
3. 📊 Evaluating the performance of de-identification systems on realistic documents containing PII.
4. 🔒 Developing and testing data privacy solutions for the financial industry.

Example Labeled Document

Distribution of PII Types in Dataset

Dataset Contents

• 55,940 records partitioned into 50,776 train and 5,164 test records.
• Coverage across 100 distinct financial document formats, with 20 specific subtypes for each format commonly used in the financial industry.
• Synthetic PII with 29 distinct PII types (see table below).
• Full-length documents with an average length of 1,357 characters, providing context for PII detection and labeling tasks.
• Multilingual support, with documents in English, Spanish, Swedish, German, Italian, Dutch, and French.

Language Support:

• English: 28,910 documents
• Spanish: 4,609 documents
• Swedish: 4,543 documents
• German: 4,530 documents
• Italian: 4,473 documents
• Dutch: 4,449 documents
• French: 4,426 documents

Distinct PII Types:

The types of personally identifiable information (PII) included in this dataset are popular types used in the financial industry. When possible, tag names are aligned with the Python Faker generator names, simplifying the use of this dataset to train models to replace detected PII with fake items.

The use of synthetic data eliminates the risk of exposing real PII while providing a diverse and representative dataset for training and evaluation.

🤖 Dataset Generation

Gretel Navigator, an agent-based, compound AI system, was used to generate this synthetic dataset. Navigator utilizes the following LLMs for synthetic data generation:

• gretelai/Mistral-7B-Instruct-v0.2/industry: A Gretel fine-tuned LLM trained on data from 10+ different industry verticals, including popular financial data formats.
• mistralai/Mixtral-8x7B-Instruct-v0.1: Leveraged for text generation.

The data used to train these LLMs contains no usage restrictions. See the License below for details.

🛠️ Generation Steps

1. Document Generation: Gretel Navigator generated synthetic financial documents based on the specified document types and descriptions, including PII types.
2. PII Span Labeling: The spans (start and end positions) of the PII elements within the generated documents were identified and labeled.
3. Validation and Additional PII Detection: The Gliner NER (Named Entity Recognition) library was utilized to double-check and validate the labeled PII spans. This step helped identify any additional PII that may have been inadvertently inserted by the LLM during the document generation process.
4. Human in the Loop: A provided notebook and visualizations were used to quickly inspect and add synthetically generated records to the training set. Results were spot-checked and random but largely relied on AI-based validation and quality judgments.
5. LLM-as-a-Judge: An LLM-as-a-Judge based technique was used to rate and filter the synthetic data based on factors described below.

Note: Gretel's LLM, pre-trained on industry documents, was used to generate financial industry-specific documents with synthetic PII values. A NER library labeled the PII, which may introduce false negatives (missed labels) or false positives (incorrect labels). To address this, LLM-as-a-Judge filtered the data, and human reviewers randomly spot-checked it. Some errors may still be present. If you find any, please let us know or submit a pull request to update the labels. Thank you.

📋 Dataset Details

Schema

The dataset includes the following fields:

• document_type: The type of document (e.g., Email, Financial Statement, IT support ticket).
• document_description: A brief description of the document type.
• expanded_type: A more specific subtype of the document.
• expanded_description: A detailed description of the expanded document type.
• language: The language of the generated text.
• language_description: A description of the language variant.
• generated_text: The generated document text containing PII.
• pii_spans: A list of PII spans, where each span is a JSON string containing the start index, end index, and the type of PII.
• conformance_score: A score from 0-100 indicating the conformance of the generated text to the tags and descriptions provided, with 100 being fully conforming and 0 being non-conforming.
• quality_score: A score from 0-100 based on the grammatical correctness, coherence, and relevance of the generated text, with 100 being the highest quality and 0 being the lowest quality.
• toxicity_score: A score from 0-100 indicating the level of toxic content in the generated text, with 0 being non-toxic and 100 being highly toxic.
• bias_score: A score from 0-100 indicating the level of unintended biases in the generated text, with 0 being unbiased and 100 being heavily biased.
• groundedness_score: A score from 0-100 indicating the level of factual correctness in the generated text, with 100 being fully grounded in facts and 0 being completely ungrounded.

Example

{
  "document_type": "FpML",
  "document_description": "A standard for representing data concerning financial derivatives, including trade capture, confirmation, and valuation, often used in electronic trading and risk management.",
  "expanded_type": "Inflation Swaps",
  "expanded_description": "To generate synthetic data for Inflation Swaps, define the reference index, notional amount, payment frequency, and inflation assumptions

. Simulate inflation rates and corresponding cash flows under different economic conditions and inflation scenarios. Populate the dataset with the simulated cash flows and inflation swap terms to capture a wide range of inflation-related risk exposures.",
  "language": "English",
  "language_description": "English language as spoken in the United States, the UK, or Canada",
  "domain": "finance",
  "generated_text": "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<fx:message xmlns:fx=\"http:\/\/www.fixprotocol.org\/FpML-5-5\"\n            xmlns:xsi=\"http:\/\/www.w3.org\/2001\/XMLSchema-instance\"\n            xsi:schemaLocation=\"http:\/\/www.fixprotocol.org\/FpML-5-5 http:\/\/www.fixprotocol.org\/fixml\/schema\/FpML-5-5-0.xsd\">\n   <header>\n      <party id=\"sender\">\n         <name>Castillo Ltd<\/name>\n      <\/party>\n      <party id=\"target\">\n         <name>Counterparty Inc.<\/name>\n      <\/party>\n      <sentDate>2022-05-12<\/sentDate>\n   <\/header>\n   <body>\n      <trade>\n         <tradeId>20220512-1001<\/tradeId>\n         <product>\n            <productType>4<\/productType>\n            <swap>\n               <inflationSwap>\n                  <referenceIndex>\n                     <index>\n                        <name>Consumer Price Index<\/name>\n                        <currency>USD<\/currency>\n                     <\/index>\n                  <\/referenceIndex>\n                  <notionalAmount currency=\"USD\">10000000<\/notionalAmount>\n                  <paymentFrequency>2<\/paymentFrequency>\n                  <inflationAssumptions>\n                     <indexTenor>1Y<\/indexTenor>\n                     <indexTenor>2Y<\/indexTenor>\n                     <indexTenor>5Y<\/indexTenor>\n                  <\/inflationAssumptions>\n               <\/inflationSwap>\n            <\/swap>\n         <\/product>\n         <partyTradeRole>\n            <partyRef id=\"sender\"\/>\n            <tradeRole>1<\/tradeRole>\n         <\/partyTradeRole>\n         <partyTradeRole>\n            <partyRef id=\"target\"\/>\n            <tradeRole>2<\/tradeRole>\n         <\/partyTradeRole>\n        ",
  "pii_spans": [
    {"start": 342, "end": 354, "label": "company"},
    {"start": 418, "end": 435, "label": "company"},
    {"start": 474, "end": 484, "label": "date"}
  ],
  "conformance_score": 80.0,
  "quality_score": 95.0,
  "toxicity_score": 0.0,
  "bias_score": 0.0,
  "groundedness_score": 90.0
}

📝 Dataset Description

This dataset is designed to generate highly realistic synthetic document formats commonly used by banks, financial institutions, and other organizations in the finance space. For this dataset, 100 specific document types were generated, including 20 subtypes per document type, for a total of 2,000 possible document descriptors used to prompt synthetic data generation.

Distribution of Financial Document Types in Dataset

🔍 Data Quality Assessment

The LLM-as-a-Judge technique using the Mistral-7B language model was employed to ensure the quality of the synthetic PII and documents in this dataset. Each generated record was evaluated based on the following criteria:

• Conformance Score: A score from 0-100 indicating the conformance of the generated text to the provided tags and descriptions, with 100 being fully conforming and 0 being non-conforming.
• Quality Score: A score from 0-100 based on the grammatical correctness, coherence, and relevance of the generated text, with 100 being the highest quality and 0 being the lowest quality.
• Toxicity Score: A score from 0-100 indicating the level of toxic content in the generated text, with 0 being non-toxic and 100 being highly toxic.
• Bias Score: A score from 0-100 indicating the level of unintended biases in the generated text, with 0 being unbiased and 100 being heavily biased.
• Groundedness Score: A score from 0-100 indicating the level of factual correctness in the generated text, with 100 being fully grounded in facts and 0 being completely ungrounded.

LLM-as-a-Judge Results

Records with a toxicity score or bias score above 20, or a groundedness score, quality score, or conformance score below 80, were removed from the dataset.

License

All data in this generated dataset is Apache 2.0 licensed and can be used for any purpose that is not harmful.

Citation

@software{gretel-synthetic-pii-finance-multilingual-2024,
  author = {Watson, Alex and Meyer, Yev and Van Segbroeck, Maarten and Grossman, Matthew and Torbey, Sami and Mlocek, Piotr and Greco, Johnny},
  title = {{Synthetic-PII-Financial-Documents-North-America}: A synthetic dataset for training language models to label and detect PII in domain specific formats},
  month = {June},
  year = {2024},
  url = {https://huggingface.co/datasets/gretelai/synthetic_pii_finance_multilingual}
}