PasteProof PII Detector v3

A fine-tuned ModernBERT model for detecting Personally Identifiable Information (PII) in text before it's accidentally leaked.

Model Description

This model performs Named Entity Recognition (NER) to identify 27 types of sensitive information commonly found in code, configuration files, chat messages, and documents.

Base model: answerdotai/ModernBERT-base
Task: Token Classification (NER)
Training data: 150K synthetic examples with intentional variation to prevent overfitting
F1 Score: 0.97 on held-out test set

Detected Entity Types

API Key Formats Detected

Stripe (sk_live_, pk_test_), OpenAI (sk-), Anthropic (sk-ant-), GitHub (ghp_, github_pat_), Slack (xoxb-), AWS (AKIA...), SendGrid (SG.), and generic patterns.

Intended Uses

✅ Primary use case: Browser extension or IDE plugin that warns users before pasting sensitive data into web forms, chat apps, or public repositories.

✅ Other valid uses:

• Pre-commit hooks to scan for secrets
• Chat/support systems to redact PII before logging
• Document scanning for compliance (HIPAA, GDPR, PCI-DSS)
• Data pipeline filtering

Limitations

⚠️ Not a security guarantee. This model is a helpful tool, not a replacement for proper secrets management.

⚠️ False negatives are possible. The model may miss:

• Obfuscated or encoded secrets
• PII in languages other than English
• Novel API key formats not in training data
• Context-dependent PII (e.g., a name that's only sensitive in certain contexts)

⚠️ False positives may occur on:

• UUIDs and transaction IDs that resemble API keys
• Test/example data that looks real
• Random strings in code

⚠️ Known limitations:

• Trained on synthetic data, may not perfectly match real-world distribution
• 512 token context limit (longer texts should be chunked)
• English-only training data

How to Use

from transformers import pipeline

detector = pipeline(
    "token-classification",
    model="joneauxedgar/pasteproof-pii-detector-v3",
    aggregation_strategy="simple"
)

text = '''const config = {
    apiKey: "sk_live_abc123def456",
    email: "[email protected]"
};'''

results = detector(text)
for entity in results:
    print(f"{entity['entity_group']}: {entity['word']} ({entity['score']:.1%})")

Output:

API_KEY: sk_live_abc123def456 (99.9%)
EMAIL: [email protected] (99.9%)

Training Details

• Dataset: joneauxedgar/pasteproof-pii-dataset-v3
• Training examples: 120,000
• Validation examples: 15,000
• Epochs: 3
• Learning rate: 5e-5
• Batch size: 32
• Hardware: NVIDIA A100

Training Data Design

To prevent overfitting, the training data includes:

• 30% misleading key names (e.g., data, x, field1 instead of apiKey)
• 20% raw PII without contextual clues
• 15% mixed real + fake data (real PII alongside test cards, example.com emails)
• 15% hard negatives (placeholder values, test data that should NOT be flagged)

Evaluation Results

The model correctly ignores:

• [email protected] (placeholder email)
• 4111111111111111 (test credit card)
• 123-45-6789 (example SSN)
• process.env.API_KEY (reference, not a value)

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 32
• eval_batch_size: 64
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• num_epochs: 3
• mixed_precision_training: Native AMP

Training results

Framework versions

• Transformers 4.57.3
• Pytorch 2.9.0+cu126
• Datasets 4.0.0
• Tokenizers 0.22.1

Citation

@model{pasteproof_pii_detector,
  author = {Jonathan Edgar},
  title = {PasteProof PII Detector},
  year = {2025},
  publisher = {Hugging Face},
  url = {https://huggingface.co/joneauxedgar/pasteproof-pii-detector-v3}
}

Contact

Part of the PasteProof project - preventing accidental data leaks before they happen.

License

BSL 1.0 Use for individual or research purposes. Reach out [email protected] for commercial license or usage questions.