PII Redaction Service

A PII detection and redaction microservice for finblade.ai. Accepts a document via REST API, detects personally identifiable information, and returns a redacted file with an audit report — layout preserved, underlying text genuinely removed. This service ships the English detector.

Status: English serving MVP complete — containerised, air-gappable, with HTTP upload/download and path-based redaction, plus optional GPU acceleration (CPU by default). Detection quality (precision/recall/F1) is measured by the eval harness (eval/, report at eval/reports/report.md); see docs/plans/2026-06-07-post-mvp-evaluation-and-next-steps.md.

Build status (Track 1 chunks)

---

Overview

POST /redact  {"input_path": "/data/in/doc.pdf", "output_path": "/data/out/doc.pdf"}
      │
      ▼
 EXTRACT ──► DETECT ──► REDACT
 (PDF/DOCX    (Presidio:   (true in-place:
  → text +     regex +      glyphs removed,
  positions)   NER model)   not just boxed)
      │
      ▼
 <output_path>                 # redacted file at the caller-chosen path
 <output_path stem>.report.json   # JSON audit report written alongside it

Two integration styles are supported:

• By path (above): POST /redact accepts any input/output file path the caller supplies (the deployment volume layout isn't known ahead of time) — best when caller and service share a volume. output_path is optional (omitted → media/output/<input filename>); a bare filename (resolved under media/input/ → media/output/) is still accepted as a convenience. Confine paths to a fixed set of directories with PII_ENFORCE_PATH_ROOTS=true (+ optional PII_ALLOWED_ROOTS).
• By upload (no shared filesystem): POST /redact/upload takes the file(s) over HTTP and returns a report + download_url per file — the location-independent contract for a microservice mesh (see Quickstart).

Detection layers

• Regex recognizers — structured PII: email, phone, IBAN, credit card, Saudi National ID / Iqama, passport
• NER model — contextual PII: names, addresses, locations (English)

---

Stack

---

Quickstart

# Local
pip install -r requirements.txt
python -m spacy download en_core_web_sm
uvicorn app.main:app --reload

# Docker (first: cp .env.example .env — compose needs the file present)
docker compose up --build

GPU (optional): the default docker compose build is CPU-only and never requires a GPU. GPU acceleration with automatic CPU fallback is built in (PII_DEVICE=auto, the default): the recognizers load their models on CUDA when a CUDA-enabled torch build and a GPU are present, else CPU. The CPU image bundles the CPU torch wheel, so on it auto resolves to CPU. For a GPU image, build with the cu124 override:

docker compose -f docker-compose.yml -f docker-compose.gpu.yml up --build

This installs the CUDA torch wheel (larger image) and reserves the host GPU(s). The host needs an NVIDIA driver new enough for CUDA 12.4 (≈550+) and the NVIDIA Container Toolkit. Force the device explicitly with PII_DEVICE=cpu|cuda. See docs/plans/track-1-serving-mvp/06-packaging-and-e2e.md.

# Redact a file by path — input and output can be any path the container can reach
curl -X POST http://localhost:8000/redact \
     -H "Content-Type: application/json" \
     -d '{"input_path": "/app/media/input/contract.pdf", "output_path": "/app/media/output/contract.pdf"}'

# Or use the media-folder convenience: drop the file in media/input/ and pass a bare filename
curl -X POST http://localhost:8000/redact \
     -H "Content-Type: application/json" \
     -d '{"filename": "contract.pdf"}'   # → media/output/contract.pdf

No shared filesystem? Upload over HTTP and fetch the result back by URL. This is the location-independent contract for a microservice mesh — the caller never needs to know (or share) this service's volume layout. One request can carry multiple files (one result per file):

# Upload one or more files; get a report + download URL per file
curl -X POST http://localhost:8000/redact/upload \
     -F "[email protected]" -F "[email protected]"
# → {"results":[{"input":"contract.pdf","status":"ok",
#                "download_url":"/redact/result/<token>/contract.pdf",
#                "report_url":"/redact/result/<token>/contract.report.json",
#                "entity_counts":{...},"total_detections":12, ...}, ...]}

# Then fetch the redacted file (and the JSON audit report) back
curl -O http://localhost:8000/redact/result/<token>/contract.pdf
curl    http://localhost:8000/redact/result/<token>/contract.report.json

Redacted results are written to a temp store (PII_RESULTS_DIR) and swept after PII_RESULT_TTL_SECONDS (default 1h). Upload size is capped by PII_MAX_UPLOAD_MB (default 50). Set PII_PUBLIC_BASE_URL to return absolute download URLs behind a reverse proxy.

# Tests
pytest            # fast suite — no network/model download (models are stubbed)
pytest -m slow    # real-model E2E tests against spaCy + GLiNER2 (no stubs)

Note: the pytest suite proves the pipeline's machinery (glyphs deleted, reports PII-safe, format-blind). Detection quality (precision/recall/F1) is measured separately by the eval harness under eval/ (report: eval/reports/report.md); next steps in docs/plans/2026-06-07-post-mvp-evaluation-and-next-steps.md.

---

Running with Docker

End-to-end: build the image, start the service, redact files over HTTP.

Prerequisite — create a .env: docker compose reads env_file: .env, so the file must exist. Run cp .env.example .env once. The template's defaults work as-is and serving needs no secrets or tokens — the models are public and baked into the image. (Building with docker compose build and running via docker run don't strictly need it; docker compose up does.)

1 — Build the image

# CPU image (default)            → tag: icsarabia-pii-redactor:latest
docker compose build

# GPU image (CUDA 12.4)          → tag: icsarabia-pii-redactor:cu124
docker compose -f docker-compose.yml -f docker-compose.gpu.yml build

Both bake the spaCy + GLiNER2 + nvidia/gliner-PII weights into the image (air-gappable — no network needed at serve time). The GPU image adds the CUDA torch wheel (~15 GB vs ~8 GB) and, on a GPU host, uses the GPU automatically (PII_DEVICE=auto), falling back to CPU otherwise.

2 — Start the service

docker compose up -d                                                   # CPU
docker compose -f docker-compose.yml -f docker-compose.gpu.yml up -d   # GPU

• First start loads three models, so warmup takes ~2–5 min — wait until GET /health returns {"status":"ok"}. (The container may report unhealthy during that window; that's just the healthcheck's startup grace, not a failure.)
• Needs ~6 GB RAM (both NER models). GPU hosts also need the NVIDIA driver (≥ ~550 for CUDA 12.4) plus the NVIDIA Container Toolkit.

curl http://localhost:8000/health     # {"status":"ok"} when ready
#   …or open the interactive API docs: http://localhost:8000/docs

3 — Redact over HTTP (URL workflow — no shared volume)

The location-independent path for a microservice mesh: POST the file(s), get back a per-file report plus a download_url, then GET the redacted file. One request may carry many files.

# (a) upload one or more files
curl -s -X POST http://localhost:8000/redact/upload \
     -F "[email protected]" -F "[email protected]"

{ "results": [
  { "input": "contract.pdf", "status": "ok", "total_detections": 12,
    "entity_counts": { "PERSON": 2, "EMAIL_ADDRESS": 1, "IBAN_CODE": 1 },
    "download_url": "/redact/result/<token>/contract.pdf",
    "report_url":   "/redact/result/<token>/contract.report.json" }
] }

# (b) fetch the redacted file + JSON audit report using the returned URLs
curl -O http://localhost:8000/redact/result/<token>/contract.pdf
curl    http://localhost:8000/redact/result/<token>/contract.report.json

The <token> is generated by the server per request — read it from the response, don't construct it. Results live in a temp store (PII_RESULTS_DIR) and are swept after PII_RESULT_TTL_SECONDS (default 1h); upload size is capped by PII_MAX_UPLOAD_MB (default 50); set PII_PUBLIC_BASE_URL to emit absolute URLs behind a reverse proxy. A per-file status: "error" (e.g. unsupported type) never fails the rest of the batch.

4 — Redact by path (shared-volume alternative)

If the caller shares a filesystem with the service, pass paths instead of uploading:

curl -X POST http://localhost:8000/redact -H "Content-Type: application/json" \
     -d '{"input_path": "/data/in/contract.pdf", "output_path": "/data/out/contract.pdf"}'

# bare filename convenience (media/input → media/output):
curl -X POST http://localhost:8000/redact -H "Content-Type: application/json" \
     -d '{"filename": "contract.pdf"}'

Mount the caller's directories into the container (compose volumes: or -v host:container). Set PII_ENFORCE_PATH_ROOTS=true (+ PII_ALLOWED_ROOTS) to confine accepted paths to a fixed set of roots.

5 — Ship the image

docker save icsarabia-pii-redactor:cu124 -o pii-redactor-cu124.tar   # load on the host, or push to a registry

---

Project Structure

app/
├── extractors/     # PDF (PyMuPDF) + DOCX text extraction
├── detectors/      # Presidio engine, regex recognizers, NER recognizer
├── redactors/      # True in-place redaction
├── core/           # Config, pipeline orchestrator
└── api/            # Routes, request/response schemas

media/input/        # Drop input files here
media/output/       # Redacted files + JSON audit reports