Data Extraction Pipeline

A multi-stage pipeline that extracts structured text, tables, figures, and charts from academic PDFs (born-digital, scanned, and hybrid) into a clean canonical JSON export suitable for downstream LLM consumption.

Features

• Forensics-driven routing: classifies each page (born-digital / scanned / hybrid, layout style, text quality) to choose the right render DPI + OCR strategy.
• Layout + reading order via PP-DocLayoutV3, MinerU 2.5, and olmOCR 2.
• Figure/table detection with caption linking, table structure recognition (Docling / Paddle cross-validated), and chart-to-data extraction (DePlot + VLM).
• Canonical resolver merges multiple text/OCR candidates with confidence scoring and optional VLM escalation, then a visual-verification pass validates tables, charts, and captions.
• Final clean export emits canonical JSON + CSV/Markdown sidecars.
• Resumable, parallel per-paper (CPU stages), with per-stage timeouts and retries.

Pipeline stages

Auto-discovered from pipeline/stages/ (run python run_pipeline.py --list-stages):

Requirements

• Python 3.10+
• Java 8+ (only if you run GROBID locally via the bundled service; stage 01 calls the GROBID HTTP API at http://localhost:8070 by default).
• CUDA-capable GPU for the VLM/GPU stages (05, 06, 06a, 07, 08a, 02a, 02b). CPU fallbacks exist for some stages; see pipeline/pipeline_config.json.

Install Python deps:

pip install -r requirements.txt

Configuration

The authoritative config is pipeline/pipeline_config.json. An annotated example with inline comments is at pipeline/pipeline_config.example.jsonc. For every config key, see docs/PIPELINE_CONFIG.md.

Key runtime config:

• grobid.url — GROBID service URL (default http://localhost:8070).
• vlm_manager.server_host — Qwen3.5-4B VLM server URL (default http://localhost:8000, served by llama-cpp-python). The model downloads automatically via HuggingFace Hub on first use (cache_dir: null => ~/.cache).
• resolver.vlm_model, verifier_vlm.* — VLM models for escalation/verification.

No secrets are required. The default server_api_key is the literal "not-needed" placeholder.

Usage

# Run the full pipeline on a directory of PDFs
python run_pipeline.py --input_dir corpus

# Run a subset of stages
python run_pipeline.py --input_dir corpus --only-stages 00,01,01a
python run_pipeline.py --input_dir corpus --from-stage 02

# Resume after a crash / re-run failures
python run_pipeline.py --input_dir corpus --resume --max-workers 4
python run_pipeline.py --input_dir corpus --failed-only

# Force a stage to re-run
python run_pipeline.py --input_dir corpus --only-stages 02_layout_anchor --force

# List discovered stages or dump resolved config
python run_pipeline.py --list-stages
python run_pipeline.py --print-config

A demo PDF is included in corpus/ so you can do a quick smoke test:

python run_pipeline.py --input_dir corpus --to-stage 01a --continue-on-error

Outputs are written to runs/<run_id>/ (one subdirectory per paper, with per-stage out/, err/, and a DONE marker; a manifest.json tracks status).

Standalone assets helper

generate_assets.py re-implements stages 01a/02/03 inline (no pipeline import) and is useful to regenerate just the rendered pages + layout + figures/tables:

python generate_assets.py --input_dir corpus --run_id myrun --force

Project layout

run_pipeline.py            # pipeline entrypoint (auto-discovers stages)
generate_assets.py         # standalone assets regenerator (01a/02/03)
requirements.txt
pipeline/
  core/                    # config loader, manifest, paths, stage loader/runner, util
  stages/                  # one file per stage (auto-discovered by STAGE dict + run_stage())
  tools/                   # OCR/chart/VLM helper modules, model loader, model cache
  pipeline_config.json     # authoritative runtime config
  pipeline_config.example.jsonc
docs/
  PIPELINE_CONFIG.md
  architecture_overview.txt
  architecture_details.txt
corpus/                    # demo input PDFs (one sample included)

Notes / Limitations

• Stages 02a, 02b, 05, 06, 06a, 07, 08a require a GPU and are excluded from paper parallelism (see --max-workers). CPU-only stages parallelize safely.
• The pipeline expects a running GROBID service for stage 01 and a running Qwen3.5-4B VLM server (llama-cpp-python) for the VLM stages; both download models on first use.