PDF Document Processing System

A sophisticated PDF document processing system that leverages AI-powered extraction to convert unstructured PDF documents into structured data. Built with modern Python technologies and designed for enterprise document processing workflows.

Key Features

AI-Powered Data Extraction

• Llama Cloud Integration: Advanced AI service for intelligent document understanding
• Local GLM-OCR Integration: Schema-directed extraction through an administrator-managed Ollama server, with optional complete-document resolution across ordered PDF pages
• Local LiteParse + Ollama Integration: Embedded PDF text or local OCR feeds a configurable structured extraction model, defaulting to qwen3.5:9b-q4_K_M, without sending PDF content to a cloud service
• Structured Data Extraction: Converts PDF content into structured JSON/CSV format
• Multi-format Support: Handles invoices, receipts, forms, and various document types
• Typed Scalar Lists: Normalize lists of text, integers, numbers, and yes/no values
• Typed Object Support: Extract fixed flat objects whose properties mix text, integer, number, and yes/no values
• Array-of-Objects Support: Extract line items and tabular data from complex documents

Multiple Input Methods

• Watch Folder Monitoring: Automated processing of dropped PDF files
• Web Upload Interface: User-friendly web portal for manual uploads
• Upload Progress Feedback: Batch uploads show aggregate browser-to-server transfer progress before processing begins
• Operator UI Guidance: Keyboard-friendly upload controls, cancellable transfers, retry actions, consistent queue tables, and lightweight contextual help
• Polled Processing Status: Browser pages refresh workflow progress at short intervals
• Batch Uploads: /app/upload supports multi-file upload and creates batch/document records for tracking

Configurable Pipeline Architecture

• Modular Design: Pluggable processing steps (extraction, storage, archiving, rules)
• Prefect Workflow Orchestration: Reliable task execution and error handling
• Supervised Local Runtime: The parent process runs migrations and supervises the web and worker subprocesses; a single durable worker executes queued root workflows
• Persisted Review Resume: Human review pauses document work in SQLite; completion starts a new flow at the next task
• Versioned Configuration: Named SQLite pipeline templates with editable drafts, immutable published versions, and exact review-form dependencies
• Explicit Routing: Uploads select one exact published version and each watch-folder binding pins one exact version
• Deployment YAML: config.yaml owns paths, web/auth/logging settings, custom-task approvals, and secret aliases—not new workflow definitions
• Extensible Framework: Easy to add new processing steps
• SQLite-backed Workflow State: Batches, documents, task runs, extracted fields, review items, artifacts, settings, and audit history are persisted in SQLite
• Pipeline Retention Controls: Archive retired pipelines separately from previewed cleanup of finished processing history; protect active work and retain definitions, audit events, and physical files

Flexible Data Storage

• CSV Export: Structured data in spreadsheet format
• JSON Export: Hierarchical data preservation
• Table-Aware Storage: Row-per-item expansion for tabular data
• Local File Management: Organized storage with metadata tracking
• Artifact Registry: Source originals, split working files, archives, PDFs, CSVs, and JSON exports are registered as durable document files

Modern Web Interface

• FastAPI Backend: High-performance REST API
• Operator App: /app/upload, /app/processing, /app/review, /app/failures, and /app/reports
• Admin App: /app/admin, /app/admin/users, /app/admin/pipeline, /app/schemas, /app/admin/tasks, /app/settings/validation, and /app/admin/audit
• Review Workflows: Human review queues for low-confidence or policy-triggered extracted fields
• Citation-Aware PDF Review: Local PDF.js source viewing with provider-specific page and bounding-box navigation
• Responsive Design: Mobile-friendly interface

Table of Contents

• Quick Start
• Installation
• Configuration
• Usage
• API Reference
• Project Structure
• Documentation
• Contributing
• Testing
• Troubleshooting

Quick Start

Prerequisites

• Python 3.13+
• Node.js/npm for frontend asset rebuilds (not required when using committed assets)
• At least one extraction provider: Llama Cloud API access, or a separately installed Ollama server with the models configured by a GLM-OCR or LiteParse pipeline. LiteParse uses local OCR for scanned pages.

Installation

1. Clone the repository

   git clone <your-repo-url>
   cd pdfdoc_extraction
   

2. Install dependencies

   py -3.13 -m venv .venv
   .\.venv\Scripts\python.exe -m pip install --upgrade pip
   .\.venv\Scripts\python.exe -m pip install -r requirements-dev.txt
   npm ci
   

   npm ci installs the frontend build dependencies used when rebuilding committed CSS or browser assets. Node.js/npm is not required to run the application when those committed assets are already present.

3. Configure the system Open the ignored local config.yaml and set deployment paths, web/auth settings, and provider secret aliases. Do not place new tasks or pipeline blocks there. Create folders required by watch_folder.dir and web.upload_dir before startup.

4. Run the application

   .\.venv\Scripts\python.exe main.py
   

   main.py owns database migration when database.run_migrations_on_startup is enabled. Its web and worker children verify the resulting schema before starting.

5. Access the web interface Open your browser and navigate to http://localhost:8000/app/upload

Installation

System Requirements

• Operating System: Windows 11 is the primary supported development/runtime environment
• Python Version: 3.13+
• Memory and Storage: Size for the PDF volume, provider workload, retained artifacts, and SQLite database; no fixed production capacity is enforced by the application

Dependencies Installation

.\.venv\Scripts\python.exe -m pip install -r requirements.txt

requirements.txt contains pinned application dependencies. Contributors and test environments should install requirements-dev.txt, which includes the runtime set plus pytest, coverage, and browser-test tooling:

.\.venv\Scripts\python.exe -m pip install -r requirements-dev.txt

Key dependencies:

• FastAPI: Modern web framework for the API and web interface
• LlamaCloud SDK (llama-cloud): AI-powered document extraction through Extract v2
• Ollama Python client (ollama): Native local API access for GLM-OCR and its optional document resolver
• LiteParse (liteparse): Local PDF text, OCR and word-coordinate parsing for the LiteParse task; its adapter calls the loopback Ollama HTTP API
• PyMuPDF: In-memory PDF page rendering for local vision extraction
• PDF.js (pdfjs-dist): Vendored browser module and worker for source-PDF review
• Prefect: Workflow orchestration engine
• Pandas: Data manipulation and CSV processing
• Uvicorn: ASGI server for production deployment

Configuration

Basic Configuration Structure

# config.yaml
web:
  host: "127.0.0.1"
  port: 8000
  secret_key: "your_secret_key"
  upload_dir: "web_upload"

database:
  path: "data/app_state.sqlite3"
  run_migrations_on_startup: true

review:
  default_queue_name: "default_review"
  lock_timeout_minutes: 60

watch_folder:
  dir: "watch_folder"  # required startup-compatibility path, not pipeline routing
  validate_pdf_header: true
  processing_dir: "processing_folder_default"

pipeline_secrets:
  llamacloud-primary: "SET-ONLY-IN-THIS-LOCAL-IGNORED-FILE"

Create review forms and pipelines visually after startup: publish and activate a review-form version when schema-driven review is needed, add tasks to a pipeline draft, reference secrets as { "$secret": "llamacloud-primary" }, save, validate, publish, and activate the pipeline. config.yaml does not need top-level tasks or pipeline sections after migration.

The local GLM-OCR task is document-generic: each pipeline supplies its own field schema and semantic instructions. Complete-document resolution can reconcile scalar and flat-object values across multi-page PDFs and one configured logical table from structured page evidence. A bill of lading, insurance document, or invoice therefore uses the same task implementation but should use a dedicated pipeline and matching review form. Published definitions that omit resolution_mode retain legacy page-merge behavior; newly added GLM-OCR tasks in the visual editor default to complete-document resolution.

For text/layout parsing followed by local structured extraction, add Lite Parse Extract in Admin → Pipeline. Configure fields and document guidance, then add a matching Review Gate and JSON or CSV storage. The LiteParse task requires a downloaded local GGUF model and a credential-free HTTP loopback endpoint. It supplies no numeric confidence; retained word boxes are diagnostic and are not shown as verified review highlights. See the LiteParse setup guide and six-invoice validation report.

LiteParse can process suitable documents faster because a normal attempt uses one Ollama request, avoiding GLM-OCR's multiple per-page and resolver calls. Actual speed depends on document size, model choice, OCR, and available memory.

For larger documents whose complete evidence or item output cannot fit one local-model request, GLM-OCR is better positioned through page-based extraction and chunked table resolution. Its document-mode header resolver still receives all page images, so this is a scalability selection criterion rather than a guarantee of unlimited size or better accuracy. See the document-size selection criteria.

Output folders are task-owned in the exact SQLite pipeline version. CSV/JSON tasks use data_dir, local PDF storage uses files_dir, archive tasks use archive_dir, and split tasks use split_dir. CSV/JSON/local-PDF/split tasks create their output directory when execution begins if permissions allow; ArchivePdfTask requires its configured archive directory to exist. Deployment _dir keys are the paths that ConfigManager creates at startup, except watch_folder.dir.

The current runtime uses the llama-cloud SDK and LlamaCloud client. New code should not use the legacy llama-cloud-services / LlamaExtract agent flow. agent_id is legacy; use configuration_id for saved Extract v2 configurations or omit it to build an inline schema from fields.

Manual LlamaCloud Smoke Check

After validating a saved Extract v2 configuration in the LlamaCloud UI, you can run a one-file SDK and workflow fit check against a local PDF. Sample PDF documents are not included in the GitHub repository; provide your own test document or use a PDF from your existing document set. The manual tool reads a YAML task definition; it does not read SQLite or resolve pipeline_secrets. Export/copy the selected pipeline definition to an ignored local smoke file and provide the API key through the current process:

$env:LLAMA_CLOUD_API_KEY = "set-locally"
$pdfPath = "C:\path\to\your\test-document.pdf"
.\.venv\Scripts\python.exe tools\llamacloud_extract_smoke.py --config smoke-pipeline.yaml --file $pdfPath --configuration-id "cfg-..."

If configuration_id is already set in the smoke definition, omit the override flag. Do not commit the smoke definition or output because they can contain customer/provider data.

The script writes:

• raw_extract_result.json: raw result.extract_result from LlamaCloud.
• workflow_normalized_data.json: data after matching returned keys to workflow fields and applying configured types.
• workflow_fit_report.json: missing workflow fields, extra raw keys, validation errors, and pass/fail status.

To re-check a saved raw result without another LlamaCloud call:

.\.venv\Scripts\python.exe tools\llamacloud_extract_smoke.py --config smoke-pipeline.yaml --raw-json test\data\llamacloud_smoke\raw_extract_result.json

Environment Variables

• CONFIG_PATH: Custom path to configuration file
• USE_RELOAD: Enable auto-reload for development (true/false)

Runtime State

SQLite is the primary workflow-state store. The application records:

• ingestion batches and documents
• task-run lifecycle and errors, plus document status transition history
• extraction results and field-level confidence/review state
• review queue items and decisions
• registered document artifacts
• admin settings versions and audit events

Workflow state is persisted in SQLite. /api/files and /api/status/{file_id} remain as legacy response shapes backed by SQLite; filesystem status files are not used. Use /app/* pages for browser workflows.

Admins can remove eligible whole-batch history through Clean up processing data on the Pipeline page. Deleted pages become reusable; immediate database file shrinkage is not promised and compaction is separate maintenance. See the pipeline retention guide for archive prerequisites, protected work, retained files, and recovery limits.

Usage

Web Interface

Before first login, initialize the two fixed SQLite users:

.\.venv\Scripts\python.exe tools\setup_users.py --config config.yaml

For an upgrade from YAML credentials, preserve the existing admin hash while creating the operator account:

.\.venv\Scripts\python.exe tools\setup_users.py --config config.yaml --legacy-config config.yaml

After a successful import, remove the legacy authentication block. Passwords must contain uppercase, lowercase, numeric, and symbol characters and be 12–72 UTF-8 bytes.

1. Login: Access the app at http://localhost:8000/app/upload and select admin or operator
2. Upload PDFs: Use /app/upload, select one exact eligible published pipeline version for the whole batch, and submit one or more PDFs
3. Monitor Progress: Use /app/processing or /app/batches/{batch_id} to track queued work, splitting, extraction, review, and completion state
4. Review Exceptions: Use /app/review and /app/review/{review_item_id} for human review queues
5. Inspect Results: Use /app/documents/{document_id}/extraction for extracted fields and source PDF access
6. Inspect Batch History: Use /app/reports and click a recent batch row to view document task timelines and task-run details
7. Administer Configuration: Admin users create/version pipelines in /app/admin/pipeline, review forms in /app/schemas, inspect tasks and validation, manage users, and view the admin_ audit subset

Watch Folder

1. Create a binding: In the bottom of /app/admin/pipeline, publish and activate the pipeline, then bind an existing, non-overlapping incoming folder to one exact published version
2. Drop PDFs into that bound directory; watch_folder.dir alone is only a required startup-compatibility path and does not assign a workflow
3. Automatic Processing: Enabled bindings are scanned and files are pinned to their bound version before processing
4. Status Updates: Monitor progress through Reports and Processing Overview

Command Line

# Run with default configuration
.\.venv\Scripts\python.exe main.py

# Use custom configuration
.\.venv\Scripts\python.exe main.py --config-path custom_config.yaml

# Run without web server (watch folder only)
.\.venv\Scripts\python.exe main.py --no-web

API Reference

Authentication

• POST /api/login: API user authentication
• POST /api/session/refresh: Refresh the authenticated session
• POST /login: Browser login form
• GET /logout: Browser session termination
• GET /api/admin/users: List fixed accounts (admin only)
• PUT /api/admin/users/{username}/password: Change an account password (admin only)

Repeated failed login attempts are temporarily throttled and may return HTTP 429 Too Many Requests. The admin account has full access. The operator account cannot access administrative pages or APIs. Admins manage both passwords at /app/admin/users; changing a password revokes existing sessions for that account.

Runtime Health

• GET /health/live: Public web-process liveness with no operational details.
• GET /health/ready: Public supervised-runtime readiness (503 until ready).
• GET /api/admin/runtime-health: Run-scoped component and database diagnostics (admin only).

File Operations

• POST /upload: Retired non-durable upload endpoint; returns 410 Gone
• GET /api/pipelines/available: List exact pipeline versions eligible for the current user
• POST /api/batches/upload: Upload a batch of PDFs with a required pipeline_version_id, create SQLite batch/document records, and enqueue durable processing jobs; multipart file data is streamed through disk-backed staging under configured file-count, file-size, request-size, and concurrency limits
• GET /api/batches: List ingestion batches
• GET /api/upload-submissions/{submission_id}: Resolve an upload receipt; cleaned history returns status: purged and requires a deliberate new upload
• GET /api/batches/{batch_id}: Get batch details
• GET /api/batches/{batch_id}/documents: List batch documents
• GET /api/batches/{batch_id}/split-results: Show split source/child document relationships
• GET /api/documents/{document_id}/task-runs: List task runs for a document
• GET /api/documents/{document_id}/extraction: Read extraction result and normalized fields
• GET /api/documents/{document_id}/fields: Read extracted/reviewed fields
• GET /api/documents/{document_id}/file/pdf: Stream the source PDF or registered PDF artifact
• POST /api/documents/{document_id}/resume: Resume a document after review
• GET /api/files: Legacy compatibility list backed by SQLite documents
• GET /api/status/{file_id}: Legacy compatibility detail backed by SQLite documents, task runs, and artifacts

Review, Reports, and Admin

• GET /api/review/items: List review queue items
• POST /api/review/items/{review_item_id}/claim: Claim a review item
• POST /api/review/items/{review_item_id}/complete: Complete review and persist corrected values
• GET /api/reports/summary: Processing and review activity summary
• GET /api/admin/settings, PUT /api/admin/settings: Admin configuration state
• GET /api/admin/audit: Admin audit events
• /api/admin/pipeline-templates: Create/list named templates; per-template endpoints manage metadata, clone, draft save/validate/import/export/diff, immutable publish, and version history
• GET /api/admin/pipeline-templates/{template_id}/archive-preview: Explain archive impact and watch-folder blockers
• POST /api/admin/pipeline-templates/{template_id}/archive: Archive an inactive pipeline after name confirmation; definitions and history remain
• POST /api/admin/pipeline-templates/{template_id}/cleanup-preview: Preview eligible whole batches before an exclusive UTC cutoff_date
• POST /api/admin/pipeline-templates/{template_id}/cleanup: Confirm a current preview and exact pipeline name to delete eligible processing history; request/response details
• /api/admin/review-schemas: Create/list review forms; per-template endpoints manage metadata, draft save/validate/import/export, immutable publish, versions, and dependency usage
• /api/admin/watch-folder-bindings: List/create bindings; per-binding PATCH and DELETE update or remove eligible unreferenced bindings
• GET /api/admin/pipeline, PUT /api/admin/pipeline/draft, POST /api/admin/pipeline/publish, review-gate-rules, split-settings, and filesystem /api/schemas endpoints are legacy compatibility/configuration surfaces. New workflow authoring uses the versioned template endpoints.

Project Structure

pdfdoc_extraction/
├── 📄 main.py                    # Application entry point
├── 📄 config.yaml               # Ignored local deployment configuration (not committed)
├── 📄 requirements.txt          # Pinned runtime Python dependencies
├── 📄 requirements-dev.txt      # Runtime plus test/browser tooling
├── 📄 package.json              # Production frontend asset tooling
├── 📁 modules/                  # Core application modules
│   ├── api_router.py            # API composition and request handling
│   ├── config_manager.py        # Deployment configuration
│   ├── db/                      # SQLite schema, migrations, and repositories
│   ├── services/                # Workflow, review, upload, queue, admin, and artifact use cases
│   ├── workflow_loader.py       # Approved configured-task execution
│   └── workflow_manager.py      # Root, split-child, and resumed flows
├── 📁 standard_step/           # Processing pipeline steps
│   ├── extraction/             # Data extraction tasks
│   │   ├── extract_pdf.py       # Generic PDF extraction
│   │   ├── llama_cloud_v2.py    # LlamaCloud Extract v2
│   │   ├── glm_ocr_extract.py   # Local Ollama-backed GLM-OCR
│   │   └── liteparse_extract.py # Local LiteParse + Ollama extraction
│   ├── storage/                # Data storage tasks
│   ├── split/                   # PDF split and fan-out tasks
│   ├── review/                  # Human-review gate
│   ├── rules/                   # Business rule tasks
│   ├── archiver/                # File archiving tasks
│   ├── context/                 # Context management tasks
│   └── housekeeping/            # Cleanup and maintenance tasks
├── 📁 tools/                    # Administrative tools and config checker
│   └── config_check/            # CLI: python -m tools.config_check
├── 📁 web/                     # Web interface
│   ├── server.py               # FastAPI application
│   ├── templates/              # HTML templates
│   └── static/                 # CSS, JavaScript, assets
├── 📁 test/                    # Unit, service, workflow, integration, security, and visual tests
├── 📁 docs/                    # Architecture, operations, administration, and user guides
├── 📁 tasks/                   # Task creation guidance, plans, and archived work
└── 📁 pipeline_visual_editor_prototype/ # Separate React/Vite prototype, not production UI

Runtime directories such as data/, files/, processing*/, output/, web_upload/, and watch_folder/ hold local state or generated/customer artifacts; they are not source folders to commit.

Documentation

Start at the documentation entrance and choose your reading path:

• Operator: submit PDFs, monitor results and complete reviews.
• Application Admin: configure pipelines, forms, folders and app settings.
• Architecture: runtime boundaries, state, persistence and tradeoffs.
• Developer: understand, maintain and extend the code.
• IT Support: install, operate, upgrade and recover the deployment.
• Technical Reference: exact task parameters, formats and API contracts.
• History: dated evidence and the preserved original documents.

See the product roadmap for agreed future work and the linked assessment of comparable extraction products.

Development

Setting Up Development Environment

1. Clone and install

   git clone <repository-url>
   cd pdfdoc_extraction
   py -3.13 -m venv .venv
   .\.venv\Scripts\python.exe -m pip install -r requirements-dev.txt
   

2. Run tests

   .\.venv\Scripts\python.exe -m pytest -v
   

3. Development mode

   $env:USE_RELOAD="true"
   .\.venv\Scripts\python.exe main.py
   

Code Structure Guidelines

• PEP 8 Compliance: Follow Python style guidelines
• Type Hints: Use comprehensive type annotations
• Documentation: Concise Google-style docstrings for public and non-obvious modules, classes, and functions
• Error Handling: Structured error handling with custom exceptions
• Testing: Unit tests for all new features

Testing

Running Tests

# Run all tests
.\.venv\Scripts\python.exe -m pytest -v

# Run specific test file
.\.venv\Scripts\python.exe -m pytest -v test/core/test_config_manager.py

# Run static type checking when Pyright is installed (it is not pinned in requirements.txt)
pyright

Example focused validation for SQLite-backed ingestion and operator state:

.\.venv\Scripts\python.exe -m pytest -v test\integration\test_sqlite_only_workflow_state.py test\integration\test_batch_upload_api.py test\integration\test_reports_api.py test\integration\test_settings_api.py

Run the full suite before handing off broad or cross-cutting changes.

The end-to-end workflow fixture config also passes config-check:

.\.venv\Scripts\python.exe -m tools.config_check validate --config test\data\config.yaml --import-checks

Test Categories

• Unit Tests: Individual component testing
• Integration Tests: End-to-end workflow validation
• Third-party Tests: External service integration testing

Troubleshooting

Common Issues

Application won't start

• Verify Python 3.13+ is installed
• Check configuration file syntax
• Ensure all required directories exist
• Check app.supervisor.log for a missing or incompatible database schema

PDF processing fails

• Verify Llama Cloud API credentials
• Check PDF file format and accessibility
• Review application logs for detailed errors

Web interface issues

• Confirm web server is running on correct port
• Check browser console for JavaScript errors
• Verify authentication credentials

Logging

logging.log_file is the base path for role-specific logs. With the default app.log, normal startup writes app.supervisor.log, app.web.log, and app.worker.log; relative paths are resolved beside the active configuration file. logging.log_level applies to all three processes.

Getting Help

1. Check the User Guide for detailed instructions
2. Review Design Architecture for technical details
3. Use the Configuration Checker Troubleshooting Guide for config-check findings and stored pipeline validation
4. Review the Review Form Administrator Guide for review-form authoring and versioning
5. Review the Frontend Deployment and Rollback Guide when releasing or reverting browser modules
6. Examine application logs for error details
7. Check the repository's issue tracker for known problems

Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Make your changes
4. Add tests for new functionality
5. Run the test suite
6. Commit your changes (git commit -m 'Add amazing feature')
7. Push to the branch (git push origin feature/amazing-feature)
8. Open a Pull Request

Development Guidelines

• Follow PEP 8 style guidelines
• Add comprehensive type hints
• Include unit tests for new features
• Update documentation as needed
• Ensure backward compatibility

License

No repository license file is currently present. Confirm the applicable usage and distribution terms with the project owner before redistributing the code.

Acknowledgments

• Llama Cloud for AI-powered document extraction
• Prefect for workflow orchestration
• FastAPI for the web framework
• Open source community for various tools and libraries

Support

For support and questions:

• Create an issue in the repository's issue tracker
• Check existing documentation in the docs/ folder
• Review the User Guide for detailed instructions

---

Built with ❤️ for intelligent document processing