Multisource PDF to Markdown

A complete, installable Codex agent skill for converting scientific, academic, scanned, two-column, math-heavy PDFs into provenance-rich Markdown. It combines a Docling-led local pipeline with independent layout, OCR, academic-structure, table, formula, and VLM evidence. Every automated result is treated as a draft: the skill requires an agent to compare the original PDF and the Markdown page by page before delivery.

The repository includes the complete skill tree, eight prefix-free micromamba environment specifications, portable runtime configuration, installers, tests, fixtures, documentation, release automation, and a reproducible two-agent benchmark.

[!IMPORTANT] This is a quality-first research workflow, not a one-click fidelity guarantee. Exit status 0 means the automated gates passed; it does not replace direct PDF inspection or manual Markdown repair.

The embedded animation and downloadable H.264 version are Full HD (1920 × 1080).

Contents

• Multisource PDF to Markdown
  • Contents
  • What the skill produces
  • Repository layout
  • Requirements and sizing expectations
  • Quick start
    • 1. Clone or unpack the repository
    • 2. Install the skill where Codex can discover it
    • 3. Install the runtime profile
    • 4. Verify the selected runtimes
    • 5. Ask the agent to convert a PDF
    • 6. Review before accepting the output
  • Runtime profiles
    • Install or repair each environment separately
  • Relocating environments and configuring absolute paths
  • Conversion controls
    • GROBID lifecycle
    • Cloud providers are strict opt-in
  • Independent two-agent demo
    • Recorded result
    • Page-by-page comparison and inline diff
  • Reproduce the demo metrics and media
  • Development and validation
  • Build publication archives
  • License and third-party boundaries
  • 🤝 Contributing

What the skill produces

For an input such as paper.pdf and an output named paper.md, the default quality-first run produces:

• paper.md: searchable Markdown with <!-- page: N --> provenance markers;
• paper.md.report.json: runtime, GPU, source, quality-gate, census, warning, and escalation evidence;
• paper.md.provenance.json: page, bounding box, source, evidence, and artifact metadata for selected blocks;
• paper_assets/: only the local images referenced by the final Markdown;
• paper.md.sources/: retained extractor and engine sidecars when --keep-sources is enabled.

The best profile uses local CUDA Docling as the canonical document source, then consults native PDF geometry, PyMuPDF4LLM, Poppler, pdfplumber, GROBID, MinerU, PP-StructureV3, and benchmark-gated olmOCR evidence. A specialist can repair a failed page-local element; it never silently replaces the document skeleton. Cloud providers are disabled unless the caller explicitly opts in.

Repository layout

pdf-to-markdown-multisource/
├── skills/pdf-to-markdown-multisource/   # installable Agent Skill
│   ├── SKILL.md
│   ├── agents/openai.yaml
│   ├── config/runtime.example.json
│   ├── environments/                   # all eight environment.yaml files
│   ├── scripts/                        # conversion, bootstrap, verification
│   ├── tests/
│   ├── references/
│   └── benchmarks/
├── demo/                                # CC BY paper, both runs, metrics, media
├── scripts/install_skill.py              # repository/user installer
├── scripts/install.sh                    # environment installer entry point
├── scripts/validate_repository.py        # integrity and portability checks
└── scripts/build_release.py              # repository + skill-only ZIPs

Requirements and sizing expectations

The environment specifications target:

• 64-bit Linux;
• an NVIDIA driver compatible with the bundled CUDA 12.6 packages;
• an NVIDIA GPU for the supported default path;
• a 24 GiB-class GPU for the heaviest automatic VLM path (benchmarked on an RTX 3090); the deterministic extractors may work with less VRAM, but that has not been characterized here;
• Python 3 only to launch the bootstrap script;
• micromamba;
• Git and network access while GROBID, packages, and model weights are downloaded;
• tens of gigabytes of free disk space for all environments and first-use model caches.

The environment YAMLs contain packages, not model weights. Hugging Face, ModelScope, PaddleX, and related weights are downloaded on first use into cache directories owned by the corresponding environment prefix. This keeps the repository small and avoids silently depending on a user-wide cache.

CPU execution is available only as an explicit diagnostic choice with --docling-device cpu; the bundled and verified production path is CUDA. There is no automatic CPU fallback after a GPU failure.

Quick start

1. Clone or unpack the repository

git clone https://github.com/CamiloMartinezM/pdf-to-markdown-multisource.git
cd pdf-to-markdown-multisource

A release ZIP works identically after extraction. Set three installation-specific paths for the commands below:

PDF_MD_REPO=/absolute/path/to/pdf-to-markdown-multisource
PDF_MD_ENVS=/absolute/path/to/pdf-md-environments
PDF_MD_MICROMAMBA=/absolute/path/to/micromamba

2. Install the skill where Codex can discover it

Repository-scoped installation is recommended for teams because the dependency is visible to everyone working in that project:

python3 "$PDF_MD_REPO/scripts/install_skill.py" \
  --repo /absolute/path/to/another-project \
  --mode symlink

This creates:

/absolute/path/to/another-project/.agents/skills/pdf-to-markdown-multisource

For a user-scoped installation:

python3 "$PDF_MD_REPO/scripts/install_skill.py" --user --mode symlink

That places the skill below ~/.agents/skills/. Use --mode copy instead when the installed skill must survive moving or deleting the repository checkout. The installer refuses to overwrite an existing destination; move the old installation aside deliberately before upgrading it.

OpenAI's public guidance describes repository and user skill discovery in Build skills. The directory also follows the portable Agent Skills specification, and its metadata mirrors the examples in openai/skills.

Restart Codex if the newly installed skill is not discovered in the current session.

3. Install the runtime profile

The recommended profile contains every runtime used automatically by --profile best:

"$PDF_MD_REPO/scripts/install.sh" \
  --micromamba "$PDF_MD_MICROMAMBA" \
  --env-root "$PDF_MD_ENVS" \
  --profile recommended

The installer:

1. creates separate prefixes from the bundled environment YAMLs;
2. clones and builds pinned GROBID 0.9.0 when its runtime is selected;
3. leaves existing prefixes unchanged unless --update-existing is explicit;
4. writes the local absolute paths to skills/pdf-to-markdown-multisource/config/runtime.local.json.

Inspect all targets and commands before changing the machine with --dry-run:

"$PDF_MD_REPO/scripts/install.sh" \
  --micromamba "$PDF_MD_MICROMAMBA" \
  --env-root "$PDF_MD_ENVS" \
  --profile recommended \
  --dry-run

4. Verify the selected runtimes

PDF_MD_SKILL="$PDF_MD_REPO/skills/pdf-to-markdown-multisource"

python3 "$PDF_MD_SKILL/scripts/run_in_core.py" \
  "$PDF_MD_SKILL/scripts/verify_environments.py" \
  --profile recommended \
  --output /tmp/pdf-md-runtime-verification.json

Verification performs real CUDA tensor operations rather than relying on package imports or nvidia-smi alone. It also checks Poppler, Tesseract, English tessdata, PP-StructureV3, Java, and the local GROBID distribution. A nonzero exit means the runtime contract is not ready.

For a copied skill installation, run bootstrap and verification from the copied skill path so its own untracked config/runtime.local.json is populated. Existing environment prefixes will be reused:

PDF_MD_INSTALLED=/absolute/path/to/another-project/.agents/skills/pdf-to-markdown-multisource

python3 "$PDF_MD_INSTALLED/scripts/bootstrap_environments.py" \
  --micromamba "$PDF_MD_MICROMAMBA" \
  --env-root "$PDF_MD_ENVS" \
  --profile recommended

5. Ask the agent to convert a PDF

A useful prompt is:

Use $pdf-to-markdown-multisource to convert /absolute/path/to/paper.pdf to
/absolute/path/to/paper.md with the best local profile and retained source
evidence. Inspect the original PDF page by page, manually repair the Markdown,
validate every equation, table, figure, caption, reference, page marker, and image
link, and report any unresolved uncertainty. Do not upload the PDF.

The equivalent direct launcher is:

python3 "$PDF_MD_SKILL/scripts/run_converter.py" \
  /absolute/path/to/paper.pdf \
  --output /absolute/path/to/paper.md \
  --profile best \
  --keep-sources

Do not pass --overwrite unless replacing the named output is intended. Do not use --allow-quality-warnings for an acceptance run; it exists to retain exploratory diagnostics after a known gate failure.

6. Review before accepting the output

An agent using this skill must still:

1. read output.md.report.json and require blocking_gate_count == 0;
2. inspect source statuses, escalation, page decisions, material-element actions, and censuses;
3. open the original PDF and compare every page with the Markdown;
4. repair reading order, headings, word boundaries, duplicate fragments, formulas, tables, figures, captions, and furniture only when supported by visible or independent page-local evidence;
5. re-read every changed region beside the source;
6. validate local image links, page markers, HTML, LaTeX, empty grids, and section order;
7. record manual changes and unresolved uncertainty.

Runtime profiles

Every runtime is isolated because its Torch, Transformers, Paddle, Java, Pillow, and model constraints are not mutually compatible.

Install or repair each environment separately

Each command below is independent. Add --update-existing only when intentionally reconciling an already-created prefix with the pinned environment specification.

"$PDF_MD_REPO/scripts/install.sh" --micromamba "$PDF_MD_MICROMAMBA" --env-root "$PDF_MD_ENVS" --only core
"$PDF_MD_REPO/scripts/install.sh" --micromamba "$PDF_MD_MICROMAMBA" --env-root "$PDF_MD_ENVS" --only grobid
"$PDF_MD_REPO/scripts/install.sh" --micromamba "$PDF_MD_MICROMAMBA" --env-root "$PDF_MD_ENVS" --only mineru
"$PDF_MD_REPO/scripts/install.sh" --micromamba "$PDF_MD_MICROMAMBA" --env-root "$PDF_MD_ENVS" --only paddleocr
"$PDF_MD_REPO/scripts/install.sh" --micromamba "$PDF_MD_MICROMAMBA" --env-root "$PDF_MD_ENVS" --only olmocr
"$PDF_MD_REPO/scripts/install.sh" --micromamba "$PDF_MD_MICROMAMBA" --env-root "$PDF_MD_ENVS" --only marker
"$PDF_MD_REPO/scripts/install.sh" --micromamba "$PDF_MD_MICROMAMBA" --env-root "$PDF_MD_ENVS" --only chandra
"$PDF_MD_REPO/scripts/install.sh" --micromamba "$PDF_MD_MICROMAMBA" --env-root "$PDF_MD_ENVS" --only infinity

Verify one repaired prefix without paying the cost of every probe:

python3 "$PDF_MD_SKILL/scripts/run_in_core.py" \
  "$PDF_MD_SKILL/scripts/verify_environments.py" \
  --only core,grobid

Use --profile all for the complete eight-runtime verifier used by the VLM engine-selection benchmark. The document demo below verifies the five runtimes in the recommended profile.

Relocating environments and configuring absolute paths

Absolute installation paths belong only in a local runtime configuration selected in this order:

1. the file named by PDF_MD_RUNTIME_CONFIG;
2. config/runtime.local.json inside the installed skill;
3. legacy config/runtime.json, if a downstream installation still has one.

If environments already exist elsewhere, create the local configuration without reinstalling packages:

cp "$PDF_MD_SKILL/config/runtime.example.json" \
  "$PDF_MD_SKILL/config/runtime.local.json"

Edit these JSON fields to absolute paths on the new machine:

• micromamba;
• environments.core;
• environments.grobid;
• environments.mineru;
• environments.paddleocr;
• environments.marker;
• environments.chandra;
• environments.olmocr;
• environments.infinity.

All eight keys remain in the schema even when only a subset is installed. A profile requests only its own environments. For immutable or shared configuration, keep the JSON outside the skill and set:

export PDF_MD_RUNTIME_CONFIG=/absolute/path/to/pdf-md-runtime.json

The environment YAMLs intentionally have no prefix: entries. After moving the repository or its environments, update this one JSON file; no Python or SKILL.md edits are necessary. Model identifiers in that JSON are names, not filesystem paths.

Run the portability guard after any relocation-related change:

python3 "$PDF_MD_REPO/scripts/validate_repository.py"

Conversion controls

The most useful controls are:

• --profile best: full quality-first automatic path;
• --profile native-only: deterministic/native diagnostic path;
• --keep-sources: retain engine evidence for audit and repair;
• --max-pages N: bounded smoke test, recorded as truncated;
• --vlm never: disable VLM comparison;
• --vlm auto: use the benchmark selection only when quality gates require it;
• --paddleocr auto: region-scoped PP-StructureV3 rescue for unresolved material;
• --no-grobid, --no-mineru, --no-docling, --no-pymupdf4llm, and --no-formula-enrichment: diagnostic switches, not a way to conceal a failed runtime in a final run.

Marker and Chandra additionally require --allow-restricted-models. Review the current upstream code and model terms before enabling them.

GROBID lifecycle

The converter starts the loopback-only GROBID service when the best profile needs it. Manual lifecycle commands are:

python3 "$PDF_MD_SKILL/scripts/run_in_core.py" "$PDF_MD_SKILL/scripts/grobid_service.py" ensure
python3 "$PDF_MD_SKILL/scripts/run_in_core.py" "$PDF_MD_SKILL/scripts/grobid_service.py" status
python3 "$PDF_MD_SKILL/scripts/run_in_core.py" "$PDF_MD_SKILL/scripts/grobid_service.py" stop

PID and log state live below the GROBID prefix. The default service URL is http://127.0.0.1:8070.

Cloud providers are strict opt-in

Local processing is the default. A cloud request runs only when the profile is best, the provider is named, upload is explicitly allowed, credentials are present, and unresolved local quality still requires rescue:

# Requires MATHPIX_APP_ID and MATHPIX_APP_KEY.
python3 "$PDF_MD_SKILL/scripts/run_converter.py" paper.pdf \
  --output paper.md \
  --cloud mathpix \
  --allow-cloud-upload

# Requires MISTRAL_API_KEY.
python3 "$PDF_MD_SKILL/scripts/run_converter.py" paper.pdf \
  --output paper.md \
  --cloud mistral \
  --allow-cloud-upload

Both adapters redact credentials and attempt remote deletion in finally, but no client can guarantee a provider's backups, legal retention, or policy. Do not upload confidential, personal, unpublished, or licensed material without checking the provider's current terms and obtaining the necessary authorization.

Independent two-agent demo

The benchmark paper is Kostas, Aroca-Ouellette, and Rudzicz (2021), BENDR: Using Transformers and a Contrastive Self-Supervised Learning Task to Learn From Massive Amounts of EEG Data, doi:10.3389/fnhum.2021.653659. Its 15 pages combine two-column prose, cross-page continuations, equations, subscripts and superscripts, signed decimals, three tables, five multi-panel figures, dense references, and an appendix table whose learning rates all use negative exponents. The article and derived benchmark images are distributed under CC BY 4.0.

Two separate fresh-context agents used gpt-5.6-terra at high reasoning effort and ran sequentially:

• Visual-only control: received 15 rendered JPEG pages and could use only native image viewing, reasoning, and output-file authoring. The PDF text layer, OCR, extraction tools, scripts, environments, skills, shell inspection, web services, reference, and treatment output were prohibited.
• Multisource skill treatment: received the PDF, page renders, installed skill, and configured local runtimes. It verified the recommended five-runtime profile, ran --profile best --keep-sources, reviewed every page, repaired the two stale equation-image targets that failed the initial hard gate, and checked the final structure and local assets. It could not access the reference or control output.

Each arm had a disjoint working-directory contract. Both ran on the same host, so isolation was enforced by fresh context and explicit path and tool restrictions rather than a separate kernel or container sandbox. Existing environments, model caches, and weights were reused; the timing is a warm-runtime benchmark and does not include installation. Paper selection, copying, hashing, page rendering, reference construction, scoring, and media generation are excluded from both arm times.

Recorded result

The skill produced 4,902 fewer narrative-word edits and 394 fewer numeric edits: reductions of 67.8% and 69.2% relative to the visual-only control. It also preserved nearly complete reading order and all five reusable figure assets, raising the fidelity proxy by 20.39 points and the ingestability proxy by 18.50 points. The visual-only arm was 2.24 times faster and did better on critical symbols, heading recall, and the ten audited table facts.

The control's largest failure was omission: it reduced dense methods, results, and references to 6,013 candidate words against an 11,575-word reference and represented figures only as prose descriptions. The skill recovered 12,255 candidate words and the figures, but its extra content included duplicated page-transition passages and misplaced continuations. It also rendered the five appendix learning rates as forms such as 1 × 10 - 5 instead of preserving the negative fifth-power exponent, which caused all five appendix fact checks to fail. Both arms recovered all three equation concepts and all three table structures; neither preserved mathematical notation reliably enough to treat the Markdown as unchecked source data.

The skill arm's own run note reported no remaining material uncertainty after review. The later source-adjudicated scoring pass found the residual duplication, boundary, and exponent errors above. This is why the skill contract treats agent review as necessary but not infallible, and why the report, provenance, retained sources, metrics, and visible diffs are published with the Markdown.

Word, numeric, and math inaccuracies are substitutions, deletions, and insertions from deterministic token alignment against a source-adjudicated semantic reference. Narrative scoring excludes Markdown syntax, image targets, display math, and accessibility-only figure descriptions. Numeric scoring keeps scientifically material signs, digits, decimal precision, fractions, exponents, and percentages distinct. Math scoring covers tokens placed inside math delimiters; consequently, missing delimiters count as missing mathematical content. Equivalent Unicode and LaTeX forms are accepted only where the manifest says they are semantically equivalent. Fidelity and LLM ingestability are transparent benchmark-specific weighted proxies, not external standards or human-preference scores. Their formulas and every mismatch are in results.json and evaluate_outputs.py.

Page-by-page comparison and inline diff

Each comparison places the source page, visual-only Markdown, and skill Markdown on one white canvas. Each diff aligns one reconstruction with the source-adjudicated reference: cyan underlined + tokens occur only in the candidate, vermillion struck − tokens are missing from it, and unmarked text is shared context. The symbols and text decoration make the diff readable without color.

Page 1 — title, abstract, metadata, and introduction

Page 2 — introduction and pre-training survey

Page 3 — pre-training with DNNs and alternative methods

Page 4 — contrastive objective and Equation 1

Page 5 — datasets, Table 1, and preprocessing

Page 6 — preprocessing and model architecture

Page 7 — architecture, training, and Equation 2

Page 8 — Figure 1 and downstream fine-tuning

Page 9 — Figure 2 and pre-training results

Page 10 — Figures 3 and 4 and downstream results

Page 11 — Figure 5 and discussion

Page 12 — conclusion and author statements

Page 13 — dense two-column references

Page 14 — reference continuation

Page 15 — appendix hyperparameter table and exponents

The complete benchmark record includes the path-neutral arm contracts, timing protocol, source-reference review, raw outputs, converter evidence, limitations, and reproduction commands.

Reproduce the demo metrics and media

The committed source PDF has SHA-256:

9be6c54972362ceb760363d02866d2a2ce5f330371447daab7827a0758acc7a7

Recompute the recorded scores with the Python standard library:

python3 demo/scripts/evaluate_outputs.py \
  --reference demo/reference/reference.md \
  --manifest demo/reference/manifest.json \
  --manual demo/runs/manual/output.md \
  --skill demo/runs/skill/output.md \
  --timing demo/runs/timing.json \
  --output-json demo/metrics/results.json \
  --output-csv demo/metrics/results.csv

Media generation additionally requires Pillow, Pandoc, Chromium, Poppler's pdftoppm, and FFmpeg:

python3 -m pip install -r demo/requirements.txt
python3 demo/scripts/build_demo_media.py

The source paper, audited reference, arm outputs, and timing record are fixed benchmark inputs. Rerunning the two agent contracts in demo/prompts creates a new experiment; it is not part of CI.

Development and validation

The core environment contains the test and lint tools:

python3 "$PDF_MD_SKILL/scripts/run_in_core.py" -m pytest -q "$PDF_MD_SKILL/tests"
python3 "$PDF_MD_SKILL/scripts/run_in_core.py" -m ruff check "$PDF_MD_SKILL/scripts" "$PDF_MD_SKILL/tests"
python3 "$PDF_MD_SKILL/scripts/run_in_core.py" -m compileall -q "$PDF_MD_SKILL/scripts" "$PDF_MD_SKILL/tests"
python3 "$PDF_MD_REPO/scripts/validate_repository.py"

The static validator checks required metadata, all eight environment manifests, JSON syntax, skill metadata, generated-state exclusions, path portability, and the recorded demo fixture digest.

Before changing automatic VLM selection, rerun all four challengers on the bundled adversarial fixture and review GPU evidence, license boundaries, reading order, content, and pollution gates:

python3 "$PDF_MD_SKILL/scripts/run_in_core.py" \
  "$PDF_MD_SKILL/scripts/benchmark_vlm_engines.py" \
  --output-dir "$PDF_MD_SKILL/benchmarks/latest" \
  --fixture "$PDF_MD_SKILL/benchmarks/latest/adversarial-academic.pdf" \
  --resume \
  --acknowledge-restricted-models \
  --write-selection

See SECURITY.md for private vulnerability reporting and PDF/credential boundaries.

Build publication archives

Release archives are generated only from tracked Git content. From a clean commit:

python3 scripts/build_release.py --version 0.1.0

This writes:

• dist/pdf-to-markdown-multisource-0.1.0.zip: the complete repository;
• dist/pdf-to-markdown-multisource-skill-0.1.0.zip: the installable skill subtree;
• dist/SHA256SUMS: digests for both archives.

The release workflow validates the repository and attaches those files when a v* tag is pushed. Publishing a tag or pushing to GitHub is an external action and is intentionally not performed by setup scripts.

License and third-party boundaries

Repository-authored code and documentation are available under the MIT License. The BENDR source paper and benchmark images that reproduce or adapt it remain under CC BY 4.0, with the complete citation and attribution boundary in demo/source/README.md.

The repository and release ZIPs do not bundle third-party packages, a GROBID checkout or build, model weights, or model caches. Running the installer or invoking an engine causes upstream packages, source, and models to be obtained into caller-selected environment prefixes and caches under their own terms. The skill can initiate those downloads on the caller's behalf, but it does not provide or relicense the downloaded artifacts. Optional cloud routes upload a PDF only after explicit provider selection, upload authorization, and credential configuration.

THIRD_PARTY_NOTICES.md records the reviewed terms and acquisition boundary for each principal engine, configured model, and cloud service, including the modified MinerU license, GPL and modified OpenRAIL components, dynamic model selection, and redistribution obligations. The engine survey separately explains technical selection and benchmark gates.

Release history is in CHANGELOG.md, and citation metadata is in CITATION.cff.

---

🤝 Contributing

Contributions are welcome! Feel free to:

• Report bugs
• Suggest new features
• Submit pull requests

See CONTRIBUTING.md for development and validation requirements.

---