paperpipe

The problem: You're implementing a paper. You need the exact equations, want to verify your code matches the math, and your coding agent keeps hallucinating details. Reading PDFs is slow; copy-pasting LaTeX is tedious.

The solution: paperpipe maintains a local paper database with PDFs, LaTeX source (when available), extracted equations, and coding-oriented summaries. It integrates with coding agents (Claude Code, Codex, Gemini CLI) so they can ground their responses in actual paper content.

Typical workflow

# 1. Add papers you're implementing (multiple at once, mixed sources OK)
papi add 2303.08813 1706.03762 "Attention Is All You Need"

# Or one at a time
papi add 2303.08813                    # LoRA paper
papi add https://arxiv.org/abs/1706.03762  # URL
papi add "Attention Is All You Need"   # Search by title

# 2. Check what equations you need to implement
papi show lora --level eq             # prints equations to stdout

# 3. Verify your code matches the paper
#    (or let your coding agent do this via the /papi skill)
papi show lora --level tex            # exact LaTeX definitions

# 4. Ask cross-paper questions (requires RAG backend)
papi ask "How does LoRA differ from full fine-tuning in terms of parameter count?"

# 5. Keep implementation notes
papi notes lora                       # opens notes.md in $EDITOR

Installation

# Basic (uv recommended)
uv tool install paperpipe

# With features
uv tool install paperpipe --with "paperpipe[llm]"      # better summaries via LLMs
uv tool install paperpipe --with "paperpipe[paperqa]"  # RAG via PaperQA2
uv tool install paperpipe --with "paperpipe[leann]"    # local RAG via LEANN
uv tool install paperpipe --with "paperpipe[figures]"  # figure extraction from LaTeX/PDF
uv tool install paperpipe --with "paperpipe[mcp]"      # MCP server integrations (Python 3.11+)
uv tool install paperpipe --with "paperpipe[all]"      # everything

pip install paperpipe
pip install 'paperpipe[llm]'
pip install 'paperpipe[paperqa]'  # PaperQA2 + multimodal PDF parsing
pip install 'paperpipe[leann]'
pip install 'paperpipe[figures]'        # figure extraction from LaTeX/PDF
pip install 'paperpipe[mcp]'
pip install 'paperpipe[all]'

git clone https://github.com/hummat/paperpipe && cd paperpipe
pip install -e ".[all]"

What paperpipe stores

~/.paperpipe/                         # override with PAPER_DB_PATH
├── index.json
├── .pqa_papers/                      # staged PDFs for RAG (created on first `papi ask`)
├── .pqa_index/                       # PaperQA2 index cache
├── .leann/                           # LEANN index cache
├── papers/
│   └── lora/
│       ├── paper.pdf                 # for RAG backends
│       ├── source.tex                # full LaTeX (if available from arXiv)
│       ├── equations.md              # extracted equations with context
│       ├── summary.md                # coding-oriented summary
│       ├── tldr.md                   # one-paragraph TL;DR
│       ├── meta.json                 # metadata + tags
│       ├── notes.md                  # your implementation notes
│       └── figures/                  # extracted figures (if available)
│           ├── figure1.png
│           └── figure2.pdf

Why this structure matters:

• equations.md — Key equations with variable definitions. Use for code verification.
• source.tex — Original LaTeX. Use when you need exact notation or the equation extraction missed something.
• summary.md — High-level overview focused on implementation (not literature review). Use for understanding the approach.
• tldr.md — Quick 2-3 sentence overview of the paper's contribution.
• figures/ — Architecture diagrams, network structures, and result plots extracted from LaTeX source or PDF.
• .pqa_papers/ — Staged PDFs only (no markdown) so RAG backends don't index generated content.

Core commands

Run papi --help or papi <command> --help for full options.

Import/Export

Share your paper collection with others or back it up.

Export:

# Export full list to JSON
papi list --json > my_papers.json

# Export specific tag
papi list --tag "computer-vision" --json > cv_papers.json

Import:

# Import from JSON (preserves custom names and tags)
papi add --from-file my_papers.json

# Import from text file (one arXiv ID per line)
papi add --from-file paper_ids.txt --tags "imported"

# Import from BibTeX file (requires bibtexparser)
papi add --from-file papers.bib
# or install with BibTeX support:
# uv tool install paperpipe --with "paperpipe[bibtex]"

Title Search:

# Add papers by title (auto-selects if high confidence match)
papi add "Attention Is All You Need"
papi add "NeRF: Representing Scenes as Neural Radiance Fields"

Semantic Scholar Support:

# Add papers from Semantic Scholar
papi add https://www.semanticscholar.org/paper/...
papi add 0123456789abcdef0123456789abcdef01234567  # S2 paper ID

Multiple papers at once (mixed sources OK):

papi add 2303.08813 1706.03762 "Attention Is All You Need"
papi add 2303.08813 https://www.semanticscholar.org/paper/... "NeRF"

Exact text search (fast, no LLM required):

papi search --rg "AdamW"                # case-insensitive, literal string (default)
papi search --rg --case-sensitive "NeRF"  # match exact case
papi search --rg --regex "AdamW|Lion"   # regex or OR mode (opt-in)

Ranked search (BM25 via SQLite FTS5, no LLM required):

papi index --backend search --search-rebuild    # builds <paper_db>/search.db
papi search "surface reconstruction"             # uses FTS if available (default)
papi search --no-fts "surface reconstruction"    # force in-memory scan (disables FTS, uses fuzzy matching)
papi search --no-fts --exact "exact phrase"      # force scan with exact matching only

Hybrid ranked+exact search:

papi search --hybrid "surface reconstruction"
papi search --hybrid --show-grep-hits "surface reconstruction"

Limit search to specific papers:

papi search "attention" -p attention-is-all-you-need
papi search "loss" -p paper1,paper2,paper3

What are FTS and BM25?

• FTS = Full-Text Search. Here it means SQLite’s FTS5 extension, which builds an inverted index so searches don’t have to rescan every file on every query.
• BM25 = Okapi BM25, a standard relevance-ranking function used by many search engines. It ranks results based on term frequency, inverse document frequency, and document length normalization.

References (external):

https://sqlite.org/fts5.html
https://en.wikipedia.org/wiki/Okapi_BM25

export PAPERPIPE_SEARCH_MODE=auto   # auto|fts|scan|hybrid

[search]
mode = "auto" # auto|fts|scan|hybrid

Agent integration

paperpipe is designed to work with coding agents. Install the skill and MCP servers:

papi install                          # installs skill + MCP for detected CLIs
# or be specific:
papi install skill --claude --codex --gemini
papi install mcp --claude --codex --gemini

After installation, your agent can:

• Use /papi to get paper context (skill)
• Call MCP tools like retrieve_chunks for RAG retrieval
• Verify code against paper equations

Custom skills

For a ready-to-paste snippet for your repo's agent instructions, run papi docs or see AGENT_INTEGRATION.md.

What the agent sees

When you (or your agent) run papi show <paper> --level eq, you get structured output like:

## Equation 1: LoRA Update
$$h = W_0 x + \Delta W x = W_0 x + BA x$$
where:
- $W_0 \in \mathbb{R}^{d \times k}$: pretrained weight matrix (frozen)
- $B \in \mathbb{R}^{d \times r}$, $A \in \mathbb{R}^{r \times k}$: low-rank matrices
- $r \ll \min(d, k)$: the rank (typically 1-64)

This is what makes verification possible — the agent can compare your code symbol-by-symbol.

# Install MCP servers for all supported agents (user scope)
papi install mcp

# Install for specific agents
papi install mcp --claude
papi install mcp --codex
papi install mcp --gemini

# Install repo-local MCP configs (Claude + Gemini) and Codex globally
papi install mcp --repo

# Customize embedding model
papi install mcp --embedding text-embedding-3-small

RAG backends (papi ask)

paperpipe supports two RAG backends for cross-paper questions:

# PaperQA2 (default if installed)
papi ask "What regularization techniques do these papers use?"

# LEANN (local)
papi ask "..." --backend leann

The first query builds an index (cached under .pqa_index/ or .leann/). Use papi index to pre-build.

# API keys should be in env
export OPENAI_API_KEY=...
export ANTHROPIC_API_KEY=...
export GEMINI_API_KEY=...
export VOYAGE_API_KEY=...
export OPENROUTER_API_KEY=...

# Ollama (local) + Ollama embeddings
papi index --backend pqa --pqa-llm ollama/olmo-3:7b --pqa-embedding ollama/nomic-embed-text

# GPT + OpenAI Embeddings
papi index --backend pqa --pqa-llm gpt-4.1 --pqa-summary-llm gpt-4.1-mini --pqa-embedding text-embedding-3-small

# Gemini + Google Embeddings
papi index --backend pqa --pqa-llm gemini/gemini-3-flash-preview --pqa-embedding gemini/gemini-embedding-001

# Claude + Voyage Embeddings
papi index --backend pqa --pqa-llm claude-sonnet-4-5 --pqa-summary-llm claude-haiku-4-5 --pqa-embedding voyage/voyage-4

# OpenRouter + Voyage Embeddings
papi index --backend pqa --pqa-llm openrouter/google/gemini-3.5-flash --pqa-embedding voyage/voyage-4

# Ollama (local)
papi ask "how is neus different from nerf?" --backend pqa --pqa-llm ollama/olmo-3:7b --pqa-embedding ollama/nomic-embed-text

# GPT
papi ask "how is neus different from nerf?" --backend pqa --pqa-llm gpt-4.1 --pqa-summary-llm gpt-4.1-mini --pqa-embedding text-embedding-3-small

# Gemini
papi ask "how is neus different from nerf?" --backend pqa --pqa-llm gemini/gemini-3-flash-preview --pqa-embedding gemini/gemini-embedding-001

# Claude
papi ask "how is neus different from nerf?" --backend pqa --pqa-llm claude-sonnet-4-5 --pqa-summary-llm claude-haiku-4-5 --pqa-embedding voyage/voyage-4

# OpenRouter
papi ask "how is neus different from nerf?" --backend pqa --pqa-llm openrouter/google/gemini-3.5-flash --pqa-embedding voyage/voyage-4

export OPENAI_API_KEY=...
papi index --backend pqa --pqa-embedding text-embedding-3-small

export GEMINI_API_KEY=...
papi index --backend pqa --pqa-embedding gemini/gemini-embedding-001

export VOYAGE_API_KEY=...
papi index --backend pqa --pqa-embedding voyage/voyage-4

export OPENAI_API_BASE=https://api.voyageai.com/v1
export OPENAI_API_KEY="$VOYAGE_API_KEY"
papi index --backend pqa --pqa-embedding openai/voyage-4

papi ask "..." --backend leann --leann-provider ollama --leann-model qwen3:8b
papi ask "..." --backend leann --leann-host http://localhost:11434
papi ask "..." --backend leann --leann-top-k 12 --leann-complexity 64

# API keys should be in env
export OPENAI_API_KEY=...
export ANTHROPIC_API_KEY=...
export GEMINI_API_KEY=...
export VOYAGE_API_KEY=...
export OPENROUTER_API_KEY=...

# Ollama (local) + Ollama embeddings
papi index --backend leann --leann-embedding-mode ollama --leann-embedding-model nomic-embed-text

# OpenAI + OpenAI embeddings
papi index --backend leann --leann-embedding-mode openai --leann-embedding-model text-embedding-3-small --leann-embedding-api-key $OPENAI_API_KEY

# Gemini + Gemini embeddings (OpenAI-compatible)
papi index --backend leann --leann-embedding-mode openai --leann-embedding-model gemini-embedding-001 --leann-embedding-api-base https://generativelanguage.googleapis.com/v1beta/openai/ --leann-embedding-api-key $GEMINI_API_KEY

# Voyage embeddings (OpenAI-compatible) — endpoint + key auto-routed from VOYAGE_API_KEY
papi index --backend leann --leann-embedding-mode openai --leann-embedding-model voyage-4

# Ollama (local)
papi ask "how is neus different from nerf?" --backend leann --leann-provider ollama --leann-model olmo-3:7b --leann-index papers_ollama_nomic-embed-text

# OpenAI
papi ask "how is neus different from nerf?" --backend leann --leann-provider openai --leann-model gpt-4.1 --leann-api-key $OPENAI_API_KEY --leann-index papers_openai_text-embedding-3-small

# Anthropic + Voyage embeddings
papi ask "how is neus different from nerf?" --backend leann --leann-provider anthropic --leann-model claude-sonnet-4-5 --leann-api-key $ANTHROPIC_API_KEY --leann-index papers_openai_voyage-4

# OpenRouter + Voyage embeddings
papi ask "how is neus different from nerf?" --backend leann --leann-provider openai --leann-model google/gemini-3.5-flash --leann-api-base https://openrouter.ai/api/v1 --leann-api-key $OPENROUTER_API_KEY --leann-index papers_openai_voyage-4

# Gemini (OpenAI-compatible)
papi ask "how is neus different from nerf?" --backend leann --leann-provider openai --leann-model gemini-3-flash-preview --leann-api-base https://generativelanguage.googleapis.com/v1beta/openai/ --leann-api-key $GEMINI_API_KEY --leann-index papers_openai_gemini-embedding-001

# Ollama (local)
papi index --backend leann --leann-embedding-mode ollama --leann-embedding-model nomic-embed-text

# OpenAI
export OPENAI_API_KEY=...
papi index --backend leann --leann-embedding-mode openai --leann-embedding-model text-embedding-3-small --leann-embedding-api-key $OPENAI_API_KEY

# Gemini (OpenAI-compatible)
export GEMINI_API_KEY=...
papi index --backend leann --leann-embedding-mode openai --leann-embedding-model gemini-embedding-001 --leann-embedding-api-base https://generativelanguage.googleapis.com/v1beta/openai/ --leann-embedding-api-key $GEMINI_API_KEY

# Voyage (OpenAI-compatible) — endpoint + key auto-routed from VOYAGE_API_KEY
export VOYAGE_API_KEY=...
papi index --backend leann --leann-embedding-mode openai --leann-embedding-model voyage-4

[leann]
index_by_embedding = false

[leann]
llm_provider = "ollama"
llm_model = "qwen3:8b"
embedding_model = "nomic-embed-text"
embedding_mode = "ollama"

[leann]
llm_model = "openrouter/deepseek/deepseek-v4-pro"

papi index --backend leann

# Override common LEANN build knobs (maps to `leann build ...`):
papi index --backend leann --leann-embedding-mode ollama --leann-embedding-model nomic-embed-text
papi index --backend leann --leann-embedding-mode ollama --leann-embedding-host http://localhost:11434
papi index --backend leann --leann-doc-chunk-size 350 --leann-doc-chunk-overlap 128

LLM configuration

paperpipe uses LLMs for generating summaries, extracting equations, and tagging. Without an LLM, it falls back to regex extraction and metadata-based summaries.

# Set your API key (pick one)
export GEMINI_API_KEY=...       # default provider
export OPENAI_API_KEY=...
export ANTHROPIC_API_KEY=...
export VOYAGE_API_KEY=...       # for Voyage embeddings (recommended with Claude)
export OPENROUTER_API_KEY=...   # 200+ models

# Override the default model
export PAPERPIPE_LLM_MODEL=gpt-4o
export PAPERPIPE_LLM_TEMPERATURE=0.3  # default: 0.3

Local-only via Ollama

export PAPERPIPE_LLM_MODEL=ollama/qwen3:8b
export PAPERPIPE_EMBEDDING_MODEL=ollama/nomic-embed-text

# Either env var name works (paperpipe normalizes both):
export OLLAMA_HOST=http://localhost:11434
# export OLLAMA_API_BASE=http://localhost:11434

Ollama defaults to a small context window (~4k tokens) and silently truncates longer prompts, so a full paper's LaTeX would be cut off. paperpipe avoids this by sizing the context window to each prompt, capped at 32768 tokens. Lower the cap to save memory, or raise it for very long papers:

export PAPERPIPE_OLLAMA_NUM_CTX=16384   # or set [llm] ollama_num_ctx in config.toml

This applies to local and Ollama Cloud (ollama/<model>:cloud, after ollama signin) models.

Many reasoning-capable models "think" by default and can spend their whole output budget on hidden reasoning, returning empty content for extraction prompts. paperpipe disables thinking for Ollama models by default since it does structured extraction, not reasoning. Re-enable it if you want:

export PAPERPIPE_OLLAMA_THINK=true   # or set [llm] ollama_think in config.toml

Via the Claude Code CLI (no API key)

If you have the Claude Code CLI installed and signed in (including subscription/OAuth logins), paperpipe can route summary/equation/tag/title generation through it instead of an API key:

export PAPERPIPE_LLM_MODEL=claude-cli/sonnet   # or claude-cli/opus, claude-cli/<full-model-name>

paperpipe shells out to claude -p for each generation step using the CLI's own authentication. Notes:

• Summarization only. papi ask (PaperQA2/RAG) and embeddings still need an API key or local Ollama — the CLI backend cannot drive them.
• Calls count against your Claude Code usage/rate limits, not API billing.
• Each invocation boots the CLI runtime (~3s), so generation is slower than an API backend.
• temperature is not configurable on this backend.

Via the Antigravity CLI / Google AI Pro (no API key)

If you have the Antigravity CLI (agy) installed and signed in with your Google AI Pro / Antigravity subscription:

export PAPERPIPE_LLM_MODEL=agy/gemini-3.7-flash   # or agy-cli/gemini-3.7-flash, agy/gemini-3.1-pro
export PAPERPIPE_LLM_REASONING_EFFORT=high        # low, medium, high

Or in ~/.paperpipe/config.toml:

[llm]
model = "agy/gemini-3.7-flash"
reasoning_effort = "high"

paperpipe routes prompts through agy via standard stream-json input using the CLI's subscription authentication. Check which models work with your keys:

papi models                    # probe default models for your configured keys
papi models latest             # probe latest model candidates (gpt-5, Gemini via OpenRouter/Gemini, Claude, Voyage 4)
papi models last-gen           # probe previous generation
papi models all                # probe broader superset
papi models --verbose          # show underlying provider errors

Tagging

Papers are auto-tagged from:

1. arXiv categories (cs.CV → computer-vision)
2. LLM-generated semantic tags (biased toward existing tags for consistency)
3. Your --tags flag

papi add 1706.03762 --tags my-project,priority
papi list --tag attention

# Edit tags on one or more papers (no LLM)
papi regenerate <paper>... --tags nerf,3d        # add
papi regenerate <paper>... --remove-tags 3d      # remove
papi regenerate <paper>... --set-tags nerf,slam  # replace all
papi regenerate <paper>... --clear-tags          # remove all

# Edit tags across the whole database
papi tags --audit                  # find duplicate/similar tags
papi tags --merge old-tag new-tag  # rename a tag across all papers
papi tags --delete junk-tag        # remove a tag from all papers

Non-arXiv papers

papi add ./paper.pdf                                       # local PDF (auto-detected)
papi add "https://example.com/paper.pdf"                   # PDF URL (auto-detected)
papi add --pdf ./paper.pdf --title "My Paper" --no-llm     # --pdf for explicit metadata options
papi add --pdf "https://example.com/paper.pdf" --tags siggraph

Configuration file

For persistent settings, create ~/.paperpipe/config.toml (override location with PAPERPIPE_CONFIG_PATH):

[llm]
model = "gemini/gemini-2.5-flash"
temperature = 0.3
# reasoning_effort = "high"   # reasoning effort (low/medium/high) for supported models
# ollama_num_ctx = 32768   # max context window for ollama/* models (default 32768)
# ollama_think = false     # enable Ollama "thinking" for reasoning models (default false)
# timeout = 120            # per-request seconds; raise for slow local/reasoning models (default 120)

[embedding]
model = "gemini/gemini-embedding-001"

[paperqa]
settings = "default"
index_dir = "~/.paperpipe/.pqa_index"
summary_llm = "gpt-4o-mini"
enrichment_llm = "gpt-4o-mini"
# reasoning_effort = "high"          # reasoning effort for answer LLM
# summary_reasoning_effort = "low"   # reasoning effort for summary LLM
# agent_reasoning_effort = "medium"  # reasoning effort for agent LLM
# Unset, reasoning_effort inherits [llm] reasoning_effort, and agent_reasoning_effort follows
# whichever answer-LLM effort applies. summary_reasoning_effort does not inherit.
# agent_llm drives PaperQA2's search agent. PaperQA2's own default is gpt-4o; paperpipe
# instead inherits the answer llm (above) unless you set this. Set agent_type = "fake" for
# deterministic, low-token retrieval that skips the agent LLM's tool-calling loop.
# agent_llm = "gpt-4o-mini"
# agent_type = "fake"

# Optional: override LEANN separately (otherwise it follows [llm]/[embedding] for openai/ollama model ids)
[leann]
llm_provider = "ollama"
llm_model = "qwen3:8b"
embedding_model = "nomic-embed-text"
embedding_mode = "ollama"

[tags.aliases]
cv = "computer-vision"
nlp = "natural-language-processing"

Precedence: CLI flags > env vars > config.toml > built-in defaults.

Development

git clone https://github.com/hummat/paperpipe && cd paperpipe
pip install -e ".[dev]"
make check                            # format + lint + typecheck + test

# Bump version in pyproject.toml, then:
make release

Credits

• PaperQA2 by Future House — RAG backend. Skarlinski et al., "Language Agents Achieve Superhuman Synthesis of Scientific Knowledge", 2024. arXiv:2409.13740
• LEANN — (local) RAG backend. Wang et al., "LEANN: A Low-Storage Vector Index", 2025. arXiv:2506.08276

License

MIT