Podcast Scraper

Download, transcribe, and summarize podcast episodes. Fetches transcripts from RSS feeds (Podcasting 2.0), generates them when missing, detects speakers, creates summaries, and optionally extracts structured insights (GIL) and knowledge graphs (KG). Use local models (Whisper, transformers) or cloud APIs (OpenAI, Gemini, Anthropic, Mistral, DeepSeek, Grok, Ollama) — your choice.

Learning project: This is a personal project where I'm exploring AI-assisted coding and hands-on work with edge and cloud AI/ML technologies.

Personal use only. Downloaded content must remain local and not be redistributed. See Legal Notice.

---

Features

• Transcript Downloads — Automatic detection and download from RSS feeds

• Episode selection — Order (newest / oldest), optional publish-date window (--since / --until), offset, and max_episodes for large back-catalogs (CONFIGURATION.md, GitHub #521)

• Multi-feed corpus — One config or CLI invocation for multiple shows: feeds / rss_urls in YAML, --feeds-spec for structured feeds.spec.yaml / JSON, or repeatable --rss / legacy --rss-file; isolated output under <output_dir>/feeds/<stable_id>/ per feed. With vector_search, a single parent index is built under <output_dir>/search after all feeds finish; corpus_manifest.json, corpus_run_summary.json, and structured log lines record batch status (RFC-063, CONFIGURATION.md). Inspect offline: python -m podcast_scraper.cli corpus-status --output-dir <corpus_parent>.

• Transcription — Local Whisper, DGX tailnet Whisper, OpenAI, Gemini, Mistral, or Deepgram Nova-3; automatic API audio chunking for oversized files (Audio Pipeline Guide)

• Speaker screenplay — Gap-based or neural diarization (pyannote; default on for local Whisper, --no-diarize to disable); maps to NER-detected host/guest names

• Commercial cleaning — Confidence-scored sponsor/ad detection (patterns + position; optional diarization signals when speaker timeline exists)

• Audio Preprocessing — Optimize audio files before transcription (reduce size, remove silence, normalize loudness)

• Speaker Detection — Identify speakers using spaCy NER, OpenAI, Google Gemini, Grok (real-time info), or other providers

• Summarization — Episode summaries via local transformers (BART/LED), hybrid MAP-REDUCE (hybrid_ml), or any of 7 LLM providers

• GIL Extraction — Structured insights and verbatim quotes with evidence grounding (gi.json per episode, PRD-017)

• KG Extraction — Topic-entity knowledge graphs from transcripts (kg.json per episode, RFC-055)

• Metadata Generation — JSON/YAML metadata per episode

• Resumable — Skip existing files, handle interruptions gracefully

• Provider System — Swap between local and cloud providers via config

• MPS Exclusive Mode — Serialize GPU work on Apple Silicon to prevent memory contention and improve reliability (enabled by default)

• Reproducibility — Deterministic runs with seed control, pinned model revisions, and comprehensive run manifests (Issue #379)

• Operational Hardening — Configurable HTTP retries (media vs RSS), application-level episode retries on transient errors, timeout enforcement, --fail-fast / --max-failures, structured JSON logging, and failure_summary in run.json when episodes fail (CONFIGURATION.md — Download resilience, Issue #379)

• Security — Path validation, model allowlist validation, safetensors format preference, and trust_remote_code=False enforcement (Issue #379)

• Diagnostics — doctor command for environment validation and dependency checks (Issue #379)

• Hybrid corpus search — Two-tier (transcript segments + GIL insights) BM25 + dense vector via RRF over LanceDB, with compound results; the single search path when a two-tier index exists (FAISS retired in #995, ADR-099) (RFC-090). Enabled by vector_search; search / index CLIs, and semantic gi explore --topic (guide, RFC-061)

• Run Tracking — Per-episode stage timings, run summaries, episode index files, and metrics.json fields for download retries (http_urllib3_retry_events, episode_download_retries) (Issue #379)

• Live pipeline monitor — Optional --monitor: separate process + rich dashboard (or .monitor.log if stderr is not a TTY) and .pipeline_status.json. Optional pip install -e ".[monitor]": --memray for heap profiling; with monitor + TTY, f runs py-spy to debug/flamegraph_*.svg (RFC-065, guide, #512)

• GI / KG Viewer (v2) — Optional browser UI: graph visualization, dashboard, semantic search, explore, Corpus Library (feed/episode browser, RFC-067), and Corpus Digest (time-windowed highlights, RFC-068) — all against a pipeline output folder (RFC-062; see below). Cross-surface navigation is owned by a single graph handoff FSM (ADR-094, RFC-085) with envelope contract, generation-token concurrency, stuck detection, and visible error UX.

• Generic MCP server — Optional agent-facing tool surface: expose the platform's read capabilities (hybrid search, the relational-query layer, CIL intelligence, catalog) as composable, read-only MCP tools so agentic clients (Claude Desktop/Code, Cursor) use a corpus as a tool — resolve an entity, search for grounded evidence, ask "who said what across shows", pull a person's positions or position-arc, all provenance-bearing (PRD-034, RFC-095). Two transports: stdio for local agents (podcast mcp --corpus <dir> --transport stdio) and Streamable HTTP for remote, bring-your-own-model agents over the public edge — a hosted OAuth 2.1 authorization server + personal-access tokens + the mcp_access entitlement, connectable from claude.ai / Claude Code / Cursor (RFC-112). Ships in [dev]; needs [search] for retrieval. See the Server Guide — MCP server + Player launch — MCP.

• Personal corpus + knowledge export — a read-time definition of what a user has engaged with (experienced = heard ∪ captured; saved = save-for-later), versioned by a per-user revision counter + change log (RFC-114), plus a graph-aware Obsidian vault export (wikilinked, incremental, bridge-only) via /api/app/export (RFC-113).

• Operator playbook — Always-on Hetzner VPS hosting design with Tailscale-only ingress (RFC-082, PROD_RUNBOOK), version-aware corpus snapshot backup + restore (RFC-084, ADR-092, make restore-corpus / restore-corpus-prod), instance-to-instance corpus portability without going through the backup repo (make export-corpus / make import-corpus; laptop ↔ VPS, prod ↔ codespace, airgapped — Corpus airgap runbook), a ledgered corpus-upgrade framework (#862, CORPUS_UPGRADE, make upgrade-check / upgrade-corpus) that carries deployed corpora across release boundaries and is auto-run on prod restore (#1176), and a canonical stack contract (ADR-093) covering dev / codespace / pre-prod / prod with one health surface and one stack-test discipline.

• Three-tier viewer test pyramid — Tier-1 mocks, Tier-2 production-shaped fixtures, Tier-3 real corpus via make ci-ui-validation CORPUS=… (ADR-095, RFC-086). Real-bug-first matrix rule keeps coverage honest.

Repository layout (Python + web)

The Python package (CLI, pipeline, FastAPI) lives at the repo root (pyproject.toml, Makefile, tests/). The GI/KG viewer is a Node app under web/gi-kg-viewer/ (package.json, Vite, Vitest, Playwright). They share one repository but use separate install steps and different .env.example files (root vs web/gi-kg-viewer/) by design.

Onboarding in one place: Polyglot repository guide.

---

GI / KG Viewer (optional)

Browse Grounded Insight and Knowledge Graph artifacts, use semantic search and explore on your corpus, and view a dashboard—all against the same --output-dir you use for the pipeline.

What you need

Run the server (typical)

From the repository root, with your virtualenv active:

pip install -e ".[dev]"
cd web/gi-kg-viewer && npm install && npm run build && cd ../..
python -m podcast_scraper.cli serve --output-dir /path/to/your/run

Then open http://127.0.0.1:8000 (default port). In the sidebar, set Corpus root folder to that same directory, click List files, select .gi.json / .kg.json, and Load selected into graph.

FastAPI / HTTP API: Full /api/* reference, architecture, and tests — Server Guide. With the server running, interactive OpenAPI (Swagger) is at /docs and /openapi.json.

Development (API + hot-reload UI): make serve SERVE_OUTPUT_DIR=/path/to/your/run runs the FastAPI app and the Vite dev server together (UI proxied to the API). See web/gi-kg-viewer/README.md and the Server Guide above.

Testing the viewer: make test-ui (Vitest unit tests, ~1s) and make test-ui-e2e (Playwright browser E2E, Firefox).

---

Quickest start: Docker Compose

The fastest way to try the platform end-to-end — no Python venv, no Node install, no provider keys required. You get the viewer, the API, and an on-demand pipeline runner all wired together; you drive the platform from your browser.

make stack-test-build      # build api + viewer + pipeline images (~5–15 min first time)
make stack-test-up         # bring up viewer (8090) + api + bundled mock RSS fixtures

Then open http://127.0.0.1:8090, open Configuration → Feeds, add an RSS URL (or use the bundled fixture http://mock-feeds/p01_fast_with_transcript.xml), pick a profile under Job Profile (airgapped_thin runs locally with Whisper + transformers, no cloud calls), click Save, then Dashboard → Pipeline → Run pipeline job. Artifacts populate the Library, Digest, Search, and Graph tabs as the job runs.

To use cloud providers (OpenAI Whisper API + Gemini for everything else — faster + better quality), put OPENAI_API_KEY and GEMINI_API_KEY in a repo-root .env file, then:

make stack-test-build-cloud   # adds the cloud-thin pipeline-llm image
# in the UI, choose the "cloud_thin" profile under Job Profile and Save

Tear down (preserves the corpus): make stack-test-down. Tear down + drop everything: make stack-test-down STACK_TEST_DOWN_VOLUMES=1.

Full guide with architecture, troubleshooting, and production hints: Docker Compose guide.

---

Quick Start (Python)

For development, scripted CLI runs, or when you'd rather not run the whole stack — install the package in a venv and use the CLI directly. The Docker Compose path above is the recommended way to operate the platform; the Python path below is for development and headless CLI scripting.

Requirements

• Python 3.10+ — Check with python3 --version
• ffmpeg (only needed for local Whisper transcription):
  • macOS: brew install ffmpeg
  • Linux: apt install ffmpeg or yum install ffmpeg
  • Windows: Download from ffmpeg.org
  • Note: Not required if using OpenAI providers only
• Graphviz (only needed to regenerate architecture diagrams locally):
  • macOS: brew install graphviz
  • Linux: apt install graphviz or yum install graphviz
  • Windows: Download from graphviz.org
  • Note: Required for make visualize. Diagrams must be committed; make ci / make ci-fast and CI run visualize and fail if they are stale.

Installation Options

Choose the installation method based on your use case:

Quick decision guide:

• Using LLM APIs (OpenAI, Gemini, etc.)? → pip install -e ".[llm]"
• Want to run models locally (Whisper, spaCy, Transformers)? → pip install -e ".[ml]"
• Want both? → pip install -e ".[ml,llm]" (recommended)
• Contributing / developing? → pip install -e ".[dev,ml,llm]"

Note: LLM provider SDKs (openai, google-genai, anthropic, mistralai, httpx) are not in core — they require the [llm] extra. Core (pip install -e .) gives you the pipeline framework only.

vector_search and embedding cache

Corpus indexing (vector_search, LanceDB index) calls the embedder with allow_download=False, so sentence-transformers weights must already be on disk in the Hugging Face hub cache. Before offline or sandbox runs, run make preload-ml-models and omit SKIP_GIL=1 so evidence embeddings (including the model named by vector_embedding_model) are pre-cached; GIL’s gi_embedding_model is checked separately when GIL uses transformers. If you set HF_HUB_CACHE, use the same value when preloading and when running the pipeline. See Semantic Search Guide.

Install

Tip: For end users who just want to run the tool, consider using pipx or uv for easier installation. See the Installation Guide for all methods.

Method 1: pip (Standard - Recommended for Development)

Use the latest released version for normal usage.

For LLM API users (OpenAI, Gemini, etc. — no local ML):

# Clone the repository
git clone https://github.com/chipi/podcast_scraper.git
cd podcast_scraper
git checkout <latest-release-tag>   # e.g. v2.6.0

# Create and activate virtual environment
python3 -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Verify Python version (must be 3.10+)
python --version  # Should show Python 3.10.x or higher

# CRITICAL: Upgrade pip and setuptools before installing
pip install --upgrade pip setuptools wheel

# Install with LLM provider SDKs (OpenAI, Gemini, Anthropic, Mistral, etc.)
pip install -e ".[llm]"

For local ML users (Whisper, spaCy, Transformers):

# Same setup as above, then:
# Install package with ML dependencies (includes Whisper, spaCy, Transformers)
pip install -e ".[ml]"

Note: LLM provider SDKs require the [llm] extra. For both local ML and LLM APIs: pip install -e ".[ml,llm]".

Method 2: pipx (Recommended for End Users)

For isolated installation without managing virtual environments:

For LLM-only users:

# Install pipx (if not already installed)
# macOS: brew install pipx
# Linux: pip install --user pipx && pipx ensurepath

# Clone and install with LLM SDKs
git clone https://github.com/chipi/podcast_scraper.git
cd podcast_scraper
pipx install -e ".[llm]"

# Verify
python -m podcast_scraper.cli --help

For local ML users:

# Same as above, but install with ML dependencies:
pipx install -e ".[ml]"

See Installation Guide for details.

Method 3: uv (Fast Installation)

For faster installation using uv:

For LLM API users:

# Install uv (if not already installed)
# macOS/Linux: curl -LsSf https://astral.sh/uv/install.sh | sh
# macOS: brew install uv

# Clone and install with LLM SDKs
git clone https://github.com/chipi/podcast_scraper.git
cd podcast_scraper
uv venv
source .venv/bin/activate
uv pip install -e ".[llm]"

For local ML users:

# Same as above, but install with ML dependencies:
uv pip install -e ".[ml]"

See Installation Guide for details.

Method 4: Docker (Containerized)

For most users the Quickest start: Docker Compose path above is the recommended way to run the platform — it brings up the viewer + API + on-demand pipeline jobs as a single stack.

The single-container docker run style below is appropriate when you want the pipeline as a long-running service triggered by an external scheduler (cron, systemd, supervisord), not via the operator UI.

LLM-only variant (small, fast, ~500 MB):

# Build LLM-only image
docker build --build-arg INSTALL_EXTRAS=llm -f docker/pipeline/Dockerfile -t podcast-scraper:llm .

# Run with config file
docker run -v ./config.yaml:/app/config.yaml \
           -v ./output:/app/output \
           -e OPENAI_API_KEY=sk-your-key \
           podcast-scraper:llm

ML-enabled variant (full features, ~3-5 GB):

# Build ML-enabled image (default)
docker build --build-arg INSTALL_EXTRAS=ml -f docker/pipeline/Dockerfile -t podcast-scraper:ml .

# Run with config file
docker run -v ./config.yaml:/app/config.yaml \
           -v ./output:/app/output \
           podcast-scraper:ml

The default diarization model (community-1, v4) is non-gated — no Hugging Face token needed, and the ML image bakes it in. A token is only required if you switch to the gated pyannote/speaker-diarization-3.1 fallback (whose segmentation-3.0 is also gated and cannot be redistributed in a published image). For that fallback, accept the terms on Hugging Face, then pass your token and a persistent cache so the one-time download survives restarts:

docker run -v ./config.yaml:/app/config.yaml \
           -v ./output:/app/output \
           -v ~/.cache/huggingface:/root/.cache/huggingface \
           -e HF_TOKEN=hf_... \
           podcast-scraper:ml

Without a token the pipeline falls back to gap-based screenplay formatting (or run with --no-diarize).

See Docker Compose guide for the recommended end-to-end stack flow, Docker Service guide for the single-container deployment style, and Docker Variants guide for image-tier (ml vs llm) trade-offs.

Development (main)

Use this if you are contributing or experimenting. This branch may contain unreleased changes.

# Clone the repository
git clone https://github.com/chipi/podcast_scraper.git
cd podcast_scraper

# Create and activate virtual environment
python3 -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Verify Python version (must be 3.10+)
python --version  # Should show Python 3.10.x or higher

# CRITICAL: Upgrade pip and setuptools before installing
# This is required for editable installs with pyproject.toml
pip install --upgrade pip setuptools wheel

# Install package with ML dependencies
pip install -e ".[ml]"

Important Notes:

• Python 3.10+ is REQUIRED — The project uses features that require Python 3.10 or higher. Always verify with python --version after activating the venv.
• Installation is required — You must run pip install -e . (or pip install -e ".[ml]" for ML) before running CLI commands. Without it, you'll get ModuleNotFoundError: No module named 'podcast_scraper'.
• LLM API users — If you're using LLM-based providers (OpenAI, Gemini, Anthropic, etc.), install with pip install -e ".[llm]". LLM provider SDKs are not in core — the [llm] extra is required.
• Local ML users — If you want to use local Whisper, spaCy, or Transformers, install with pip install -e ".[ml]" to get ML dependencies.
• Upgrade pip/setuptools first — If you see "editable mode currently requires a setuptools-based build" error, run pip install --upgrade pip setuptools wheel and try again.
• Always activate the venv — Remember to activate your virtual environment (source .venv/bin/activate) before running any commands.

Configure Environment Variables (Required for LLM Providers)

If you plan to use LLM-based providers (OpenAI, etc.) for transcription, speaker detection, or summarization, you must set up a .env file with your API key:

# Copy the template
cp config/examples/.env.example .env

# Edit .env and add your LLM API key (REQUIRED for LLM providers)
# For OpenAI: OPENAI_API_KEY=sk-your-actual-api-key-here
# For Gemini: GEMINI_API_KEY=your-gemini-api-key-here

Important variables:

• OPENAI_API_KEY - Required if using OpenAI providers (transcription, speaker detection, or summarization)
• GEMINI_API_KEY - Required if using Gemini providers (transcription, speaker detection, or summarization)
• DEEPGRAM_API_KEY - Required if using Deepgram transcription (--transcription-provider deepgram)
• HF_TOKEN - Required for neural speaker diarization (pyannote gated models) when diarize is on (default for local Whisper)
• LOG_LEVEL - Controls logging verbosity (DEBUG, INFO, WARNING, ERROR, CRITICAL)
• OUTPUT_DIR - Custom output directory (default: ./output/)
• CACHE_DIR - ML model cache location (only needed for local ML providers)
• Performance tuning variables (WORKERS, TIMEOUT, etc.)

See config/examples/.env.example for all available options and detailed documentation.

Verify Installation

Before running, verify the installation worked:

# Make sure venv is activated
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Test that the package is installed
python -c "import podcast_scraper; print('[ok] Installation successful')"

# Check CLI is available
python -m podcast_scraper.cli --help

Run

Prerequisite: Make sure you've completed the installation steps above and activated your virtual environment.

Typical run: profile + operator config + feed list

After install, most people want packaged defaults (profile), their own knobs (operator YAML), and where the RSS URLs live (feeds file or spec). The CLI wires all three together; you do not need to merge YAML by hand.

# From the repo root, venv active, API keys in .env or the environment:
python -m podcast_scraper.cli \
  --profile cloud_balanced \
  --config config/manual/operator_defaults.yaml \
  --feeds-spec config/manual/feeds.spec.registry_10.yaml

Do not combine --feeds-spec with --rss-file or with explicit RSS URL arguments in the same invocation (the CLI rejects that mix).

Alternative: --config config/profiles/cloud_balanced.yaml alone is valid when the feed URL(s) live inside that YAML (rss / rss_urls / feeds) or you pass a URL on the command line. Use --profile + --config when your operator file is small (paths, limits, flags) and the profile stays the packaged preset.

In-repo references: config/examples/feeds.spec.example.yaml, config/manual/operator_defaults.yaml, config/profiles/cloud_balanced.yaml. Deeper detail: CONFIGURATION.md (RSS / multi-feed), RSS_GUIDE.md.

Basic Usage with Example Config (Recommended for First-Time Users)

The easiest way to get started is using an example config file:

# Make sure venv is activated
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Set your API key (if using LLM providers)
export OPENAI_API_KEY=sk-your-actual-api-key-here

# Run with an example config file
python -m podcast_scraper.cli --config config/examples/config.example.yaml

Available example config:

• config/examples/config.example.yaml — Example configuration with local ML providers (Whisper, spaCy, Transformers)

To customize: Copy the example config and edit the rss field with your podcast feed URL:

# Copy the example config
cp config/examples/config.example.yaml my-config.yaml

# Edit my-config.yaml and change the RSS feed URL
# Then run:
python -m podcast_scraper.cli --config my-config.yaml

Advanced Usage (Command-Line Options)

Replace https://example.com/feed.xml with your podcast's RSS feed URL.

For LLM-only users (no ML dependencies):

# Make sure venv is activated
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Set LLM API key (if not in .env file)
# For OpenAI:
export OPENAI_API_KEY=sk-your-actual-api-key-here

# For Gemini:
export GEMINI_API_KEY=your-gemini-api-key-here

# Download transcripts (automatically generates missing ones with LLM API)
# Using OpenAI:
python -m podcast_scraper.cli https://example.com/feed.xml \
  --transcription-provider openai

# Using Gemini:
python -m podcast_scraper.cli https://example.com/feed.xml \
  --transcription-provider gemini

# Only download existing transcripts (skip transcription)
python -m podcast_scraper.cli https://example.com/feed.xml \
  --no-transcribe-missing

# Full processing with OpenAI providers: transcripts + speaker detection + summaries + metadata
python -m podcast_scraper.cli https://example.com/feed.xml \
  --transcription-provider openai \
  --speaker-detector-provider openai \
  --summary-provider openai \
  --generate-metadata \
  --generate-summaries

# Full processing with Gemini providers:
python -m podcast_scraper.cli https://example.com/feed.xml \
  --transcription-provider gemini \
  --speaker-detector-provider gemini \
  --summary-provider gemini \
  --generate-metadata \
  --generate-summaries

For local ML users (with ML dependencies installed):

# Make sure venv is activated
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Download transcripts (automatically generates missing ones with local Whisper)
python -m podcast_scraper.cli https://example.com/feed.xml

# Full processing with local ML: transcripts + speaker detection + summaries + metadata
python -m podcast_scraper.cli https://example.com/feed.xml \
  --generate-metadata \
  --generate-summaries

Using a config file:

See the Basic Usage with Example Config section above for the recommended approach. You can also use any config file from the config/examples/ directory or create your own.

Output: Files are organized in output/ with subdirectories:

• transcripts/ — Transcript files (RSS may serve .vtt/.srt; the pipeline normalizes those to .txt plus *.segments.json for GI quote audio timing when cues parse cleanly; plain .txt/.html feeds have no segment timing unless you add a sidecar). When save_adfree_transcript is enabled (default) the pipeline also writes the ad-free processing base — *.adfree.txt + *.adfree.segments.json + *.adfree.admap.json — which GI, search, and enrichment prefer for exact char offsets (#974); the raw .txt is the untouched canonical source.
• metadata/ — JSON/YAML metadata, plus gi.json (GIL) and kg.json (KG) when enabled
• search/ — LanceDB search index under search/lance_index/ (when vector_search is enabled)
• run.json, index.json, metrics.json — Run tracking artifacts

Multi-feed corpora add feeds/<stable_id>/ per feed, plus corpus_manifest.json and corpus_run_summary.json at the corpus root.

Use --output-dir to customize the location (default: ./output/).

Troubleshooting Installation

Problem: ModuleNotFoundError: No module named 'podcast_scraper'

Solution: Make sure you:

1. Activated the virtual environment: source .venv/bin/activate
2. Installed the package: pip install -e ".[ml]"
3. Are using the venv's Python: python -m podcast_scraper.cli (not python3 if system Python is different)

Problem: "editable mode currently requires a setuptools-based build"

Solution: Upgrade pip and setuptools first:

pip install --upgrade pip setuptools wheel
pip install -e ".[ml]"

Problem: Python version is < 3.10

Solution: Create venv with a newer Python:

# Find available Python versions
python3.11 --version  # or python3.12, etc.

# Create venv with specific version
python3.11 -m venv .venv
source .venv/bin/activate

For more help, see Troubleshooting Guide.

---

Documentation

Contributing? See CONTRIBUTING.md.

---

License

MIT License — see LICENSE.

Note: The license applies to source code only, not to podcast content downloaded using this software. See Legal Notice.