---

title: Sketchnote emoji: ✏️ colorFrom: gray colorTo: blue sdk: gradio sdk_version: 6.18.0 app_file: app.py python_version: "3.11" pinned: false tags:

• build-small-hackathon
• gradio
• openbmb
• nvidia
• video

add the correct track + badge tags per the hackathon field guide

---

✏️ Sketchnote

Turn a PDF (e.g. a textbook) into a whiteboard sketch-animation video with synced voice narration — chapter by chapter. Upload a PDF, pick a chapter range, and Sketchnote reads each chapter's summary aloud while a hand-drawn-style sketch animation plays in sync. Think "Golpo.ai, but fully local and open-source."

This is a submission to the Hugging Face "Build Small" hackathon. Every AI model is open-weight, self-hosted, and under 32B parameters — no proprietary hosted model APIs are used for any AI work (text, speech, image, or parsing).

Why it fits the rules

Models (every one < 32B — proof)

All AI models are individually well under the 32B cap (largest = 8B).

How it works

User → Gradio Space
        │  upload PDF + choose chapter range
        ▼
  Ingestion:
    • PyMuPDF first — text + TOC for clean digital PDFs (fast, in-Space)
    • If no TOC / scanned / complex → render pages to images,
      send to Nemotron Parse (Modal GPU) → structured text + title/section
      classes used to recover chapter structure  (results cached)
        │  for each chapter:
        ▼
  Modal GPU ── MiniCPM4.1-8B ──► { narration_script, visual_concepts[] }
        ▼
  Visual builder: whiteboard concept card  (optional: SDXL-Turbo) → PNG
        ▼
  Kokoro-82M → narration WAV  (audio FIRST → measures duration)
        ▼
  ai-img2sketch (OpenCV) → sketch reveal sized to the audio duration
        ▼
  ffmpeg: mux audio onto the sketch clip → per-chapter MP4
        ▼
  ffmpeg: concatenate chapter clips → final MP4
        ▼
  Gradio shows the final video + per-chapter transcript

Compute split: heavy models (Nemotron Parse, MiniCPM, optional SDXL-Turbo) run on Modal serverless GPUs; light work (Kokoro, OpenCV, ffmpeg, UI) runs inside the Space. If Modal isn't deployed, Sketchnote falls back to a non-AI extractive summary so it still produces a video.

Repository layout

app.py            Gradio UI + pipeline orchestration
modal_app.py      Modal GPU functions: summarize_chapter, parse_pages, generate_image
pipeline/
  config.py       env + paths + ffmpeg resolver (tokens via os.environ, never logged)
  pdf_parser.py   PyMuPDF fast path + Nemotron hard path → chapters (+ parse cache)
  llm.py          Modal client + prompt templates + extractive fallback
  tts.py          Kokoro narration → (wav, duration)
  visuals.py      whiteboard concept card (+ optional SDXL-Turbo)
  sketch.py       vendored ai-img2sketch reveal, sized to narration
  video.py        ffmpeg mux + concat
assets/sample.pdf a short sample for the demo
requirements.txt  Space (in-Space) deps    packages.txt  ffmpeg, espeak-ng
.env.example      infra-token placeholders  .gitignore   includes .env

Setup & run

# 1. credentials (infra only — HF + Modal)
cp .env.example .env        # then fill HF_TOKEN, MODAL_TOKEN_ID, MODAL_TOKEN_SECRET

# 2. environment (Python 3.11 recommended)
python -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -r requirements.txt
# system deps (Linux): the Space installs these via packages.txt
#   sudo apt-get install ffmpeg espeak-ng

# 3. deploy the GPU models to Modal (self-hosting our own open weights)
modal deploy modal_app.py
modal run modal_app.py      # optional smoke test of summarize_chapter

# 4. run the Gradio app
python app.py               # open the printed local URL

If Modal is not deployed, the app still runs end-to-end using PyMuPDF ingestion + the non-AI extractive summarizer (degraded, but never hard-fails).

Sponsor models

• OpenBMB — MiniCPM4.1-8B is the primary summarizer that writes each chapter's narration script and visual concepts.
• NVIDIA — Nemotron Parse v1.1 upgrades our weakest step: ingesting scanned / multi-column / complex PDFs and classifying titles & sections to drive chapter splitting. We self-host its open weights on Modal — we do not call NVIDIA's hosted NIM / build.nvidia.com service.

Credits & licenses

• Sketch animation adapted from ai-img2sketch by DasLearning (https://github.com/daslearning-org/ai-img2sketch) and the storyboard-ai project by Yogendra Yatnalkar. Credit preserved here and in pipeline/sketch.py; see those projects for their original licenses.
• NVIDIA Nemotron Parse v1.1 — NVIDIA Open Model / Community Model License (tokenizer under CC-BY-4.0). See the model card.
• MiniCPM4.1-8B — OpenBMB model license (see model card).
• Kokoro-82M — Apache-2.0 (hexgrad/Kokoro-82M).

Manual checklist (for the maintainer)

1. cp .env.example .env and fill HF_TOKEN, MODAL_TOKEN_ID, MODAL_TOKEN_SECRET.
2. modal deploy modal_app.py (first call downloads weights to a Modal Volume).
3. Create the public HF Space in the hackathon org (Gradio SDK) and push this repo.
4. Add the correct hackathon track + badge tags to the frontmatter.
5. Record the demo video.