VoxWeave

BGM-robust subtitles for anime, film, and clips.

Vocal separation and song-skip so ASR never hallucinates on background music, OP/ED, or insert songs. Local-first Qwen3 ASR, forced alignment, and edit-and-resync — CJK-aware.

https://github.com/user-attachments/assets/e75b6dd3-fa37-4afe-89db-b6ee2c28f6bc

_{Sliced clip under heavy BGM · voxweave Test.mp4 · Qwen3-ASR-1.7B}

[!NOTE] Local-first. Separation, ASR, and forced alignment all run in-process on your GPU — no network endpoints, no audio leaves the machine. Runs on NVIDIA CUDA (PyTorch) and on Apple Silicon, where ASR and the Qwen aligner use the native MLX Qwen3 models. Weights download once on first run. (Translation and ASR-correction are the only optional features that call an external LLM, and only when you invoke them.)

[!NOTE] Hardware. The default pipeline (Qwen3-ASR-0.6B, peak load strategy) runs in ~8 GB of VRAM — the separator is freed before ASR + alignment load, so peak ≈ max(stage), not their sum. --asr-model qwen3-asr-1.7B adds roughly +2 GB and still fits 8 GB under the default peak strategy. load_strategy = "sum" (keeps models resident between passes, skipping reloads on big cards) makes peak the sum of the resident models — plan for 12 GB+. On Apple Silicon the MLX 8-bit weights roughly halve the Qwen footprint (so 1.7B fits comfortably in 16 GB unified memory). --hybrid loads a Whisper engine alongside Qwen to trade VRAM for accuracy — it does not save memory; the only knob that lowers it is staying on 0.6B + the peak strategy.

VoxWeave derives from the WhisperX "edit-and-resync" workflow: transcribe once, then edit the text and re-align it against the original audio for frame-accurate timestamps. Where it differs is the front end — vocal separation and song-skip keep background music out of the ASR, and a CJK-aware layout/alignment stack (MMS-300m for Japanese, BudouX/jieba for line breaks) handles Chinese/Japanese/English as first-class.

Contents

• Why VoxWeave
• Setup
• Quickstart
• Usage
  • Transcribe (voxweave <media>)
  • Label speakers (speakers)
  • Re-align after editing (align)
  • Re-layout offline (render)
  • ASR correction (correct)
  • Translate (translate)
  • Export (export)
  • Pack soft subtitles (pack)
  • Burn hard subtitles (burn)
• The edit-and-resync workflow
• Migration notes
• How it works
• Configuration
  • Performance knobs
• Data contract
• Testing
• Support
• License
• Acknowledgments

Why VoxWeave

• BGM removal before ASR. A Mel-Band Roformer vocal separator (pure torch, full-band 44.1k) strips music first, so ASR doesn't transcribe lyrics or hallucinate on score.
• Song-skip. PANNs detects singing/music on the separated vocals and skips OP/ED and insert songs before ASR — on by default, --no-skip-songs to keep them.
• Local Qwen3 ASR + forced alignment. Text and word-level timestamps in one pass, fully on-device — in-process PyTorch on NVIDIA, or the native MLX Qwen3 models on Apple Silicon. A Whisper hybrid engine is also available for when you prefer Whisper text (faster-whisper on NVIDIA, the native MLX Whisper port on Apple Silicon).
• Edit-and-resync. Fix the transcript by hand, then align re-derives timestamps from the audio — timestamps are never hand-written.
• CJK-aware. Japanese aligns with MMS-300m + uroman (zero-OOV, immune to the per-cue drift that breaks wav2vec2-xlsr on rare kanji); line breaks use BudouX phrase atoms + jieba.
• Optional LLM steps. correct cleans up ASR typos/garbled names before alignment; translate does whole-episode context-aware translation while preserving cue count (dual-speaker -line/-line cues are translated one speaker at a time and re-assembled).
• Ship the result. pack soft-muxes finished VTTs into the media as titled subtitle tracks (instant stream copy); burn hardcodes them at constant quality with NVENC / VideoToolbox acceleration, matching the source bit depth.

Setup

Two install variants: voxweave[cuda] (NVIDIA GPU — Blackwell sm_120 / cu128 by default) and voxweave[mps] (Apple Silicon / macOS). Both need ffmpeg on PATH.

# Ubuntu / Debian
sudo apt update && sudo apt install ffmpeg
# Arch Linux
sudo pacman -S ffmpeg
# macOS (Homebrew)
brew install ffmpeg

Install from PyPI (puts the global voxweave command on PATH):

# NVIDIA / Linux:
uv tool install --torch-backend=cu128 \
  --overrides <(printf "onnxruntime; sys_platform == 'darwin'\n") \
  "voxweave[cuda]"                                       # full pipeline + faster-whisper hybrid
# Apple Silicon / macOS:
uv tool install "voxweave[mps]"                          # full pipeline + MLX Whisper hybrid

--torch-backend in uv tool install needs uv 0.9.19 or newer (uv self update). On Linux the --overrides line keeps the CPU onnxruntime wheel (pulled in by ctc-forced-aligner) from shadowing onnxruntime-gpu, which would silently move Japanese MMS alignment onto the CPU; the Makefile passes the same rule from overrides.txt.

The full local pipeline — vocal separation, ASR, forced alignment (incl. MMS-300m for Japanese/CJK), layout, song-skip, and speaker-diarization support — plus CJK line-break and translation — is baked into the core dependencies. The variant selects the compute platform and the ASR/alignment backend:

• [cuda] (NVIDIA/Linux): the in-process PyTorch Qwen3-ASR + forced aligner (qwen-asr), GPU onnxruntime (CUDAExecutionProvider for MMS alignment), and the faster-whisper hybrid engine.
• [mps] (Apple Silicon/macOS): ASR runs on the native MLX Qwen3-ASR from mlx-audio (Metal kernels + quantization). Alignment keeps the same per-language stack as [cuda]: English on wav2vec2 CTC (torch, runs on MPS; the forced-align DP falls to CPU as torchaudio has no Metal kernel), Japanese/CJK on the ONNX MMS aligner (CPU onnxruntime; CoreML is avoided because its Metal context crashes alongside MLX). Only the Qwen fallback (zh·yue, or any CTC failure) is served by the MLX Qwen3-ForcedAligner, since the torch qwen-asr aligner is absent here. The Whisper hybrid/fusion engines (--asr-model large-v3, --hybrid) run on the native mlx-whisper Metal port instead of faster-whisper (ctranslate2 has no Metal backend). Vocal separation (MelBandRoformer) + PANNs song-skip stay on torch-MPS. qwen-asr is excluded because its transformers==4.57.6 pin conflicts with mlx-audio, so [cuda] and [mps] are mutually exclusive — pick one per host.

Speaker diarization support ships by default on both variants but remains opt-in at runtime. The default model is pyannote/speaker-diarization-community-1, which separates and counts multiple speakers noticeably better than 3.1. It is gated separately on Hugging Face: before using --diarize, accept the conditions for the model you select, then set VOXWEAVE_HF_TOKEN / HF_TOKEN or authenticate once with hf auth login (a uv tool install does not put hf on PATH; run it as uvx --from huggingface_hub hf auth login). Users who have accepted only the 3.1 gate can keep it with --diarize-model 3.1.

Measured with pyannote.audio 4.0.7 on the same Japanese 16 kHz test waveform and an RTX PRO 4000 24 GB. Peak allocated CUDA memory was identical in that run:

Community-1 generally improves speaker counting and separation; 3.1 stays selectable per run for users who have accepted only the existing 3.1 gate.

Voiceprint embedders. With --voiceprints, pyannote still finds the speaker turns, but the per-speaker voiceprints (cross-episode matching and Split this speaker) come from a dedicated speaker-embedding model chosen per language. Each checkpoint is downloaded once into ~/.cache/voxweave/audio/, pinned by size and SHA-256, and verified at the start of a --voiceprints run, before any audio work (with auto and no --language, both checkpoints). If that fails, the run warns and continues without voiceprints:

auto (the default) picks anime-va for Japanese episodes and redimnet2 otherwise. Voice stores are per embedding space: a store built by one embedder never matches another. Because the dedicated embedders are independent of the diarizer, switching --diarize-model no longer orphans a store built by redimnet2 or anime-va.

Voiceprint speaker clustering (opt-in). By default, --diarize keeps pyannote's own answer to who is who. Enable the alternative with --speaker-clustering voiceprint (or VOXWEAVE_DIARIZE_CLUSTERING=voiceprint, or [diarize].clustering = "voiceprint" in the config). pyannote still finds the speaker turns (who speaks when, including overlaps); the voiceprint-v1 recipe then regroups them with the redimnet2 checkpoint from the table above, whatever the episode's language: the same download and cache, and the same CC BY-NC-SA 4.0 (non-commercial) weights licence. The recipe embeds the speech of each turn that no other speaker talks over, clusters the turns with at least 1 s of such speech (turns that mostly overlap in time never share a speaker, and a cluster needs 6 s of that speech to count as a speaker), then gives every other turn to the closest voice nearby, or drops it from speaker_turns when no voice is close enough (background voices, noise, fragments too short to tell); the subtitle words a dropped turn covered stay with the surrounding speaker. Measured against pyannote's clustering, it confused speakers less on three of four public test sets (AliMeeting, VoxConverse and JVS-conv; slightly more on AMI) and attributed short backchannels in an online meeting correctly more often (37 of 40 clips under 1 s, against 32), but it can split one person into more speakers than pyannote does, and a speaker who talks very little may be merged into others or dropped. --min-speakers/--max-speakers bind this stage too. A --min-speakers above the number of voices it finds is met by dividing those voices, because someone with only a few seconds of speech cannot form a speaker of their own. pyannote does the same under that bound, and measured worse: given the true speaker count on the public recordings where the stage finds too few, the stage's DER rose from 19% to 23% and pyannote's from 19% to 30%. Set it only when a missing speaker matters more than a split one. If the stage fails (a download, out of memory), cannot meet the bounds, or finds no turn long enough to anchor a voiceprint, the run warns and keeps pyannote's speakers. With --voiceprint-model pyannote the run keeps pyannote's clustering, because those legacy voiceprints are keyed by pyannote's labels. The setting never changes which voice stores an episode's voiceprints match. Switching it (or --diarize-model) on an episode whose speakers you already named renumbers the speakers, while the saved names stay keyed by speaker id: a transcription run warns when a named id's turns changed, and you should review the names with voxweave speakers <media> before voxweave speakers enroll stores a voice under a wrong name.

From source (for development or pulling new code):

make install       # auto-detects the platform: Apple Silicon -> [mps], anything else -> [cuda]
                   # (cu128 torch wheel with an NVIDIA driver, CPU wheel without)
make reinstall     # after pulling new code (same platform detection)
make uninstall

Override the detection per invocation: make install VARIANT=mps or make install TORCH_BACKEND=cpu.

Quickstart

# Transcribe a video to a timestamped VTT (+ a JSON source of truth)
voxweave episode.mkv

# Opt in to speaker diarization (requires gated-model access; see Setup)
voxweave interview.mkv --diarize

# Stay on the older 3.1 pipeline if that is the gate you have accepted
voxweave interview.mkv --diarize --diarize-model 3.1

# ...edit episode.vtt by hand (fix wording, line breaks)...

# Re-align the edited text against the original audio
voxweave align episode.vtt

# Optionally translate the aligned subtitles to Chinese
voxweave translate episode.vtt --target zh

Usage

All processing commands share numbered workflow steps, a segmented dot bar, and elapsed time. For example, while translating:

[2/3] translate cues
translate -> zh       [⣿⣿⣿⣿⣿⣿⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀] 128/400 0:00:12

The denominator reflects enabled steps, not an estimate of elapsed time. Downloads and retries stay inside their current step. Cyan means active, green complete, yellow warning, and red failure; elapsed time is subdued. Unknown totals use a pulse instead of a fabricated percentage. Burn reports actual encoded frames/time. Non-terminal runs emit static step lines and occasional encoding updates to stderr, with no animation; stdout remains reserved for result paths or the speaker-service URL.

A run that finishes cleanly closes with one muted stderr line of wall-clock time per workflow step, so "where did the time go" is answerable from any run:

timing: inspect source 0.4s | prepare audio 1m17s | detect songs 31.6s | find speech 9.1s | transcribe and align 1m41s | detect shot changes 0.1s | layout subtitles 1.2s | write outputs 0.3s | total 3m41s

Anything under a minute is printed as seconds, anything above as NmSSs. A step entered more than once accumulates; total is the sum of the listed steps. Failed runs end in the error panel instead, and a run without a declared plan prints no summary. A transcription run with --debug additionally records the steps finished by mid-run in debug/meta.json under a timings key — that file is written while transcribe and align is still open, so it holds a prefix of the printed line, and only the transcribe path writes it.

Transcribe

voxweave transcribe <media> (or voxweave <media>) — separation → song-skip → VAD chunking → ASR + forced alignment → smart_split → writes <stem>.vtt (editable) + <stem>.json (word-level timestamp source of truth). Models load in-process (see voxweave.backend); the separator is released from VRAM before ASR+alignment load, so peak usage is ≈ max(sep, asr) rather than their sum.

voxweave episode.mkv
voxweave transcribe episode.mkv             # explicit form of the same command
voxweave clip.mp4 --no-separate          # clean speech (podcast/lecture): skip separation
voxweave episode.mkv --asr-model qwen3-asr-1.7B   # larger, more accurate ASR
voxweave episode.mkv --context "Ryland Grace, Astrophage, Hail Mary"   # bias names/terms

Unknown command words produce a command error. Bare subtitle or JSON paths are not transcribed: use align for edited VTT, export for subtitle format conversion, or render for layout from the sibling JSON. These inputs are never automatically routed to an in-place editing command. See the migration notes for old names.

Label speakers

After a diarized transcription, voxweave speakers serve <media> prepares an audition page in memory, serves it on 127.0.0.1 by default, and opens it in your browser. The page embeds up to three clean, non-overlapping speech clips per diarizer id. Listen, enter names, then select Save to write them directly to the episode's speaker mapping. The server runs until Ctrl+C; use --no-open to print the access link without opening a browser, or --port N to choose its HTTP port. No audition HTML is written to disk. voxweave speakers <media> is the supported shorthand for serve; --manual disables voice matching for that session.

To access the audition from another device, use voxweave speakers serve episode.mkv --host 0.0.0.0 --port 8765 --no-open, then open the printed http://0.0.0.0:8765/?k=... link with 0.0.0.0 replaced by the server's IP address. This binds all IPv4 interfaces.

For an ngrok tunnel on the same machine, run voxweave speakers episode.mkv --port 9999 --no-open --ngrok and, in another terminal, ngrok http http://127.0.0.1:9999. Open the tunnel's URL with the printed /?k=... appended. No public domain needs to be supplied: VoxWeave reads the local agent API at 127.0.0.1:4040 and refreshes the accepted URLs as tunnels change. Only tunnels forwarding to this local port are accepted, including HTTPS access and saves.

[!WARNING] Access is by link. The URL speakers serve prints and opens carries a random access key (?k=..., new on every start); opening it sets a session cookie, and every request without that cookie is refused, on 127.0.0.1 too. Anyone who has the link can play the episode audio, read and change speaker names and run splits, so share it only with people you trust and stop the server when you are done. With --host 0.0.0.0 the connection is plain HTTP, so the link and the audio are not encrypted on your network.

voxweave episode.mkv --diarize
voxweave speakers serve episode.mkv
# Save names in the browser, stop the server with Ctrl+C, then render them
voxweave render episode.json

If pyannote merged two people under one diarizer id, select Split this speaker on that card. VoxWeave clusters the id's individual turns and lets you audition both proposed groups before applying the split. This action requires the episode to have been captured with --voiceprints; the turns are embedded with the same embedder (and checkpoint) the voiceprints were captured with, and the proposal is refused if that embedder or the audio provenance cannot be reproduced (for example, when a bound separated-vocals cache is missing or stale). A confirmed split rewrites speaker_turns and the bound voiceprint centroids (recomputed with the capture's centroid recipe) and asks you to restart voxweave speakers serve to audition and name the new id. It also saves the files it rewrote as a one-level snapshot in the episode cache (speaker-split.undo.json). Until you restart, the card offers Undo this split, which restores the transcript JSON, the voiceprints and the speaker mapping from that snapshot. The undo is refused, and the page says which file is the reason, when one of them changed after the split (or the media did), or when there is no snapshot left; a refused undo restores nothing. Either way, restart voxweave speakers serve to audition again. The page offers the undo only in the session that applied the split; after a restart, take a split back by transcribing the episode again with --diarize --voiceprints, which diarizes it from scratch. voxweave speakers purge deletes the snapshot.

Voice matching across episodes is a separate, opt-in layer. Capture centroids with --diarize --voiceprints, review the ordinary empty mapping, then enroll only those human-entered names into your voice library:

voxweave episode.mkv --diarize --voiceprints
voxweave speakers serve episode.mkv
# Save reviewed names in the browser first
voxweave speakers enroll episode.mkv

# Later episodes, in this folder or any other: suggestions appear in the served page and a
# regenerable cache record.
voxweave speakers serve ../season-2/episode-01.mkv

Voice library

speakers enroll saves voices into one library shared by every media folder, so a voice enrolled in one season, show or folder of recordings can be suggested in another. The library directory is chosen by, first match wins:

1. --voices-dir DIR (on speakers serve, speakers enroll and every voices command);
2. VOXWEAVE_VOICES_DIR;
3. dir under [voices] in ~/.config/voxweave.conf (relative to that file's directory);
4. $XDG_DATA_HOME/voxweave/voices, else ~/.local/share/voxweave/voices (every platform).

voxweave voices where prints the directory and which of these chose it. The library is plain JSON, one file per concern, and never holds audio:

identities.json              names, aliases and scopes of every identity; space files; forgotten ids
spaces/<model>-<fp12>.json   the voice vectors of one embedding space
history.jsonl                log of changes (ids, scopes, episode labels, counts; never vectors or display names)
.library.lock                one lock for the whole library

Each identity has a random id; names are attributes, so two identities may both be called "Alex". Names are shared by every embedding space (a rename applies everywhere), while vectors are kept per space: an episode captured with redimnet2 is only compared with redimnet2 vectors, and switching embedders starts a new space in the same library. A voice sample keeps a pointer to its source (media path, media fingerprint, capture id, speaker label, episode) so a future re-embedding can find the media again.

Scopes and tiers. Every enrollment is tagged with a scope: --show NAME, else the name of the media's folder. When you review an episode, identities of its own scope are suggested as before (tier 1). Identities of every other scope (the same voice actor in another work, the same person in another folder of recordings) are suggested only above a stricter similarity bar (tier 2: VOXWEAVE_VOICES_GLOBAL_SUGGEST), are never prefilled, and are labelled with the scopes they come from. The bar is stricter because false suggestions grow with the library: if one unrelated voice clears it with probability p, a library of N unrelated voices produces at least one false suggestion with probability 1 - (1 - p)^N. A generic folder name, one that many shows share (a season, disc, part or year number such as Season 1, S02, Disc 1 or 2023, or Specials), is qualified with its parent folder: episodes in Frieren/Season 1 get the scope Frieren / Season 1. Other folder names are used as they are, so two unrelated folders with the same name (say Recordings) share a scope; pass --show to keep them apart.

A name you enter links to an existing identity only when you used that identity's suggestion button, or when an identity of the same scope carries that name. Otherwise enrollment creates a new identity, even if another scope has someone of the same name; if a name is ambiguous within a scope, enrollment stops and asks you to rename one (voxweave voices rename ID NAME).

Sharing on a NAS. Several machines can point [voices].dir at the same directory on a NAS, as the same user account. Writes take an exclusive flock on .library.lock; Linux NFS clients emulate it with byte-range locks, which works on NFSv4 and on NFSv3 with the lock manager, but a nolock mount keeps each lock local to its machine, so use a mount with working locks. Every write also checks, just before replacing each file, that it still holds what the write read, and otherwise stops ("re-run the command"). That check narrows the window but is not atomic: without working locks, two simultaneous writers can both pass it and one update can be lost. Voice samples such a race leaves behind for an identity that identities.json does not list are ignored by every reader and kept on disk, as are those left when an older identities.json is restored; only the samples of identities removed with voices forget are deleted by the next write. identities.json lists every space file, so voices forget finds all of them even when an NFS client's cached directory listing is a minute old, and stops (asking you to re-run) if a listed file is not visible yet. Files are replaced by an atomic rename within the directory. File-sync services are not a lock and can produce conflicting copies; use a real network mount. A library belongs to one user account: every file in it (the lock too) is created 0600, and a directory VoxWeave creates is 0700; a directory you created beforehand keeps its own permissions. Sharing one library between different user accounts is not supported: machines that share it must reach it as the same user (the same uid on NFS). Only the built-in location gets missing parent directories created: for --voices-dir, VOXWEAVE_VOICES_DIR or [voices].dir the parent must already exist, so an unmounted share (an empty mount point) makes speakers enroll fail instead of starting a second library on the local disk, and speakers serve warns that the library is missing.

Privacy. The library holds voice biometrics of the people you name; the first write into a new library prints a notice saying so. voxweave voices list and voxweave voices show ID|NAME inspect it, and deleting the directory removes everyone. voxweave voices forget ID removes one person from every space and keeps only the id as a tombstone, so that person is never suggested, imported or enrolled again, not even from a review page made before the forget or from a per-folder store of an earlier version. It does not edit those per-folder stores: it lists the ones that still hold the person (the stores you imported, and the ones next to their media), and their vectors stay on disk until you delete those files. The history keeps the forgotten id's entries without their scopes and episode labels. It never holds voice data or display names, but other entries keep scopes (folder or show names), episode labels (media file names) and, for imports, the imported file's path, all of which can name people too.

Per-show stores from earlier versions. A voxweave.voices.json next to the media is still read for suggestions, never written, with a one-time hint to merge it with voxweave voices import PATH. By default the imported voices keep the scopes under which the store already served suggestions: its folder's scope and the store's show (a store with another file name, only ever used with --voices, keeps its show). --scope NAME imports them under that scope instead, and importing again with another --scope adds that scope. Import keeps the store's ids, so importing it again adds no voice samples, skips samples older than the five an identity already keeps, and leaves the file unchanged. --voices FILE still selects a per-show store explicitly, with its previous behavior (a missing one is created only with both --voices and --show); it cannot be combined with --voices-dir.

A space belongs to one embedding model (see the voiceprint embedders under Setup); a per-show store captured with another embedder is skipped with a warning that names both spaces, and enrollment into such a store is refused. A store built before the dedicated embedders keeps working with --voiceprint-model pyannote (see MIGRATING.md).

Use speakers enroll --replace to replace this episode's prior contribution (the same episode label within the same scope, or the same media). Enrolling an already enrolled capture again after its folder was renamed, or with another --show, moves its voice samples to the new scope without --replace; a new capture of the same media in the new scope needs --replace, and so does enrolling again after a confirmed speaker split, which changes the voices of a capture that keeps its id. The shipped matching policy is suggest-only: VOXWEAVE_VOICES_ACCEPT defaults to off, so stored names appear as review buttons and never become authoritative mapping values. Even when an operator configures a finite accept threshold, a machine prefill exists only in the served input; the v1 mapping is still created with empty names and must be reviewed and saved by the operator.

The versioned mapping is intentionally small:

{"version": 1, "speakers": {"SPEAKER_00": "Aoi"}}

render renders mapped names as WebVTT voice tags while keeping transcript text and the sibling JSON clean. Empty or missing names remain unlabeled. An existing mapping is the normal edit flow: the server re-reads it for the page and replaces it only when you select Save. When burning a mapped SRT, recovered speaker identity is retained in the temporary ASS Name field while the visible dialogue remains prefix-free.

voxweave speakers purge episode.mkv removes that episode's voiceprint artifact and suggestion record from both supported legacy and cache locations. It also removes the cache-only one-level speaker-split undo snapshot and any legacy audition HTML while holding the episode transaction lock. It works after the media has been removed and deliberately keeps the human-edited mapping.

voxweave speakers list episode.mkv inspects that episode's speaker turns, reviewed mapping, and voiceprint state without changing them. Add --json for machine-readable output. It does not read the voice library; use voxweave voices list for that.

Re-align after editing

voxweave align <vtt> — takes the edited VTT text and re-runs forced alignment against the original audio, overwriting the timestamped VTT and updating the JSON. Does not re-run ASR or touch smart_split. Aligns on separated 16k vocals by default (prevents BGM interference); prefers the episode's cached vocals.32k.flac under the artifact cache, otherwise re-separates and caches it there. Existing media-adjacent cache/<stem>.vocals.32k.flac entries remain a legacy read/write-back lane. When a voiceprint pair requires source binding, both managed and legacy vocals caches are accepted only when their matching integrity companion validates.

Both VTT forms are accepted: the timestamped VTT and the plain-text editing draft that --no-timestamps writes. The aligner strips punctuation as a hard constraint; ASR punctuation is re-injected by time so the final output has correct spacing and breaks without stray marks.

On media longer than the single-pass alignment budget (about 30 min, ctc_max_dp_frames), English and Japanese alignment runs in pieces split at silences between cues, so there the VTT needs its cue timing lines. Overlapping or nested cues are fine; if a split would still cut a cue off from its own audio (for example a cue whose start was mistyped far too early), align refuses and names the overlapping cues so you can fix their times. Raising ctc_max_dp_frames aligns the whole file in one pass instead, at the cost of more memory.

voxweave align episode.vtt                 # finds episode.<ext> in the same dir
voxweave align episode.vtt --media original.mkv
voxweave align episode.vtt --no-separate   # align on the original audio (clean sources)

Re-layout offline

voxweave render <episode> — re-run deterministic layout from <stem>.json without any models (adjust line width / sentence breaks instantly).

voxweave render episode.json --max-line-length 14 --max-lines 1
voxweave render episode.json --no-timestamps   # plain-text editing draft

Pass the JSON, its VTT sibling, or the media path; each resolves to the same sibling JSON. This command rewrites the working VTT and JSON and re-lays out every cue from the word timings. It does not back up or protect manual VTT edits: save or align them first.

ASR correction

voxweave correct <vtt> — optional pre-align LLM pass that fixes obvious ASR typos, split words, and garbled proper nouns, producing a reviewable diff. Conservative substitution only (no completion/rewrite), gated by a code check that the matched text equals the original line-for-line. By default writes only an adjacent sidecar <stem>.asrfix.vtt plus an audit JSON in the episode artifact cache — the original VTT is untouched. To accept a reviewed sidecar, replace the VTT with it and run align. --apply instead lets the LLM rewrite the VTT in place and then re-runs alignment automatically (skip that with --no-align, and pass --media when the media is not a same-stem sibling).

voxweave correct episode.vtt --glossary names.json           # review the sidecar
voxweave correct episode.vtt --glossary names.json --apply   # rewrite in place + re-align

Translate

voxweave translate <subtitle> — after align, translate each cue with whole-episode context, preserving cue count, into <stem>.<target>.<ext> (the original is left unchanged). Accepts .vtt/.srt/.ass/.ssa; the output mirrors the input format (episode.srt → episode.zh.srt).

voxweave translate episode.vtt --target zh
voxweave translate episode.vtt --target en --context "sci-fi, formal register" --glossary terms.json
voxweave translate downloaded.srt -t zh               # foreign SRT in, SRT out
voxweave translate episode.vtt --target zh --reasoning-effort low
voxweave translate episode.vtt --target zh --concurrency 1   # one whole-episode request

Translation accepts any OpenAI-compatible Chat Completions endpoint, including a self-hosted Qwen server. Configure it once in ~/.config/voxweave.conf:

[llm]
base_url = "http://127.0.0.1:8000/v1"
model = "auto"              # or the exact served model ID from /v1/models
api_key_env = ""            # keyless server; otherwise name an environment variable
reasoning_effort = "low"    # translate only; omit to keep the server default
concurrency = 8             # translate windows in flight; 1 = single whole-episode request
window_cues = 100           # cues per window when concurrency > 1

By default the episode is translated as bounded windows (window_cues cues each) with several requests in flight (concurrency); each window sees the preceding source cues as context. This trades some cross-window stylistic continuity for throughput and resilience on a self-hosted server -- --glossary and --context are the consistency tools there. --concurrency 1 sends one whole-episode request (sequential windows with translated-tail continuity only past 800 cues), the best choice for a hosted API.

Every response is checked for completeness: an answer that does not finish with stop (length cap, a server aborting its structured-output grammar) is retried, and the last attempt for that window runs without response_format (plain JSON). Cues still untranslated after the retry stage fail the command and keep the progress file, so rerunning resumes; --allow-partial writes the file anyway with those cues in source text.

CLI options override environment variables, which override this configuration. --model accepts a served model name without a built-in model list. auto requires the endpoint to advertise exactly one model. For authenticated endpoints, set api_key_env = "MY_LLM_KEY" and put the secret in that environment variable.

--reasoning-effort sends the standard top-level reasoning_effort field in the request. Accepted values depend on the served model: for example, Qwen3.8-27B supports low, medium, and xhigh. Use --reasoning-effort default to override a configured value and omit the field. VoxWeave does not inject thinking template switches or silently replace an effort rejected by the endpoint. Reasoning output is kept out of subtitle text. Interrupted translations resume only when the input, endpoint, resolved model, effort, context, glossary, and target match.

Export

voxweave export <subtitle> — convert between subtitle formats: VTT/SRT/ASS/SSA in, SRT/ASS/VTT out (written next to the input; the VTT + JSON pair stays the source of truth for voxweave-produced subtitles). ASS output carries a Default style; lyric cues (♪ ... ♪) render italic. Named VTT cues become NAME: text in SRT and use the ASS Dialogue Name field. The SRT position tag {\an8} (raise to the top) stays in SRT output, becomes a real override in ASS and is dropped from VTT. Foreign SRT/ASS files can be exported to VTT to enter the editing workflow. Every cue needs timestamps: a plain-text edit draft, or a file in which only some cues are timed, is refused with the untimed cues named (run align on a VTT first; an SRT needs its missing timing lines).

voxweave export episode.vtt --format srt
voxweave export episode.vtt -f srt -f ass
voxweave export downloaded.ass --format vtt        # foreign ASS -> VTT for editing/translate

Repeat -f, --format to request multiple output formats in one run.

Pack (soft subtitles)

voxweave pack <subtitle>... — remux the source media with the subtitle file(s) (VTT/SRT/ASS) added as proper subtitle tracks. Pure stream copy (instant, lossless, reversible); each track is titled VoxWeave <Language> with the container language tag taken from the filename (episode.zh.vtt → chi / "VoxWeave Chinese"), and the first packed track is flagged default (the source tracks lose that flag) so players select it. ASS inputs keep their styling in mkv targets (mp4/webm store text-only codecs, so styling is dropped there).

voxweave pack episode.zh.vtt                    # finds episode.<ext>, keeps its container -> episode.zh.pack.mkv
voxweave pack episode.zh.vtt episode.ja.vtt     # several tracks at once -> episode.zh.ja.pack.mkv
voxweave pack episode.zh.vtt --container mp4    # mov_text in mp4 (image subs are dropped)
voxweave pack episode.zh.vtt --media other.mkv -o out.mkv

The output goes next to the source as <media stem>.<languages>.pack.<container>, the languages taken from the subtitle file names (just .pack when none has one), so packing another language never replaces an earlier result; -o picks any other path, and its extension (.mkv/.mp4/.webm) picks the container. mkv targets keep every source stream (including attachments; mov_text subtitles, which Matroska cannot store, are converted to SRT); mp4/webm targets keep video+audio and existing text subtitle tracks only (webm also drops cover art). HEVC video muxed into mp4 is tagged hvc1 for Apple players.

Burn (hard subtitles)

voxweave burn <subtitle> — render the subtitles (VTT/SRT/ASS) into the pixels and write a clean file with all subtitle tracks removed. For VTT/SRT input a styled ASS is generated at the actual frame size (same look as export, lyric cues italic); ASS/SSA input goes to libass as-is, keeping its own styling. The video is re-encoded at constant quality with hardware acceleration when available: NVENC on NVIDIA, VideoToolbox on macOS, libx264/libx265/libsvt-av1 software fallback. Audio is stream-copied (mp4 targets re-encode mp4-incompatible codecs to AAC). The output goes next to the source as <media stem>.<language>.burn.<container> (episode.burn.mp4 when the subtitle file name carries no language) unless -o names one.

voxweave burn episode.zh.vtt                          # hevc, auto hw encoder, -> episode.zh.burn.mp4
voxweave burn episode.zh.vtt --codec h264             # legacy-device compatibility
voxweave burn episode.zh.vtt --codec av1 --container mkv   # max compression, recent hardware
voxweave burn episode.zh.vtt --quality 20 --font "Noto Sans CJK SC"

Progress is rendered with rich: countable stages (demix windows / PANNs batches / per-chunk ASR+alignment / align per-cue / translate streaming per-line) show a real x/N bar with elapsed time; indeterminate stages (decode / file write) show a pulse bar. -v/--verbose enables DEBUG logging.

The edit-and-resync workflow

voxweave episode.mkv          # 1. transcribe  -> episode.vtt + episode.json
  └─ (optional) correct       # 2. LLM ASR fix -> episode.asrfix.vtt (review it; or --apply to rewrite + re-align)
edit episode.vtt by hand      # 3. fix wording / line breaks
voxweave align episode.vtt    # 4. re-derive timestamps from audio (overwrites VTT + JSON)
voxweave translate episode.vtt --target zh   # 5. context-aware translation
voxweave pack episode.zh.vtt             # 6. soft-mux into the media (or burn for hardsubs)

Timestamps are always derived from the audio by the forced aligner — you never hand-edit them. Edit the text freely; align puts the timing back.

How it works

Configuration

Precedence: CLI flag > env var > ~/.config/voxweave.conf > built-in default. A commented default config is written on first run (migrated automatically from a pre-rename qsub.conf). Set VOXWEAVE_CONFIG to read (and create) the config file somewhere else.

# ~/.config/voxweave.conf  —  TOML
# Precedence: CLI flag > env var > this file > built-in default.

# Default ASR model (= --asr-model). Short name (qwen3-asr-0.6b | qwen3-asr-1.7b) or full HF id.
# Special value "hybrid" (= --hybrid) -> dual-ASR fusion (whisper text + Qwen punctuation).
asr_model = "Qwen/Qwen3-ASR-1.7B"        # built-in default: Qwen/Qwen3-ASR-0.6B

# Hugging Face token for the gated diarization models (top-level key; prefer the
# VOXWEAVE_HF_TOKEN / HF_TOKEN environment variables over storing it here).
# hf_token = "hf_..."

# Single-pass CTC alignment budget in emission frames (default 90000 ≈ 30 min at 50 fps);
# longer audio is split at silence anchors. Raise it on large cards for long movies.
# ctc_max_dp_frames = 150000

# Model load strategy:
#   "peak" (default) — serial peak-shaving: all-chunk ASR -> release -> all-chunk align;
#                      ASR and aligner never co-reside, peak VRAM = max(models). Works on 8 GB.
#   "sum"            — same passes as "peak", but the ASR model(s) stay resident while the aligner
#                      loads; peak VRAM = sum(models), but skips two model swap round-trips
#                      (faster on large-VRAM cards).
load_strategy = "sum"

# Inference batch sizes: windows per GPU forward (env: VOXWEAVE_SEP_BATCH / VOXWEAVE_CTC_BATCH /
# VOXWEAVE_MMS_BATCH / VOXWEAVE_ASR_BATCH). On an 8 GB-class card batch=1 already saturates
# compute — measured no speedup at 2/4, just ~+0.8 GiB VRAM per extra separation window — so the
# defaults stay at 1. Only worth raising on much wider GPUs, and only after measuring.
[batch]
separate = 1                             # vocal separation (MelBandRoformer) 8s windows
ctc      = 1                             # wav2vec2 CTC emission 30s windows (en aligner)
mms      = 4                             # MMS-300m emission batch (ja aligner)
asr      = 1                             # Qwen3-ASR chunks per decode call; 1 = the per-chunk call.
                                         # Opt-in: faster but its transcripts differ from batch 1 —
                                         # see "Performance knobs" below

# Vocal separation numerics (= env VOXWEAVE_SEP_AUTOCAST). autocast wraps only the model
# forward; the overlap-add accumulation stays fp32. CUDA only (ignored on CPU / MPS).
#   "off" (default) — fp32 forward, the reference output, byte-identical run to run.
#   "bf16" | "fp16" — mixed-precision forward: faster and slightly leaner, at the cost of tiny
#                     waveform differences in the stem. See "Performance knobs" below.
[separate]
autocast = "off"

# Diarization pipeline (= --diarize-model / env VOXWEAVE_DIARIZE_MODEL).
# Values: "community-1" (built-in default), "3.1", or any full Hugging Face pipeline id.
# Only legacy (voiceprint model "pyannote") stores depend on it: their centroids come from the
# pipeline's own embeddings, so switching pipelines starts a fresh legacy store.
# clustering: who is who once the pipeline has found the turns (= --speaker-clustering /
# env VOXWEAVE_DIARIZE_CLUSTERING). "pyannote" (built-in default) = the pipeline's own
# clustering; "voiceprint" = regroup with ReDimNet2 (CC BY-NC-SA 4.0 weights) and drop
# turns nobody can be attributed to.
[diarize]
model = "community-1"
clustering = "pyannote"

# Voiceprint embedder (= --voiceprint-model / env VOXWEAVE_VOICEPRINT_MODEL); used with --voiceprints.
#   "auto" (default) — anime-va for Japanese, redimnet2 for every other language
#   "redimnet2"      — ReDimNet2-B6; weights non-commercial (CC BY-NC-SA 4.0)
#   "anime-va"       — anime voice-actor ECAPA-TDNN; Japanese only, unclear data provenance
#   "pyannote"       — legacy: the diarization pipeline's embeddings (matches older stores)
# Voice stores are per embedding space; changing the diarizer does not affect redimnet2/anime-va.
[voiceprint]
model = "auto"

# Voice library (= --voices-dir / env VOXWEAVE_VOICES_DIR): where `speakers enroll` saves named
# voices, shared by every media folder. Default: $XDG_DATA_HOME/voxweave/voices, else
# ~/.local/share/voxweave/voices. May be a NAS path shared by several machines of the same user
# account (needs working NFS locking, see "Voice library"); several accounts cannot share one library.
# A relative path is relative to this file's directory.
[voices]
# dir = "/mnt/nas/voxweave/voices"

# Default on/off for the boolean pipeline flags. An explicit CLI flag always wins
# (e.g. separate = false here, --separate on the command line for one run).
[defaults]
separate   = true                        # vocal separation before ASR/alignment (--separate/--no-separate)
skip_songs = true                        # PANNs music detection + skip before ASR (--skip-songs/--no-skip-songs)
normalize  = false                       # loudnorm on the 16k input (--normalize/--no-normalize)
diarize    = false                       # pyannote speaker diarization (--diarize/--no-diarize; gated-model token required)
voiceprints = false                      # opt-in biometric centroid capture; requires diarize
timestamps = true                        # cue timing lines in the VTT (--timestamps/--no-timestamps)
shot_snap  = true                        # snap cue boundaries onto shot changes (--shot-snap/--no-shot-snap)
vad_mask   = false                       # suppress wav2vec2 CTC emissions outside speech; MMS/Qwen ignore it (--vad-mask/--no-vad-mask)

# LLM for translate / correct: any OpenAI-compatible chat-completions endpoint.
# Precedence per key: CLI option > env (VOXWEAVE_TRANSLATE_MODEL / VOXWEAVE_FIX_MODEL,
# OPENAI_BASE_URL) > this section > built-in (gpt-6-luna on api.openai.com, key from OPENAI_API_KEY).
[llm]
model = "auto"                           # or a model name; "auto" = the endpoint's only served model
base_url = "http://127.0.0.1:8000/v1"    # e.g. a local vLLM; remove for api.openai.com
api_key_env = ""                         # "" = keyless endpoint; else the env var holding the key
# reasoning_effort = "low"               # translate only; accepted values depend on the served model
# concurrency = 8                        # translate windows in flight; 1 = single whole-episode request
# window_cues = 100                      # cues per window when concurrency > 1

# dual-ASR fusion sub-models — only consulted when running with --hybrid.
[fusion]
whisper = "large-v3-turbo"               # Whisper size: large-v3 (best) | large-v3-turbo (~5x faster); faster-whisper on cuda, mlx-whisper on mps
qwen    = "Qwen/Qwen3-ASR-1.7B"          # punctuation model; must emit punctuation -> 1.7B, not 0.6B

# Per-language forced-alignment model. Key = ISO-639-1 code; unlisted languages use Qwen3-ForcedAligner.
# Values:
#   "mms"   — MMS-300m + uroman, full-file single pass (immune to per-cue drift; the gold standard).
#   HF id   — wav2vec2 CTC via HF transformers, also full-file; weights land in ~/.cache/voxweave/align.
#   bundle  — torchaudio bundle name, e.g. "WAV2VEC2_ASR_LARGE_LV60K_960H" (same model, cached in ~/.cache/torch).
#   ""      — explicitly fall back to Qwen for that language.
[align]
en = "facebook/wav2vec2-large-960h-lv60-self"  # English: LV60K-self CTC, full-file (HF hub)
ja = "mms"                                      # Japanese: MMS-300m + uroman full-file (= whisperx fork align_ctc)
# zh  = "mms"                                   # Chinese can also use MMS; default is Qwen (native CJK char-level)
# yue = ""                                      # force Qwen for Cantonese

Performance knobs

Two GPU throughput settings are opt-in and off by default, because both change the output. They are worth enabling only on a wide GPU, and only if you have re-checked the result on your own material — there is no truth ruler in the pipeline that can tell you whether the changed output is better or worse.

Neither has a CLI flag; precedence for both is env var > config file > built-in default. [batch].asr applies to the torch Qwen engine only — Whisper and the Apple Silicon MLX adapter take one chunk per ASR call whatever it says — and autocast applies to CUDA only, being ignored on CPU/MPS. Raising [batch].separate above 1 bought no speedup on the same GPU (the separator is already compute-bound), so it stays at 1 as well.

Batched ASR is guarded against qwen-asr #207, where a mixed-length batch can corrupt its shorter item into a lone !: chunks are grouped by duration to keep each batch's lengths close, and any batched result that comes back implausibly empty for its chunk's duration is re-run alone through the per-chunk call (thresholds: VOXWEAVE_ASR_BATCH_MIN_CPS, VOXWEAVE_ASR_BATCH_MIN_CHECK_SEC). A batch whose call raises is likewise redone chunk by chunk, so one poisoned chunk degrades alone.

Shot-change detection normally needs no setting (VOXWEAVE_SHOT_SCENE tunes its sensitivity): it is a CPU-only ffmpeg pass that now starts before transcription and is joined at its workflow step, so it overlaps the GPU stages and the detect shot changes entry in the timing line is normally ~0s.

Data contract

Each input keeps its editable delivery set beside the media:

• <stem>.json — the source of truth: word/character-level segments, language, VAD speech, plus optional replay data (shot_changes, sing_spans, speaker_turns) so render can redo shot snapping, lyric flagging, and speaker formatting without re-running any model.
• <stem>.vtt — editable subtitles. By default cues carry word-level timestamps (same precision as align output, ready to use); --no-timestamps writes a plain-text editing draft for hand-correction, which align re-times.
• Subtitle-family deliverables — translated .vtt/.srt/.ass files and derived files such as <stem>.sdh.vtt and <stem>.asrfix.vtt also stay beside the media.

Other episode state is stored beside the media, under <media directory>/cache/<stem>/. For episode.mkv, the layout is:

cache/episode/
├── source.json                    # source file-name claim (never an absolute path)
├── episode.episode.lock           # episode transaction lock
├── .episode-domain.lock           # same-directory stem publication lock
├── speakers.json                  # reviewed diarizer-id-to-name mapping
├── speakers.suggest.json          # regenerable match suggestions
├── voiceprints.json               # optional biometric centroids
├── speaker-split.undo.json        # files a confirmed split rewrote (one level)
├── vocals.32k.flac                # separated-vocals cache
├── vocals.32k.flac.meta.json      # source-bound integrity companion, when required
├── vocals.32k.flac.lock           # vocals-cache lock
├── episode.zh.progress.json       # interrupted translation resume state
├── episode.align-evidence.json    # durable alignment evidence
├── episode.asrfix.json            # correction audit
└── debug/                         # optional --debug bundle

Because the claim records only the source file name, moving or renaming the media directory (local disk to NAS, season folder reorganizations) keeps every artifact valid — the cache travels with the media. The marker normally claims the media file; for a supported standalone subtitle command with no discoverable sibling media, it claims that input file instead. If another same-stem file in the same directory already owns the plain stem (episode.mkv next to episode.mp3), VoxWeave uses <stem>--<sha1-of-file-name-first-8>/ for the second claim.

Every cache name above keeps its plain form while it fits the filesystem's file-name limit (NAME_MAX, 255 bytes on most filesystems, counted in UTF-8 bytes). Only a name that would not fit is shortened: <stem, cut to fit>--<first 8 hex digits of sha1(stem)><suffix>, for example <stem>.episode.lock once the media stem reaches about 243 bytes, or the collision directory at about 246. The shortened name is deterministic, and an entry is found under whichever form it was written. Deliverables beside the media (<stem>.json, <stem>.vtt, derived and translated subtitles, pack/burn outputs) are never shortened: when one of them would not fit, the command stops before doing any work and names the file, so rename the media (or pass a shorter -o) and run it again.

Existing adjacent machine sidecars remain compatible: when an adjacent <stem>.speakers.json, <stem>.speakers.suggest.json, <stem>.voiceprints.json, translation progress file, alignment-evidence file, correction audit, or old cache/ vocals entry exists, VoxWeave reads it first and writes updates back to that same legacy lane. New artifacts use the cache. An existing adjacent <stem>.episode.lock is joined to the cache lock so old and new VoxWeave processes still serialize the same episode. The adjacent <stem>.json transcript is not a legacy exception: it remains the current source of truth by design. The explicit/discovered show-level voxweave.voices.json is a user-managed database and also stays where the user placed it.

Sensitive and derived speaker data

Treat voiceprint artifacts, speaker-split undo snapshots, show-level voices stores, suggestion records, in-memory auditions, calibration reports, crash-temporary files, and media snapshots as sensitive or derived data. The served audition contains embedded audio but is never saved by VoxWeave; a snapshot contains private media bytes rather than biometrics. Snapshots live under ${VOXWEAVE_CACHE_ROOT:-~/.cache/voxweave}/snapshots, are mode 0600 in an owner-only directory, and inactive crash residue older than one hour is cleaned on a later snapshot creation. Active snapshots are lock-protected from that janitor.

The purge command removes only the episode voiceprints, suggestion record, split undo snapshot, and any legacy audition HTML; it preserves the reviewed mapping, transcript, subtitles, and unrelated cache state. Remove a show voices store, calibration report, inactive snapshot residue, or crash-temporary file manually when it is no longer needed. Do not remove an active snapshot. Backups, filesystem snapshots, synced folders, and manual copies are outside VoxWeave's control and are not erased by purge; delete them separately according to their retention policy.

Testing

• Unit tests (models mocked, no network): make test (= uv run --extra $(VARIANT) pytest tests/ -v). The suite ignores inherited VOXWEAVE_* settings; with a real-model gate such as VOXWEAVE_REAL_WEIGHT_TESTS=1 set it keeps VOXWEAVE_CACHE_ROOT, VOXWEAVE_MODEL_DIR and VOXWEAVE_HF_TOKEN, so those tests find weights you already have. Its P6 oracle tests also need the interpreter recorded in calibration/p6-oracle/manifest.json (make dev syncs it, as CI does), Linux x86_64, a full clone (a shallow one lacks the oracle's reference commits) and the ambient environment the oracle records, which CI sets: env -u TZ -u PYTHONHASHSEED LC_ALL=C.UTF-8 make test.
• Lint / format: make lint (make lint-check is the read-only form CI runs; both use the pinned ruff version)

Support

If VoxWeave saves you time, you can support development here:

License

MIT — see LICENSE.

Acknowledgments

• WhisperX — the forced-alignment + edit-and-resync workflow this project builds on; the Japanese MMS full-file alignment path is a faithful port of its ctc align backend.
• stable-ts — inspiration for timestamp post-processing and documentation structure.
• Qwen3-ASR / Qwen3-ForcedAligner (Alibaba) — local ASR + aligner.
• MMS-300m (Meta) via ctc-forced-aligner — zero-OOV CJK alignment.
• Mel-Band Roformer (lucidrains) + KimberleyJSN weights — vocal separation.
• BudouX, jieba, PySBD — CJK/sentence line-break.
• PANNs — song/music detection.
• Silero VAD — voice activity detection.
• ReDimNet2 (Palabra.ai) — the default voiceprint embedder; SpeechBrain ECAPA-TDNN with the anime voice-actor weights (litagin) — the Japanese one.