vvasr

This script leverages VibeVoice-ASR to perform ASR on audio and video files. The VibeVoice model has been quantized to run on GPU with 16 GB of VRAM.

To install, use one of the following commands:

uv tool install 'vvasr[amd] @ git+https://github.com/jpetazzo/vvasr'     # ROCm
uv tool install 'vvasr[nvidia] @ git+https://github.com/jpetazzo/vvasr'  # CUDA
pip install 'vvasr[nvidia] @ git+https://github.com/jpetazzo/vvasr'
pip install --extra-index-url https://download.pytorch.org/whl/rocm7.0 'vvasr[amd] @ git+https://github.com/jpetazzo/vvasr'

Notes:

• pip install doesn't need an extra index URL for nvidia cards, because the default torch package on PyPI is the CUDA one
• make sure to specify [amd] even if you specify the extra index URL for rocm, because the pinned version numbers are different between CUDA and ROCm
• ROCm is currently pinned to 7.0 because the model outputs garbled text on 7.1 and 7.2

You also need ffmpeg to convert audio to the right format.

Usage:

vvasr hello.mp3

This will convert the file in the format that VibeVoice-ASR can use, then run ASR on it.

Output:

• hello.txt (plain text transcription with timestamps, speaker, text)
• hello.vtt (transcription in subtitle format)
• hello.json (same thing but in JSON format for easy processing)

The program can also work on video files (it will then merely extract the audio track).

Options:

vvasr --prompt "Valentina, Capele, VibeVoice" recording.mp4
vvasr --prompt @hotwords.txt recording.mp4

--prompt passes hotwords or context (names, jargon, subject matter) to the model, which measurably improves recognition of proper nouns. @FILE reads the text from a file; @@ at the start of the argument escapes a literal @.

--max-new-tokens bounds a runaway generation; it defaults to 1200 per minute of audio, which is roughly double what dense speech actually needs.

Technical details

To run with less VRAM than the original, VibeVoice-ASR has been quantized. Only the Qwen2.5-7B decoder's nn.Linear layers have been quantized; everything else remains as such.

vvasr reads the quantized model from disk; and if it doesn't exist, it suggests that the user runs hf download microsoft/VibeVoice-ASR-HF to download the original model, and then vvasr --quantize microsoft/VibeVoice-ASR-HF which will quantize the downloaded model, and write out the quantized version to the path where vvasr expects it.

vvasr uses a recent version of transformers (5.3+) which ships the ASR model as a first-class architecture in the library (vibevoice_asr).

AI disclosure

The code was generated with Claude Code. I don't have an accurate measurement of the token usage, because I ran multiple fairly intensive sessions to benchmark different quantizations of the model, initially working with an older version of transformers and the initial release of the VibeVoice model; then after determining what combination worked, it was reimplemented using the VibeVoice architecture that ships in transformers 5.3 (in a few hundreds lines of code instead of thousands).

Misc references

• transformers v5.3.0 release
• VibeVoice ASR model docs
• microsoft/VibeVoice-ASR-HF