audio-model

Audio model experiments (classification, ASR, audio generation) on an RTX 3090, managed with uv.

Setup

The environment is already created. To recreate it from scratch:

uv sync

This reads pyproject.toml + uv.lock and builds .venv with Python 3.12 (pinned by .python-version). uv downloads its own CPython, so nothing depends on the system miniconda.

Verify everything works (GPU, audio decoding, HF stack):

uv run scripts/check_env.py            # offline checks
uv run scripts/check_env.py --network  # also pull a real dataset from the Hub
uv run scripts/check_env.py -v         # show tracebacks

Troubleshooting: ModuleNotFoundError: No module named 'datasets'

Nothing is broken — you are using the wrong interpreter. python on this box resolves to /root/miniconda3/bin/python (miniconda comes first on PATH), but every dependency lives in /root/audio-model/.venv. Use either:

uv run python -c "import datasets"      # preferred, no activation needed
# or
source .venv/bin/activate               # then plain `python` works

The same applies to python train.py → use uv run python train.py. In VS Code, select /root/audio-model/.venv/bin/python as the interpreter.

Usage

Never activate the venv manually — prefix commands with uv run:

uv run python train.py           # run a script
uv run scripts/check_env.py      # run the env check
uv add librosa-plus              # add a dependency (updates pyproject + lock)
uv add --group dev pytest-xdist  # add a dev-only dependency
uv remove <pkg>                  # drop a dependency
uv lock --upgrade                # refresh the lockfile
uv tree --depth 1                # inspect the dependency tree

For notebooks, select .venv/bin/python as the kernel (ipykernel is in the dev group).

What's installed

Notes on the audio stack:

• torchcodec is the primary decoder. It is also what datasets uses to decode the Audio feature. It requires FFmpeg shared libraries — system ffmpeg 4.4.2 is installed for this.
• torchaudio is upstream in maintenance mode (features were removed in 2.9 and decoding now delegates to torchcodec). It is kept because torchaudio.transforms / .functional / .pipelines are still the standard spectrogram and pretrained-pipeline APIs. Its latest release is 2.11, but it ships as +cu130 and works against torch 2.13 — verified by check_env.py.
• soundfile bundles libsndfile, so WAV/FLAC/OGG/MP3 I/O works even without FFmpeg.

Environment specifics for this box

Set in ~/.bashrc:

• HF_HOME=/root/autodl-tmp/hf-cache — model cache lives on the 50 GB data disk, since the system disk only has ~18 GB free and audio models are large (Whisper large-v3 ≈ 3 GB, Qwen-Audio ≈ 20 GB).

Network:

• huggingface.co is reachable only via the proxy that ~/.bashrc exports. Run training/download commands from a login shell so those vars are set.

• That proxy includes all_proxy=socks5://..., and huggingface_hub 1.x downloads over httpx — which refuses SOCKS without socksio. That is why httpx[socks] is an explicit dependency.

• HF_ENDPOINT=https://hf-mirror.com transfers raw files faster (~2.5×), but load_dataset fails against it — huggingface_hub validates HF-specific response headers and gives:

  FileMetadataError: Distant resource does not seem to be on huggingface.co
  ...wrapped as... LocalEntryNotFoundError / FileNotFoundError
  

  Use the mirror only for direct hf_hub_download / weight pulls, and leave HF_ENDPOINT unset (official endpoint via the proxy) for load_dataset.

• PyPI and CPython downloads are mirrored in ~/.config/uv/uv.toml (global, not in this repo, so the repo stays portable for other contributors).

Not installed (add if needed)

nemo_toolkit, pyannote.audio (diarization, gated), silero-vad, openai-whisper, faster-whisper/ctranslate2, demucs, trl, optimum, bitsandbytes (4/8-bit), flash-attn. torchmetrics[audio] adds PESQ/STOI but needs compilation.

(gradio, speechbrain and llama-cpp-python are installed — they are used by the chapter exercise scripts below.)

Hugging Audio course — chapters 4–7 practical work

Reproduces the Hugging Audio hands-on exercises. All checkpoints stay local (no push_to_hub).

Run a training script:

uv run python audio-classify/train.py --model-id facebook/wav2vec2-base \
    --epochs 20 --split-secs 15 --output-dir /root/autodl-tmp/audio_outputs/wav2vec2-gtzan-15s
uv run python audio-classify/eval_gtzan.py --model /root/autodl-tmp/audio_outputs/wav2vec2-gtzan-15s/checkpoint-4480
uv run python asr/train_whisper_minds14.py --max-steps 250
uv run python tts/finetune_speecht5.py --use-cache   # reads the cached prepared dataset
uv run python ch7/speech_to_speech.py --audio outputs/dutch_sample.wav
uv run python ch7/transcribe_meeting.py
uv run python ch7/voice_assistant.py --audio-file outputs/dutch_sample.wav   # local LLM

Local edge LLM for the voice assistant

Stage 3 uses a local llama.cpp GGUF model (default unsloth/Qwen3-0.6B-GGUF Qwen3-0.6B-Q4_K_M.gguf, ~397 MB) so it runs on CPU with no cloud call / no HF token. Options:

# CPU (edge-friendly) — the default
uv run python ch7/voice_assistant.py --audio-file outputs/dutch_sample.wav
# better quality model (1.7B) and/or offload to the GPU
uv run python ch7/voice_assistant.py --audio-file outputs/dutch_sample.wav \
    --llm-file Qwen3-1.7B-Q4_K_M.gguf --llm-repo unsloth/Qwen3-1.7B-GGUF \
    --n-gpu-layers 6

The GGUF is pulled into the HF cache on first use. --llm-ctx, --llm-threads (default capped at 16 — too many CPU threads stalls small models), --n-gpu-layers (0 = pure CPU) are configurable. If the model can't be loaded the pipeline falls back to echoing the question.

Voice assistant GUI (Gradio)

ch7/voice_assistant_gui.py wraps the same four-stage pipeline in a multi-turn chat web app — record/upload audio, hear back a spoken reply, keep chatting. The LLM carries the conversation history, so it behaves like a proper assistant instead of the single-shot CLI.

uv run python ch7/voice_assistant_gui.py                             # http://localhost:7860
uv run python ch7/voice_assistant_gui.py --share                     # public gradio.live link
uv run python ch7/voice_assistant_gui.py --no-preload                # skip warm-up (slower 1st turn)
uv run python ch7/voice_assistant_gui.py --n-gpu-layers 6            # offload LLM layers to GPU
uv run python ch7/voice_assistant_gui.py --llm-file Qwen3-1.7B-Q4_K_M.gguf \
    --llm-repo unsloth/Qwen3-1.7B-GGUF                               # swap the LLM

Notes:

• Mic works even on this headless server. Gradio records audio in the browser (WebRTC), so the operator just opens the page and talks — no /dev/snd needed on the box. File upload is the server-side path when no mic is present.
• Stages load lazily and are reused across turns (whisper + SpeechT5 + the local LLM are cached; the wake-word AST classifier loads only if you tick the checkbox).
• Models are warmed up at startup (first turn is fast); --no-preload skips that.
• Multi-turn turns trim history to the last ~20 messages before querying the LLM so it stays inside --llm-ctx.
• The spoken reply is written to outputs/gui_tts_latest.wav (gitignored).
• LLM/device load options (--llm-file/-repo/-ctx/-threads/--n-gpu-layers) are fixed at launch — the singleton can't reload mid-session. Temperature is a slider.

Notes:

• datasets 5 dropped script-based loaders, so GTZAN uses the parquet sanchit-gandhi/gtzan and speaker embeddings are computed with SpeechBrain instead of the script-based Matthijs/cmu-arctic-xvectors.
• Large datasets/checkpoints live under /root/autodl-tmp/audio_outputs (the 50 GB data disk) — the small system disk has no room for them.