Open-Source AI Music Video Generator for Consumer GPUs

Turn any song into a lip-synced music video using AI — fully local, no cloud APIs required.

How It Works · Quick Start · Installation · Hardware · Pipeline Spec · Status

---

What Is MusicVision?

MusicVision is a source-available Python pipeline that generates AI music videos from a song file and a character reference image. It segments your song into scenes, generates a storyboard, renders lip-synced video clips, upscales them, and assembles a rough cut ready for editing in DaVinci Resolve.

song.wav + lyrics.txt + reference.png
  → scene segmentation → storyboard → lip-synced video clips → upscaled → rough cut MP4 + FCPXML

Everything runs locally on consumer NVIDIA GPUs. No cloud services, no per-minute billing, no data leaving your machine.

Key Features

• Lip-synced video generation — characters sing along to the music with accurate mouth movements, facial expressions, and body motion driven by the audio
• Multiple AI video engines — HuMo (audio-reactive lip sync), LTX-Video 2 (cinematic)
• Multiple image engines — Z-Image (ungated, fast) and FLUX (LoRA support for character consistency)
• Three upscalers — SeedVR2 (pixel-space), LTX Spatial (latent-space), Real-ESRGAN (fast preview)
• Frame-accurate audio sync — integer frame math eliminates drift; original uncut audio in final assembly
• Sub-clip chaining — scenes longer than the engine's max clip length are automatically split with visual continuity (last frame → next reference image)
• Professional export — rough cut MP4, EDL, and FCPXML 1.10 for DaVinci Resolve
• Iterative workflow — review and regenerate individual scenes via React GUI or CLI; approve only what you like
• Per-scene engine selection — use different video engines for different scenes in the same project
• Fully local LLM option — vLLM with Qwen3.6-27B on a LAN server for scene segmentation and prompt generation, or use Claude API, or skip LLM entirely with auto-templates
• Config-driven projects — every project is a YAML + JSON directory; reproducible, version-controllable, shareable

Current Status

All five pipeline stages are code-complete and GPU-tested. The React storyboard GUI is functional. End-to-end pipeline test passed (2026-03-01). See STATUS.md for full details.

---

How It Works

MusicVision wraps multiple AI models into a five-stage pipeline with user review at each step:

1. Intake & Segmentation — Whisper transcription + LLM-assisted segmentation splits the song into 2–10 second scenes aligned to musical phrasing and section boundaries. AceStep JSON metadata (BPM, section markers) is auto-imported if available.

2. Image Generation & Storyboard — Z-Image or FLUX generates a reference image for each scene. Users review the storyboard grid and regenerate individual scenes until satisfied.

3. Video Generation — HuMo or LTX-Video 2 renders each scene. Long scenes are split into sub-clips with visual continuity chaining. Each engine has draft and production presets.

4. Upscaling — Per-engine upscaler selection: LTX Spatial for LTX-2 output (latent-space), SeedVR2 for HuMo (pixel-space), Real-ESRGAN for fast preview. Configurable target resolution (720p–4K, default 1080p).

5. Assembly & Export — Clips concatenated with the original uncut audio, exported as MP4 + EDL + FCPXML for DaVinci Resolve. Assembly enforces a duration assertion within one frame tolerance.

Video Engine Comparison

Note: HunyuanVideo-Avatar (HVA) was previously supported but was deprecated and removed in commit 35cda2a (2026-03-11).

---

Quick Start

# 1. Clone and install
git clone https://github.com/tsondo/musicvision.git
cd musicvision
bash setup_env.sh            # installs Python 3.11, PyTorch + CUDA, all deps

# 2. Configure
cp .env.example .env         # edit: set HUGGINGFACE_TOKEN (for FLUX), optionally ANTHROPIC_API_KEY

# 3. Create a project
musicvision create ./my-video --name "My Music Video"
musicvision import-audio --project ./my-video --audio song.wav --lyrics lyrics.txt

# 4. Run the pipeline
musicvision intake --project ./my-video --skip-transcription
musicvision generate-images --project ./my-video --model z-image-turbo
musicvision generate-video --project ./my-video --engine humo
musicvision upscale --project ./my-video --resolution 1080p
musicvision assemble --project ./my-video
# → my-video/output/rough_cut.mp4

# Or use the GUI instead:
musicvision serve                                # API server (no project — create from frontend)
cd frontend && npm install && npm run dev        # React UI at http://localhost:5173

---

Hardware Requirements

Linux / WSL2 / Windows (CUDA)

The primary GPU runs DiT/UNet inference (FLUX, video engines). The secondary GPU handles text encoders, VAE, Whisper, and audio separator. GPU roles are assigned automatically by VRAM — the highest-VRAM GPU becomes primary regardless of CUDA index. Models are fully unloaded between stages — FLUX and video engines never run simultaneously.

A single-GPU setup works if the card has ≥32 GB VRAM. The two-GPU split is a consumer hardware optimization, not a requirement.

Cloud (A100 / H100 / H200)

Single-GPU A100 80 GB or H100 80 GB runs the full FP16 model without splitting. No multi-GPU complexity needed. Minor gaps remain in tier auto-selection for high-VRAM single GPUs — see future_plans.md.

Apple Silicon (MPS) — Planned

M-series Mac support is planned but not yet implemented. Blocking issues include RoPE float64/complex128 ops and FP8 unavailability on MPS. See future_plans.md for the roadmap.

Optional: Local LLM Server

A separate GPU (e.g. RTX 3090 Ti 24 GB) can run vLLM with Qwen3.6-27B-AWQ-INT4 for scene segmentation and prompt generation, eliminating the Claude API dependency entirely. This is optional — the pipeline works without it using auto-templates or the Claude API.

---

Installation

Requires Python 3.11+ and uv.

Automated Setup

git clone https://github.com/tsondo/musicvision.git
cd musicvision
bash setup_env.sh

The script installs uv if missing, creates a Python 3.11 venv, installs PyTorch with CUDA, installs MusicVision in editable mode, and runs the test suite.

Manual Setup

git clone https://github.com/tsondo/musicvision.git
cd musicvision

# Create venv and install base deps
uv sync --extra dev

# Install PyTorch with CUDA 12.8
uv run pip install torch torchvision torchaudio \
    --index-url https://download.pytorch.org/whl/cu128

# Audio separator (separate install due to onnxruntime conflict)
pip install "audio-separator[gpu]"

flash_attn is not required. PyTorch's native SDPA provides equivalent performance on modern GPUs. Install it separately only if you want it:

pip install flash-attn --no-build-isolation

Environment Variables

Copy .env.example to .env and configure:

---

GUI

The React frontend provides a storyboard-based workflow:

• Scene grid — lyrics, reference images, video clips, prompts, and approval status per scene
• Preview panel — full-size image and video playback with regeneration controls
• Per-scene engine selection — choose a different video engine for each scene
• Per-scene regeneration — re-render individual scenes without restarting the whole pipeline
• Model management — select and switch between image/video models
• Dark theme

musicvision serve ./my-video          # start API server
cd frontend && npm install && npm run dev   # React dev server at http://localhost:5173

The API server also provides Swagger UI at http://localhost:8000/docs for direct API access.

---

CLI Reference

Image models: flux-dev, flux-schnell, z-image, z-image-turbo Video engines: humo, ltx_video Upscalers: ltx_spatial, seedvr2, real_esrgan Resolutions: 720p, 1080p, 1440p, 4k

---

Project Structure

Each MusicVision project is a self-contained directory:

my-video/
├── project.yaml          # Config: engines, style sheet, generation params
├── scenes.json           # Scene list with timestamps, prompts, approval status
├── input/                # Source audio + lyrics
├── assets/               # Characters, props, settings, LoRAs
├── segments/             # Per-scene audio (full mix → video engines)
├── segments_vocal/       # Vocal stems (→ Whisper transcription)
├── images/               # Reference images per scene
├── clips/                # Generated video clips + sub-clips
└── output/               # rough_cut.mp4, timeline.edl, timeline.fcpxml

All intermediate artifacts are saved. You can re-enter the pipeline at any stage, regenerate individual scenes, and the final assembly always uses the original uncut audio.

---

Testing

# Unit tests — ~227 tests, no GPU, < 10 seconds
uv run pytest tests/ -v

# LLM prompt tests (requires vLLM server)
python scripts/test_vllm_prompts.py

# GPU image generation (Z-Image + FLUX)
python scripts/test_image_gen.py

# GPU video generation (HuMo)
python scripts/test_gpu_pipeline.py --tier fp8_scaled --steps 6

See TESTING.md for the full test strategy.

---

Documentation

---

How MusicVision Compares

MusicVision occupies a unique niche: end-to-end music video generation running fully local on consumer GPUs.

---

Built With

• HuMo (ByteDance) — audio-conditioned video with TIA mode
• LTX-Video 2 (Lightricks) — unified audio+video generation
• FLUX (Black Forest Labs) — text-to-image with LoRA support
• Z-Image (Tongyi) — fast ungated image generation
• SeedVR2 (ByteDance) — face-aware video upscaling
• Real-ESRGAN — fast general-purpose upscaling
• Whisper (OpenAI) — speech transcription and alignment
• Kim_Vocal_2 / Demucs — vocal separation
• FastAPI + React — API server and storyboard GUI
• ffmpeg — audio slicing, video concatenation, muxing
• vLLM — local LLM serving (optional)

---

Contact

• General inquiries: [email protected]
• Contributors: [email protected]

---

License

MusicVision's source code is licensed under the Apache License 2.0 — free for personal, academic, research, and commercial use. (Versions released before the Apache 2.0 relicensing commit remain under their original, more restrictive noncommercial terms; see git history.)

Code vs. model weights. The Apache 2.0 grant covers the MusicVision code only. It does not cover the AI model weights the pipeline downloads at runtime — each model keeps its own license, and several restrict commercial use. Notably, FLUX.1-dev and the FLUX IP-Adapters are non-commercial, and LTX-2 is free for commercial use only under a revenue threshold. Commercial users should route image generation through the permissively licensed Z-Image or FLUX.1-schnell models. Apache 2.0 does not lift any model-weight restriction. See THIRD-PARTY-NOTICES.md for the full per-model breakdown — you are responsible for complying with all applicable upstream licenses.