Music Video Suite

WAV in, finished MP4 out — with frame-level prompt control and selective regeneration matching the Neural Frames workflow, running fully local on your own GPU.

The suite ships as both a headless CLI (for batching 1000+ tracks) and a PySide6 desktop app (for interactive scene/frame editing). Image generation runs through FLUX, video generation through Hunyuan-Video — both via 🤗 diffusers, both supporting GGUF-quantised weights so they fit on consumer GPUs.

---

Highlights

• Real audio analysis — librosa-based BPM, beat tracking, downbeat estimate, and structural section detection (chroma + MFCC self-similarity, not just a novelty curve).
• Whisper transcription with word-level timestamps; lyrics auto-align to scenes for prompt context.
• Beat-synced scene segmentation — ~50–100 scenes, cuts snap to the beat grid and never cross section boundaries (no chorus split between two scenes).
• Per-frame prompt model — every scene contains one or more Frames, each storing its own image prompt, motion prompt, negative prompt, seed, and generated image/clip path.
• "Remake this frame" — single button regenerates one frame and rebuilds the final MP4. The previous prompt auto-populates so you tweak rather than retype.
• Modular generator backends — flux, hunyuan, and a pipeline-validating stub that produces real PNGs/MP4s so you can exercise the full system on a laptop without a GPU.
• GGUF support — point gguf_transformer_path at a quantised weight file for either model.
• FFmpeg assembly with optional crossfades, Ken Burns fallback for image-only frames, and a single high-quality re-encode pass for clean output.
• Atomic JSON project files — every mutation goes through Project.save(), so a crash mid-render never corrupts state.
• Headless CLI for batch — mvsuite batch ./inbox processes a folder of WAVs end-to-end.

---

Install

Conda (recommended)

conda env create -f environment.yml
conda activate mvsuite
pip install -e .

This pulls PyTorch with CUDA 12.1, ffmpeg, librosa, diffusers, transformers, openai-whisper, PySide6, and everything else.

Pip-only (no GPU)

pip install -e ".[gui]"
# add ".[gen]" if you also want the heavy diffusers/transformers stack

You'll still need ffmpeg on your PATH (brew install ffmpeg, apt install ffmpeg, or grab a static build).

---

Quickstart

CLI

# 1. Create a project from a WAV.
mvsuite new "My Track" --audio path/to/song.wav

# 2. Run the whole pipeline in one shot.
mvsuite run projects/my-track --vision "neon cyberpunk skyline at night, rainy streets"

# 3. Open the result.
start projects/my-track/output/final.mp4    # macOS: 'open'  Linux: 'xdg-open'

The first run downloads the FLUX and Hunyuan-Video weights to your HuggingFace cache (~30 GB on disk for full precision, ~12 GB for GGUF Q4).

GUI

mvsuite-gui

1. New project → pick the WAV.
2. Analyse audio (toolbar) → BPM/beats/sections appear in the timeline.
3. Transcribe → lyrics align to scenes.
4. Segment scenes → enter your master vision prompt.
5. Generate images → previews appear under each frame.
6. Click any frame → tweak the image/motion prompt → Remake this frame.
7. Generate clips → Assemble final → MP4 written to projects/<slug>/output/.

Batch mode

mvsuite batch ./inbox --output-root ./projects \
    --vision "warm vintage VHS aesthetic, slow zooms"

Walks ./inbox for .wav files and produces one finished MP4 per track. Failures are logged and the loop continues to the next track.

---

Configuration

The pipeline is driven by src/mvsuite/_resources/default.yaml (shipped inside the package via importlib.resources so it works for editable, wheel, and zip installs). Override with --config <yourfile.yaml> (whole file) or via project.json's config block (per-project overrides). Notable knobs:

---

Project layout on disk

projects/<slug>/
  project.json              ← single source of truth (atomic writes)
  audio/source.wav
  images/<scene>/<frame>.png
  clips/<scene>/<frame>.mp4
  output/final.mp4
  logs/

project.json is a Pydantic-validated ProjectManifest you can read or hand-edit if you ever need to.

---

Architecture

mvsuite/
├── audio/        librosa analysis, Whisper transcription
├── scenes/       Scene/Frame models, beat-aware segmenter, prompt builder
├── generators/   ImageGenerator/VideoGenerator base + flux + hunyuan + stub
├── render/       FFmpeg-based clip assembly with optional crossfades
├── gui/          PySide6 timeline + scene/frame editor (workers in QThread)
├── pipeline.py   End-to-end orchestrator (called from both CLI & GUI)
├── cli.py        Typer commands: new / analyse / segment / generate / regen / run / batch
├── project.py    On-disk project model
└── config.py     Layered YAML config

Why this shape:

• Backends are an interface, not an if/else chain. Adding a future video model (e.g. CogVideoX, Mochi-1) means writing one ~80-line adapter and registering it.
• Frame is the atomic regen unit. "Remake this frame" → image regen → video regen → final reassemble, no rebuild of the rest of the song. This is the property the brief was specifically asking for.
• GUI never touches the pipeline directly — every long job is a JobThread emitting Qt signals. The UI stays responsive while a 10-minute generation runs.
• The project file is the API. GUI mutations and CLI mutations both go through Project.save(), so you can edit a project in the GUI, close it, run mvsuite generate from a Slurm job, then reopen in the GUI. State is portable.

---

Development

# Run the test suite (uses the stub backends — no GPU required).
pytest

# Lint.
ruff check src tests

# Type-check.
mypy src

The smoke test in tests/test_pipeline_stub.py runs the full pipeline (audio → segment → image gen → video gen → assemble) against a synthetic 10s WAV and asserts that a real MP4 lands on disk.

---

Roadmap

---

License

MIT — see LICENSE.