Musica VJ — Your AI DJ + VJ Rig

The live studio: audio-reactive Three.js visuals center stage, the Visual Bank and VJ presets on the left, and the synchronized Lyria RealTime decks (Main / Sequence / Vocalize) with the genre grid on the right — over a 16-step Lyria beat with SFX and loop lanes, MIDI + MX Console control, and one-click capture. The first run opens with the Cognitum One sign-in gate — sign in and Lyria audio just works, no bring-your-own key.

Musica VJ is a free desktop app for musicians, producers, and DJs. Hit play and it generates a live, evolving beat in real time, then lights up the screen with visuals that react to the music automatically — no timeline to program, no separate VJ software, no second laptop. Point a MIDI controller, your keyboard, or a Logitech MX Creative Console at it and perform the whole set — music and visuals — like one instrument.

Under the hood it's also an agent-directed music workstation, a governed creative-AI provider shell, and a Rust DSP research engine — but you don't need to know any of that to use it.

👉 See it in action, screenshots, and download links →

The flagship app is apps/musica-vj: a Tauri 2 desktop studio with synchronized Lyria RealTime arrangement and beat streams, a guarded vocalization lane, editable rhythm guidance, AI-directed performance templates, one-click Demo automation, Meta-LLM set planning, governed Gemini/Lyria song and loop generation, audio-reactive Three.js visuals, temporal VJ controls, Logitech MX Creative Console integration, browser MIDI, and release workflows for macOS, Linux, and Windows.

Current Direction

The table below is the detailed technical status, mainly useful for contributors. For a plain-language tour, see the website.

v0.3.0 — Meta-LLM Performance Layer

The v0.3.0 release ships the AI performance layer end to end. Everything degrades gracefully: with no account, every feature falls back to deterministic local planners, keyword mappings, and the operator's remembered history.

Web + WASM Direction

The browser version should use Web Audio and Three.js for the performance UI, then load Musica core through WASM for heavier analysis and structure-aware DSP. The repo already has a feature-gated src/wasm_bridge.rs FFI surface for the graph separation pipeline:

rustup target add wasm32-unknown-unknown
cargo build --target wasm32-unknown-unknown --features wasm --release

The intended split is pragmatic:

That keeps the web app usable without a server while preserving the stronger native security boundary for API tokens and paid generation.

Run The Studio

cd apps/musica-vj
npm ci
npm run dev

# Native desktop shell with provider commands, global shortcuts, and packaged-app behavior
npm run tauri dev

Optional AI providers are configured only through the Tauri process environment. Do not commit API tokens, .env files, generated provider responses, or shell snippets containing secrets.

MUSICA_META_LLM_ENABLED=true \
MUSICA_META_LLM_API_BASE=https://api.cognitum.one \
MUSICA_META_LLM_API_TOKEN=replace_with_token \
MUSICA_META_LLM_MODEL=meta-llm \
npm run tauri dev

Google Lyria/Gemini generation can use either a Rust-only GEMINI_API_KEY or local GCP CLI application-default credentials with MUSICA_GCP_AUTH=gcloud.

See apps/musica-vj/README.md for the complete studio documentation, controller mappings, MIDI notes, import behavior, provider setup, screenshots, release artifacts, and verification commands.

Musica Core

Under the live studio is the original Rust audio engine: structure-first audio source separation via graph Laplacian spectral clustering and dynamic mincut refinement. It remains useful as a standalone library for low-latency audio analysis, separation, embedded processing, and research.

Why Structure-First?

Traditional audio separation is frequency-first: FFT masking, ICA, NMF, neural networks. These approaches separate by learned spectral patterns.

Musica is structure-first: reframe audio as a graph partitioning problem, then find where signals naturally divide.

Nodes  = time-frequency atoms (STFT bins, critical bands)
Edges  = similarity (spectral proximity, phase coherence, harmonic alignment, temporal continuity)
Weights = how strongly two elements "belong together"

Dynamic mincut finds the minimum-cost boundary where signals separate, preserving maximum internal coherence within each source. The Fiedler vector (2nd smallest eigenvector of the graph Laplacian) provides the geometric partition that approximates the normalized cut.

Competitive Position

Latency Comparison

Embedded Viability

Separation Quality (honest assessment)

Musica is 5-8 dB behind neural SOTA on raw SDR. That gap is expected — learned models have seen thousands of labeled songs. Musica's advantages are latency, size, interpretability, and edge deployability.

Architecture

Raw Audio
    |
    v
STFT / Filterbank ──────── Zero-dep radix-2 Cooley-Tukey FFT + Hann window
    |
    v
Graph Construction ──────── Spectral + temporal + harmonic + phase edges
    |
    v
Laplacian Eigenvectors ──── Fiedler vector via Lanczos / power iteration
    |                        SIMD-friendly (chunk-of-4 auto-vectorization)
    v
Spectral Clustering ─────── Balanced initial partition (normalized cut)
    |
    v
MinCut Refinement ───────── Boundary optimization via ruvector-mincut
    |
    v
Soft Mask Generation ────── Distance-weighted softmax, Wiener normalization
    |
    v
Overlap-Add Reconstruction

Modules

Quick Start

# Build
cargo build --release

# Run full 6-part benchmark suite
cargo run --release

# Run tests (34 tests)
cargo test

Musica VJ Studio

apps/musica-vj is a Mac-first Tauri 2 performance instrument built on Musica. It combines synchronized Lyria RealTime decks, an editable beat-guidance grid, one-click Demo automation, audio-reactive Three.js scenes, official Logitech MX Creative Console controls, prompt-driven performance mutation, governed creative-music providers, and capability-probed vertical social capture that prefers H.264 and AAC.

cd apps/musica-vj
npm ci
npm run dev

# Launch the native Tauri application on macOS
npm run tauri dev

The official Logi Actions SDK companion maps the Keypad LCD buttons, Dialpad, and roller into Musica controls. An Options+ F13-F24 profile works as an installation-free fallback. Creative provider credentials remain in the Rust process. Google Lyria 3 Pro Preview is optional and disabled by default, while the Suno partner adapter stays disabled until documented official access is configured.

Reproducible vertical preview fixtures are checked in for quick review:

• Signal Bloom
• Spectral Field

See Musica VJ documentation, the Lyria integration specification, and ADRs 160-169 for architecture, security, controller setup, performance budgets, export rules, paid generation, provenance, and release gates.

Usage

Basic Two-Source Separation

use musica::{stft, audio_graph, separator};

let stft_result = stft::stft(&signal, 256, 128, 8000.0);
let graph = audio_graph::build_audio_graph(&stft_result, &audio_graph::GraphParams::default());

let config = separator::SeparatorConfig {
    num_sources: 2,
    ..separator::SeparatorConfig::default()
};
let result = separator::separate(&graph, &config);

// result.masks[i] — soft mask per source
// result.cut_value — mincut witness (separation confidence)

Hearing Aid Streaming

use musica::hearing_aid::{HearingAidConfig, StreamingState, Audiogram};

let config = HearingAidConfig {
    audiogram: Audiogram {
        frequencies: vec![250.0, 500.0, 1000.0, 2000.0, 4000.0, 8000.0],
        gains_db: vec![10.0, 15.0, 20.0, 30.0, 40.0, 50.0], // mild sloping loss
    },
    ..HearingAidConfig::default()
};
let mut state = StreamingState::new(&config);

// Per-frame streaming (call every 4ms hop)
let result = state.process_frame(&left_mic, &right_mic, &config);
// result.mask         — per-band speech/noise mask
// result.speech_score — overall speech probability
// result.latency_us   — processing time for this frame

Pipeline per frame:

1. Extract binaural features (ILD, IPD, IC, voicing, harmonicity) across 32 ERB bands
2. Build graph over rolling 5-frame window with spectral/temporal/harmonic edges
3. Compute Fiedler vector via 30-iteration power method on D^{-1}A
4. Dynamic mincut refinement for boundary stability
5. Speech/noise scoring (0.3 voicing + 0.25 harmonicity + 0.25 IC + 0.2 frontness)
6. Sigmoid sharpening + temporal smoothing (EMA)
7. Audiogram gain shaping (half-gain rule)

Multitrack 6-Stem Separation

use musica::multitrack::{separate_multitrack, MultitrackConfig, Stem};

let config = MultitrackConfig {
    window_size: 4096,
    hop_size: 1024,
    sample_rate: 44100.0,
    ..MultitrackConfig::default()
};
let result = separate_multitrack(&audio_signal, &config);

for stem in &result.stems {
    println!("{:?}: confidence={:.3}", stem.stem, stem.confidence);
    // stem.signal — reconstructed time-domain audio for this stem
    // stem.mask   — T-F soft mask
}

// result.replay_log — every mincut decision for reproducibility

Default frequency priors:

Crowd-Scale Speaker Tracking

use musica::crowd::{CrowdTracker, CrowdConfig, SpeechEvent};

let config = CrowdConfig {
    max_identities: 500,
    association_threshold: 0.4,
    ..CrowdConfig::default()
};
let mut tracker = CrowdTracker::new(config);

// Register sensors
tracker.add_sensor((0.0, 0.0));
tracker.add_sensor((10.0, 0.0));

// Ingest events from sensor 0
tracker.ingest_events(0, vec![SpeechEvent {
    time: 0.0, freq_centroid: 200.0, energy: 0.5,
    voicing: 0.8, harmonicity: 0.7, direction: 0.0, sensor_id: 0,
}]);

// Update pipeline
tracker.update_local_graphs();          // Layer 2: local Fiedler clustering
tracker.associate_cross_sensor(0.5);    // Layer 3: cross-node embedding match
tracker.update_global_identities(0.5);  // Layer 4: global identity memory

let stats = tracker.get_stats();

4-layer hierarchy:

1. Local events — Raw acoustic detections per sensor
2. Local speakers — Fiedler vector bipartition on per-sensor similarity graph (Gaussian kernel: time, frequency, energy, direction)
3. Cross-sensor association — Cosine similarity on speaker embeddings across overlapping sensor regions
4. Global identities — Exponential moving average embedding merging with confidence tracking

Lanczos Eigensolver (standalone)

use musica::lanczos::{SparseMatrix, LanczosConfig, lanczos_eigenpairs, batch_lanczos};

// Build graph Laplacian from weighted edges
let laplacian = SparseMatrix::from_edges(20, &edges); // L = D - W

// Compute smallest k eigenpairs
let config = LanczosConfig { k: 4, max_iter: 50, tol: 1e-8, reorthogonalize: true };
let result = lanczos_eigenpairs(&laplacian, &config);
// result.eigenvalues  — sorted ascending
// result.eigenvectors — Fiedler vector is eigenvectors[0] (smallest non-trivial)

// Batch mode with cross-frame alignment (Procrustes sign consistency)
let results = batch_lanczos(&laplacians, &config);

WAV I/O

use musica::wav;

// Read
let data = wav::read_wav("input.wav")?;
// data.channel_data[0] — first channel as Vec<f64>
// data.sample_rate, data.channels, data.bits_per_sample

// Write
wav::write_wav("output.wav", &samples, 16000, 1)?;

// Generate binaural test signal with ITD model
wav::generate_binaural_test_wav("test.wav", 16000, 0.5, 300.0, &[800.0], 30.0)?;

Benchmark Results

Run cargo run --release for the full 6-part suite:

Part 1: Basic Separation

Three test scenarios at 8 kHz, 256-sample window:

Part 2: Hearing Aid Streaming

Part 3: Multitrack 6-Stem

Graph: 24,230 nodes, 55,541 edges. Mask sum error: 0.0000.

Part 4: Lanczos Validation

20-node graph, 2 clusters with weak bridge:

• Fiedler clean split: YES
• Eigenvalues: [0.889, 2.041, 36.845, 60.425]
• Lanczos converged in 4 iterations

Part 5: Crowd-Scale Tracking

20 sensors, 1,500 events, 50 simulated speakers:

• Global identities resolved: 3
• Active speakers: 3
• Processing time: 97 ms

Part 6: WAV I/O

16-bit PCM roundtrip: max error = 0.000046. PASS.

Key Algorithms

Fiedler Vector Spectral Clustering

The graph Laplacian L = D - W encodes structure. Its second-smallest eigenvector (the Fiedler vector) provides the continuous relaxation of the normalized cut — nodes with the same sign in the Fiedler vector belong to the same cluster.

Given weighted adjacency W and degree matrix D:
  L = D - W
  Solve Lv = λv for smallest eigenvalues
  Fiedler vector = eigenvector for λ₂ (smallest non-zero eigenvalue)
  Partition: {nodes where v[i] > 0} vs {nodes where v[i] ≤ 0}

SIMD-Friendly Lanczos Iteration

All vector operations (dot, norm, axpy, scale) process in chunks of 4 f64 values for auto-vectorization. Selective reorthogonalization prevents ghost eigenvalues. Tridiagonal QR with Wilkinson shift extracts eigenpairs.

Dynamic MinCut Refinement

After spectral clustering provides balanced initial partitions, ruvector-mincut refines boundaries by finding the exact minimum cut. The cut value serves as a structural witness — a provable certificate of separation quality.

ERB Critical Bands

The hearing aid module uses 32 Equivalent Rectangular Bandwidth (ERB) spaced bands, matching the human cochlea's frequency resolution:

ERB(f) = 24.7 * (4.37 * f/1000 + 1)

What This Enables

Hearing Aids (product-ready)

The only sub-1ms, zero-dependency, fully explainable speech enhancer. Runs on a $2 microcontroller. No custom silicon required. An audiologist can inspect why any decision was made — which binaural features drove the speech/noise classification, what the graph partition looks like, what the mincut witness value means.

Regulatory advantage: FDA/CE medical device approval increasingly requires explainability. Black-box DNNs face scrutiny. Full auditability is a structural advantage for certification.

Browser Audio Processing

Compiles to WASM via wasm-pack with zero changes. Real-time separation in any browser AudioWorklet — no server round-trip. Applications: live transcription, teleconferencing, accessibility tools.

Hybrid Neural+Graph Pipelines

Use Musica's Fiedler partition as a preprocessing stage for lightweight neural models. The graph provides structural priors, reducing what the neural model needs to learn. Potential to reach 8+ dB SDR at <2ms latency by combining graph structure with a small learned refinement network.

Cochlear Implant Preprocessing

CI users need even lower latency than hearing aid users. At 0.20ms, Musica leaves headroom for additional processing stages (vocoder, electrode mapping) within tight latency budgets.

Smart Environments

Crowd-scale tracking enables: smart buildings with per-room speaker awareness, transit hub safety monitoring, stadium crowd analytics, search and rescue with distributed microphone arrays.

Improvement Roadmap

Near-term (quality gains)

• Real audio evaluation — Benchmark on MUSDB18, VCTK, LibriMix with proper SDR/SIR/SAR
• Adaptive graph parameters — Learn edge weights from a small labeled set (few-shot)
• Multi-resolution STFT — Different window sizes for transients vs tonal content
• Phase-aware reconstruction — Griffin-Lim or learned phase estimation instead of magnitude-only masking

Medium-term (hybrid architecture)

• Neural mask refinement — Small CNN/RNN (< 100K params) to refine graph-based masks
• Learned embeddings — Replace hand-crafted features with a tiny encoder
• WASM deployment — wasm-pack build + browser demo with Web Audio API
• MUSDB18 benchmark entry — Formal SDR evaluation for competition ranking

Long-term (platform)

• Streaming multitrack — Frame-by-frame 6-stem separation (currently batch)
• Distributed crowd consensus — Byzantine-fault-tolerant identity resolution
• Hardware acceleration — FPGA/ASIC graph partitioning for sub-microsecond latency
• Formal verification — Prove separation guarantees via mincut certificates

Project Structure

docs/examples/musica/
├── Cargo.toml
├── README.md
└── src/
    ├── lib.rs            # Module declarations
    ├── main.rs           # 6-part benchmark suite
    ├── stft.rs           # FFT + STFT/ISTFT
    ├── lanczos.rs        # Sparse eigensolver (CSR, SIMD)
    ├── audio_graph.rs    # Graph construction from STFT
    ├── separator.rs      # Spectral clustering + mincut
    ├── hearing_aid.rs    # Binaural streaming enhancer
    ├── multitrack.rs     # 6-stem music separator
    ├── crowd.rs          # Distributed speaker tracking
    ├── wav.rs            # WAV file I/O
    ├── benchmark.rs      # SDR/SIR/SAR evaluation
    └── hearmusica/       # Hearing aid DSP pipeline
        ├── mod.rs        # Pipeline orchestrator + AudioBlock
        ├── block.rs      # ProcessingBlock trait
        ├── filter.rs     # BiquadFilter (8 filter types)
        ├── compressor.rs # WDRCompressor (multi-band WDRC)
        ├── feedback.rs   # FeedbackCanceller (NLMS adaptive)
        ├── gain.rs       # GainProcessor (NAL-R prescription)
        ├── separator_block.rs # GraphSeparator (Fiedler + mincut)
        ├── delay.rs      # DelayLine (circular buffer)
        ├── limiter.rs    # Limiter (brick-wall protection)
        ├── mixer.rs      # Mixer (weighted combination)
        └── presets.rs    # 4 preset pipelines

Dependencies

Single dependency:

[dependencies]
ruvector-mincut = { path = "../../../crates/ruvector-mincut", features = ["monitoring", "approximate", "exact"] }

Everything else — FFT, filterbank, eigensolver, WAV I/O, metrics — is implemented from scratch with zero external crates.

HEARmusica — Rust Hearing Aid Framework

High-fidelity Rust port of Tympan's MIT-licensed hearing aid DSP, integrated with musica's graph-based separation. HEARmusica provides a modular pipeline of processing blocks that can be composed into complete hearing aid signal chains, from microphone input to speaker output. Each block implements the ProcessingBlock trait for uniform pipeline orchestration.

Processing Blocks

Architecture

Input -> BiquadFilter -> FeedbackCanceller -> GraphSeparator -> WDRCompressor -> GainProcessor -> Limiter -> Output

The pipeline processes stereo AudioBlock frames. Each block reads from and writes to the block's left and right sample buffers in place, minimizing allocations. The GraphSeparator block bridges musica's spectral clustering into the hearing aid chain, providing structure-aware noise reduction that traditional DSP pipelines lack.

Preset Pipelines

Four preset configurations cover common hearing aid use cases:

All presets accept an Audiogram, sample rate, and block size, and return a fully configured Pipeline.

Usage Example

use musica::hearmusica::{self, Pipeline, AudioBlock};
use musica::hearing_aid::Audiogram;

let audiogram = Audiogram::default(); // mild sloping loss
let mut pipeline = hearmusica::presets::speech_in_noise(&audiogram, 16000.0, 128);
pipeline.prepare();

let mut block = AudioBlock::new(128, 16000.0);
// Fill block.left and block.right with mic samples...
pipeline.process_block(&mut block);
// block now contains enhanced audio

Comparison vs Tympan

HEARmusica's primary advantage is the GraphSeparator block, which has no equivalent in Tympan or any other open-source hearing aid framework. By embedding musica's spectral clustering directly into the DSP pipeline, noise reduction becomes structure-aware rather than purely energy-based.

HEARmusica Benchmark Results

4 preset pipelines benchmarked at 16 kHz, 128-sample blocks, 200 blocks each:

Key findings:

• Standard and music presets process in <0.05 ms — 160x under the 8ms budget
• Speech-in-noise preset with graph separation: 0.7 ms max — 11x under budget
• Max clarity with all blocks including delay alignment: 0.75 ms max — 10x under budget

Streaming 6-Stem Results

Frame-by-frame multitrack separation at 44.1 kHz:

ADR Reference

See ADR-143 for the full architecture decision record.

References

• Stoer-Wagner minimum cut algorithm
• Spectral clustering via graph Laplacian (Shi & Malik, 2000)
• Lanczos iteration with selective reorthogonalization (Parlett & Scott, 1979)
• ERB scale and auditory filters (Glasberg & Moore, 1990)
• BSS_EVAL metrics for source separation (Vincent et al., 2006)
• BS-RoFormer (Sound Demixing Challenge 2023)
• MUSDB18 benchmark dataset (Rafii et al., 2017)
• Pseudo-deterministic canonical minimum cut (Kenneth-Mordoch, 2026)