Fragmenta is an open-source app that brings the complete generative AI text-to-audio pipeline to musicians: intuitive dataset creation, LoRA training, generation, audio editing, and live performance. After the initial installation, the app runs fully offline, and your data never leaves your device.

Built on Stable Audio 3, Fragmenta is designed for all musicians, and especially for experimental music and sonic arts practitioners, giving them the ability to shape the latent space with their own audio and musical data, no coding required. It reflects the small-data, model-bending, and artist-first approaches to AI that are central to my PhD research.

Compatibility: The beta version's engine has been removed; if you need it, use the v0.1.x-legacy tag.

---

Features

• Desktop app with a lightweight pywebview window and a pre-built React frontend
• Bulk auto-annotation — generate text prompts for your audio files via DSP analysis (Basic) or AI tagging with LAION-CLAP (Rich), with optional user-defined vocabulary
• Project-aware LoRA training with configurable rank, steps, learning rate, batch size, checkpoint frequency, and precision — trains directly on a Dataset Workbench project
• LoRA adapters — train LoRA, DoRA, or BoRA adapters (plus low-VRAM -xs variants) on top of a frozen *-base checkpoint for consumer GPUs; stack up to 4 at once with per-slot strength, bypass, and reorder at generation time
• Text-to-audio generation — variable-length clips (up to 120s small / 380s medium), with CFG scale, inference steps, seed control, and a multi-LoRA stack
• Audio editing — style transfer (audio-to-audio), region inpainting, and clip extension/continuation. Temporarily hidden: these are implemented, but the upstream Stable Audio 3 runtime does not yet produce usable results for them, so the controls are switched off rather than shipped broken. They return in a future release once upstream support lands.
• Checkpoint Manager — pick and download individual SA3 checkpoints (Small Music/SFX, Medium, and the matching *-base models) with per-item progress and hardware-compatibility hints
• Performance Mode — a 4-channel live sampler: per-channel effects (gain, pan, filter, delay, reverb), master dBFS metering, bars-mode generation, launch quantization (standalone or via Ableton Link), persistent sessions, named presets, and MIDI learn (see Performance Mode)
• Real-time GPU memory monitoring

---

Before You Start

Works on Windows, macOS, and Linux. Small Music/SFX models run on CPU, Apple Silicon, or GPU; Medium needs an NVIDIA GPU. After a one-time model download everything runs offline and on-device — the in-app Checkpoint Manager handles the downloads (see Authenticate & Download Models).

---

Run the App Locally

Requirements: Python 3.11 (download) — newer versions (3.12, 3.13) won't install the dependencies. On Mac, Fragmenta supports Apple Silicon (M1 or newer) on macOS 14 (Sonoma or newer); Intel Macs are not supported.

Open your Terminal or PowerShell and paste the following:

git clone https://github.com/MAz-Codes/fragmenta.git
cd fragmenta

Run the installer for your platform:

The installer verifies Python 3.11 is available, sets up a virtual environment, installs all dependencies, and launches Fragmenta. The first run takes a while; subsequent launches are faster — re-run the same script to start the app. The install is fully isolated: deleting the folder removes everything except Python itself.

---

Usage

0. Authenticate & Download Models

The app guides you through downloading models and authenticating with HuggingFace.

HuggingFace Token Requirement: the Stable Audio 3 weights are gated. You must:

1. Accept their license on the model page (while logged into your HF account).
2. Create a classic Read token and copy it into Fragmenta.

1. Dataset

Datasets are built in the Dataset Workbench (the Dataset tab) — an in-app workspace that takes you from raw audio to a captioned, training-ready dataset without spreadsheets or leaving the app. Everything is scoped to a project: a folder of audio clips and their text prompts.

1. Create or load a project — start fresh, or reopen an existing one to keep editing.

2. Add audio — ingest a folder by copy (duplicates the originals, safe) or symlink (references them in place, saves disk).

3. Inspect & slice — every clip shows a waveform and plays back inline. Slice long files into training-sized segments.

4. Annotate — type prompts by hand, or auto-annotate every clip (or just selected ones):

   Tier	Adds	Cost
   Basic	tempo (BPM) + musical key, via librosa	instant, CPU
   Rich	Basic + zero-shot genre / mood / instruments via LAION-CLAP	~2.35 GB one-time download

   A prompt template (Music / Instrument / SFX) shapes how the tags are formatted, and the Rich tier's CLAP vocabulary is editable so tags match your own taxonomy.

5. Check & create — a health strip flags empty or duplicate prompts, sub-1 s clips, and unsupported formats. Save keeps a restartable draft; Create Dataset writes the final, training-ready form. Optionally pre-encode latents here for a 5–10× per-step training speed-up.

The finished project shows up directly in the Training tab — no export step.

2. Train

Training produces a LoRA adapter against a *-base checkpoint (sa3-small-music-base, sa3-small-sfx-base, or sa3-medium-base).

Pick a project from the Dataset Workbench, choose an adapter type (dora-rows is the recommended default), set rank/steps/learning rate, and Start. Each step_<n>.safetensors checkpoint appears in the LoRA picker as it's written.

3. Generate

Use base or distilled models to generate audio from text prompts — up to 120 s on the Small models, 380 s on Medium. If you've trained a LoRA, pick one and dial in a multiplier to control how strongly the adapter biases the output.

Settings:

• CFG scale — how closely the model follows your prompt (higher = stricter)
• Inference steps — diffusion steps per generation. More = higher quality, proportionally slower. Locked to 8 for the distilled small model.
• Seed — control randomness; same seed produces identical results
• Batch generation — generate multiple variations at once

4. Performance

This is a 4-channel diffusion sampler designed for live performance use. Each channel has its own prompt, generated clip, transport, and effect chain.

Per-channel controls:

• Text prompt and Generate (batch 1–4 for variations)
• Gain, pan, bipolar filter (one knob: low-pass below center, high-pass above), delay, reverb
• Mute, solo, loop toggle
• Waveform display and per-channel level meter
• Fragment history — every clip a channel generates is kept; audition it, star to keep, or drag it onto the waveform to load

Master controls:

• Master fader with peak-hold dBFS metering
• Play All / Stop All, and Record the master output to WAV
• Master tempo (BPM) — drives bars-mode generation and launch quantization
• Launch quantization (Q) — snap launches to the grid (None, 1/32 → 8 Bars)
• Master reverb (selectable impulse response) and tempo-synced delay division
• Ableton Link toggle — sync tempo with any Link-enabled app on the local network. The small Link binding (~1–2 MB) isn't bundled; the app offers to install it the first time you enable Link.

Bars-mode generation: Switch a channel from sec to bars and pick a length (1, 2, 4, 8, or 16 bars). The clip is rendered to that bar count at the master BPM.

Launch quantization: With Q set to anything other than None, channel launches snap to the next quantum on the bar grid. Works standalone via an internal beat clock, or in sync with Ableton Link if enabled. The first launch (when nothing is playing) fires immediately and anchors the transport; subsequent launches snap.

Sessions and presets:

• Every panel setting (BPM, knobs, prompts, model choice, etc.) auto-persists, so reloads and tab switches don't lose state.
• Click the floppy-disk icon in the toolbar to save the current configuration as a named preset, or load a saved one. Restore defaults wipes the session and MIDI mappings if you want to start clean.

MIDI: Click MIDI in the toolbar to enter learn mode, then click any UI element and move a hardware knob/button to bind it. Mappings persist across sessions.

Other:

• Prompt injection — optional toggles append the master tempo (BPM), a prompt key, and a time signature to each prompt (BPM is skipped if the prompt already names one). Set from the bottom bar.
• Shared inference steps and seed with the Generation page.

---

Stats

Real generation times measured in Fragmenta — full clip (sampling + VAE decode), warm model, batch size 1. Yours will vary with hardware, prompt, and settings.

Test machines: NVIDIA RTX 5080 · AMD Ryzen 5 7600 (CPU, 12 threads) · Apple M1 (8-core)

Models: Small = Music or SFX (same 433M architecture, near-identical speed; up to 120 s). Medium = 1.4B, NVIDIA GPU only (requires Flash Attention 2; up to 380 s). All values in seconds.

Small (Music / SFX)

Medium (GPU only)

---

Project Structure

fragmenta/
├── app/
│   ├── backend/            # Flask API server
│   ├── frontend/
│   │   ├── build/          # Pre-built React app (served by Flask)
│   │   └── src/            # React source (only needed for development)
│   └── core/               # Core logic (generation, training, model management)
├── vendor/                 # Bundled third-party code
│   └── stable-audio-3/     # Stable Audio 3 (pinned snapshot; see UPSTREAM.md)
├── models/                 # Model configs and checkpoints
├── utils/                  # Utility modules
├── config/                 # Configuration files
├── fragmenta.sh            # Linux — first-time setup and subsequent launches
├── fragmenta.bat           # Windows — first-time setup and subsequent launches
├── fragmenta.command       # macOS — first-time setup and subsequent launches
├── install.py              # Creates the venv + installs dependencies (run by the launch scripts)
└── start.py                # Starts the backend + desktop window (run by install.py --launch)

---

Troubleshooting

---

Citation

If you use Fragmenta in your research, please cite it. You can use the Cite this repository button on GitHub (powered by CITATION.cff), or:

Azimi, M. (2026). Fragmenta (Version 1.0.0) [Computer software]. https://doi.org/10.5281/zenodo.20692998

BibTeX:

@software{azimi_fragmenta_2026,
  author  = {Azimi, Misagh},
  title   = {Fragmenta},
  year    = {2026},
  version = {1.0.0},
  doi     = {10.5281/zenodo.20692998},
  url     = {https://github.com/MAz-Codes/Fragmenta}
}

The DOI is a Zenodo concept DOI — it always resolves to the latest release.

---

Support

Fragmenta is free, open-source, and developed only by me as part of my PhD research on artist-first AI music tools. If it inspires you or powers your creativity, consider sponsoring its development. It directly funds maintenance, testing hardware, and documentation.

---

Contribution

Before proposing a substantial change, please read the contribution guidelines and file an issue for discussion before opening a pull request.

---

License

Copyright 2025-2026 Misagh Azimi

Licensed under the GNU Affero General Public License v3.0 — see LICENSE for the full text.

What this means in practice:

• You can use, study, modify, and redistribute Fragmenta freely.
• If you distribute Fragmenta (modified or not), or run a modified version as a hosted/network service, you must release your source code under the same AGPL-3.0 license.
• You must keep the copyright notice and attribution.

If you want to use Fragmenta under different terms (e.g. embedded in a closed-source commercial product), please reach out — as the sole copyright holder I'm open to discussing alternative licensing.

Third-Party Software

Fragmenta is Powered by Stability AI, using the Stable Audio 3 model family.

"This Stability AI Model is licensed under the Stability AI Community License, Copyright © Stability AI Ltd. All Rights Reserved"

• Vendored code — the Stable Audio 3 inference/training code is bundled under vendor/stable-audio-3/ (pinned snapshot; see UPSTREAM.md) under the MIT License, © 2026 Stability AI — see vendor/stable-audio-3/LICENSE.
• Model weights — not redistributed with Fragmenta; you download them from Hugging Face and accept their license then. Governed by the Stability AI Community License (free use up to USD $1M annual revenue; above that an enterprise license from Stability AI is required), and they include a T5Gemma text encoder under the Gemma Terms of Use. LoRA adapters you train on the *-base checkpoints are Derivative Works under the same Community License.
• Impulse responses — the bundled convolution-reverb impulse responses under app/frontend/public/ir/ are original, synthesised by Fragmenta, with no external licensing.

Fragmenta also depends on many open-source libraries. See NOTICE.md for the complete attribution and license list.