AI Composer

Give it a fictional anime story brief — it composes the opening theme, listens to its own work, and revises until it passes.

AI Composer takes a short fictional anime story brief (in Japanese) and autonomously produces an opening-theme (OP) song as real, sung Japanese audio. It then reviews each attempt with a deterministic, pure-code scorer (the hard gate) plus an LLM acting as an advisory critic, and iterates — re-planning and regenerating — until the result clears a quality threshold. Every run leaves a fully traceable evidence trail of plans, scores, and audio.

This is a research prototype, not a product.

---

What it does

• Plans from a story. A large language model reads the brief and writes the OP as natural-language stage directions (e.g. "quiet seaside intro on solo piano → rising tension → full-band climax"). Deterministic code then translates that language into machine-readable musical targets (mode, tempo range, energy arc).
• Generates real sung audio. An audio model sings the planned theme in Japanese — no symbolic-MIDI step, no separate vocal synthesizer.
• Reviews itself on two layers.
  • A pure-code objective reviewer is the hard pass/fail gate: it is deterministic (same input → same score) and reproducible. It scores narrative arc, hook, musical soundness, emotional congruence, and vocal/lyric delivery.
  • An LLM critic is advisory only — it offers suggestions but never decides pass/fail.
• Searches by evolution. Each round generates several candidates plus one bold "inspiration mutation"; the best-scoring candidate is kept (elitism), so quality only rises across rounds.
• Leaves an evidence trail. Each run writes the brief, the plan, the per-round scores, a radar chart, a loop curve, the chosen audio, and a CHAIN.md that walks through the whole decision path.

See ARCHITECTURE.md for the full pipeline and scoring design.

---

Requirements

• GPU(s) — generation and review run local models; an A100-class GPU (or comparable) is recommended. (Older GPUs without bf16 support are not recommended for the audio backbone.)
• A served LLM — Qwen2.5-32B-Instruct via an OpenAI-compatible endpoint (e.g. vLLM).
• An audio backbone — ACE-Step 1.5 (Apache-2.0), installed under external/.
• Lyric-scoring stack — Demucs (source separation) and faster-whisper (Japanese ASR).
• Sampleal — Qwen2-Audio-7B-Instruct for the advisory "listen to the audio" critic.
• Python — see requirements.txt.

---

Installation & configuration

# 1. Clone
git clone https://i676974687562o636f6dz.oszar.com/<user>/ai-composer.git
cd ai-composer

# 2. Python environment
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# 3. Audio backbone (ACE-Step 1.5) into external/
bash scripts/setup_acestep.sh        # clones ACE-Step into external/ (gitignored)

# 4. Configure paths
cp .env.example .env
#   edit .env:
#     MODEL_DIR=/path/to/your/models      # where model weights live
#     HF_HOME=/path/to/your/hf-cache

Download the model weights (Qwen2.5-32B-Instruct, ACE-Step 1.5, and — if you want the audio critic — Qwen2-Audio-7B-Instruct) into MODEL_DIR. Demucs and faster-whisper weights are fetched on first use into HF_HOME.

Serve the LLM (OpenAI-compatible endpoint, default http://127.0.0.1:8000/v1):

# Example with vLLM; adjust to your setup
python -m vllm.entrypoints.openai.api_server \
  --model "$MODEL_DIR/Qwen2.5-32B-Instruct" --port 8000

Cluster users: sanitized Slurm launch examples are in scripts/cluster/ — edit paths, partitions, and node names for your environment.

---

Running

Run a brief through the full plan → generate → review → iterate loop:

python evaluation/run_acestep_audio_loop.py \
  --brief data/briefs/swarm_tide.json \
  --endpoint http://127.0.0.1:8000/v1 \
  --out outputs/

Results land in outputs/<slug>_<timestamp>/:

outputs/swarm_tide_20260102_120000/
├── step1_kikaku_brief.json        # the input brief
├── step2_llm_plan.json            # LLM stage directions + code-derived targets
├── step3_ace_conditions.json      # generation conditions (caption, lyrics, bpm/key/seed)
├── round_01/ … round_NN/          # per-round samples, scores, radar, selection
│   ├── step5.1_objective.json     # pure-code scores (the gate)
│   ├── step5.2_radar.(png|svg)    # score radar
│   ├── step5.3_selected.json      # chosen sample this round
│   └── step5.4_lyric_alignment.json
├── step6_final.json               # final selection
├── step6_loop_curve.(png|svg)     # score vs round, with threshold
└── CHAIN.md                       # human-readable walkthrough of the whole run

(Audio files are written into the run folder and are gitignored.)

Smoke test (no GPU required):

bash scripts/smoke_test.sh

This runs a deterministic, GPU-free subset — module imports, plan derivation on a canned plan, the objective scorer on a synthetic clip, and the language→target translator — to verify the core logic. The real end-to-end run requires the local models above.

---

Project layout

ai-composer/
├── orchestrator/    # the bounded plan→generate→review→iterate loop
├── compose/         # planning, language→target translation, ACE conditions, generation, naming
├── review/          # objective (pure-code gate), lyric-alignment chain, subjective (advisory)
├── evaluation/      # the run loop + validation scripts
├── scripts/         # smoke test + sanitized cluster examples
├── data/briefs/     # original fictional anime briefs (the input dataset)
└── examples/        # a sample brief + small expected artifacts

---

Acknowledgements

AI Composer builds on excellent open models and tools:

• ACE-Step — audio/song generation (Apache-2.0).
• Qwen2.5-32B-Instruct and Qwen2-Audio-7B-Instruct — planning, critique, and audio understanding.
• Demucs — music source separation.
• faster-whisper — Japanese automatic speech recognition.
• librosa — audio feature analysis.

Please consult each project's license for its own terms.

---

License

Released under the MIT License — see LICENSE. Third-party models and tools listed above are governed by their own licenses.