Audio8-TTS single-stream fast path

A small, standalone gpt-fast-style static-shape inference path for the public Audio8-TTS-Preview-0.6b text-to-speech model (the arktts architecture). It reuses the model's public weights through its own static-shape forward code and compiles the whole per-audio-frame step into a single CUDA graph — taking one batch-1 stream from RTF ~0.54 (eager reference) down to ~0.091 (~11x real time) on a laptop-class RTX 5090, which is close to the card's memory-bandwidth floor.

Nothing clever in the model itself is ours — Audio8 did that. This is just a careful rewrite of the inference loop around their weights, plus the profiling notes that got us there, in case they're useful to someone else pushing a small autoregressive TTS model.

• Write-up (how we got here, in prose): https://stuff.scrappylabs.ai/public/warble/writeup.html
• Our voices (fine-tuned on top of Audio8): https://huggingface.co/scrappylabsai/warble
• Base model (the upstream weights this runs on): https://huggingface.co/Audio8/Audio8-TTS-Preview-0.6b

RTF = autoregressive seconds / audio seconds. Lower is faster; RTF < 1 is faster than real time. Codec decode (a one-shot ~40 ms step) is reported separately and not counted in RTF — the AR loop is the throughput wall.

Results

Measured on an RTX 5090 Laptop GPU (sm_120, ~576 GB/s), torch 2.13.0 / cu130, transformers 4.57.5, bf16, batch 1, ~150 frames/run, 3 warmup + 3 measured:

Best: RTF 0.091 (~11x real time) with --mode maxat. Codec decode ~40 ms (one-shot, not in RTF). VRAM peak ~2.4 GB. One-time compile cost ~47 s (compile) / ~190 s (maxat), cached by inductor across runs.

Note the shape of the table: the static rewrite is not faster than stock on its own (eager is still launch-bound). Its only job is to make every tensor shape static so the whole frame can be compiled. Compiling the whole frame is where the win actually lands.

Install

You need a CUDA GPU and a recent PyTorch — torch.compile drives the fast path.

pip install -r requirements.txt

One footgun worth calling out (this cost us an afternoon): pin transformers==4.57.5 exactly, not as a floor. The model ships its modeling code via trust_remote_code, written against the transformers 4.57.x generate/cache API. A fresh pip/uv install will happily resolve transformers 5.x, which loads the weights fine but then breaks the custom arktts forward/generate path in confusing, far-from-the-error-site ways. torch, by contrast, wants to be new (2.13 / cu130 here) — recent inductor + sm_120 support is what makes the compiled frame fast. So: new torch, pinned transformers.

Quickstart

python example.py        # loads public weights, compiles the frame, writes example.wav

example.py is just this:

import soundfile as sf
from fast_arktts import FastTTS

tts = FastTTS()          # loads scrappylabsai/warble, compiles + warms the frame
audio, sr = tts.speak("<|speaker:2|>Thanks for calling. I can help you with that today.", seed=7)
sf.write("example.wav", audio, sr)

Zero-shot voice cloning (the stable path — give it a reference clip and its exact transcript):

audio, sr = tts.speak(
    "Read this in the cloned voice.",
    reference_audio="reference.wav",
    reference_text="The exact transcript of reference.wav.",
)

Lower level, if you want the generator directly:

from fast_arktts import load_model, build_prompt, FastArktts

model, processor = load_model()                    # scrappylabsai/warble
prompt, mask = build_prompt(model, processor, "<|speaker:2|>Hello.")
fast = FastArktts(model).compile("max-autotune")   # best steady-state
codes = fast.generate(prompt, mask)
wav, lens = model.decode_audio(codes)

Reproduce the table:

python benchmark.py                                # all modes
python benchmark.py --modes eager compile maxat --runs 5

How it works (the transferable part)

The stock generate loop is launch-bound, not compute-bound: on the order of ~1500 tiny kernel launches per audio frame, plus dozens of blocking host→device constant copies and a torch.cat-growing KV cache. On a fast card the GPU spends most of each frame waiting for the next kernel to be enqueued. That's the whole diagnosis, and it's the transferable one — before reaching for quantization or a smaller model, check whether a small autoregressive model is starving the GPU on launch overhead rather than saturating it.

Once you've decided the loop is launch-bound, the fix is to make the frame a single fixed-shape region you can hand to torch.compile / capture as one CUDA graph. Everything here is in service of that:

• Preallocated static buffers for the attention mask, output codes, the RAS sampling window, and the KV caches — written by index (index_copy_ / add_ on device counters), never reallocated, no torch.cat growth inside the loop.
• Persistent on-device KV caches, allocated once and reused across calls.
• Hoisted device-side constants (positions, temperature, top-p, the top-k mask) so the hot loop never does a host→device copy.
• Prompt / reference codes computed once and reused across generate calls.
• The whole frame is one function — semantic sample + 10-codebook fast-AR chain + slow-AR step + bookkeeping — under torch.compile(fullgraph=True, dynamic=False) (or a manual CUDA-graph capture). It fullgraph-compiled on the first try precisely because every shape is static.
• Lagged pinned-memory EOS check — the stop condition is read one frame late from a pinned host buffer, so the loop never calls .item() and never forces a per-frame host sync.
• Band-restricted semantic head — logits are computed over the 4097 semantic+eos columns instead of the full 155776-column vocab, and sampled in sorted space. This is bit-compatible with the stock semantic logits processor for the default (no custom processors) configuration.

The stock model modules are never modified. This code only reuses their weights through a static-shape reimplementation of the generate loop.

Why ~0.09 is near the floor on this card

Each frame streams ~2.0 GB of bf16 weights (a 24-layer slow stack + 10 sequential 4-layer fast-stack calls + heads). At ~4.2 ms/frame that's ~470 GB/s effective — about 82% of this laptop 5090's ~576 GB/s memory bandwidth. The frame is memory-bound, so there's little headroom left in the AR loop itself; the next real lever is weight-only quantization (fp8/int8), not more scheduling tricks.

What this is (and isn't)

This is reference single-stream, batch-1 inference — one voice, one stream, as fast as we could make it on one GPU. That's the whole scope. Deliberately not included here:

• Multi-stream / batched serving. The generator is written batch-general, but only B=1 is validated in this repo.

• Steady-state throughput only. This release measures sustained generation speed, not first-audio latency.

• Training / fine-tuning / voice-design recipes. Our voices at scrappylabsai/warble are a fine-tune of Audio8; the training side isn't in here.

A couple of honest caveats on output quality:

• Built-in-voice output is sampling-sensitive on the base model. A bare <|speaker:N|> no-reference line can degenerate on some random draws (the stock reference generate does too). Pin a seed for reproducibility, or use zero-shot cloning for stable output.
• Default model id is scrappylabsai/warble; pass any compatible arktts checkpoint to load_model / FastTTS.

Credits

All the hard modeling work is Audio8's — the arktts architecture, the codec, and the Audio8-TTS-Preview-0.6b weights this runs on, released under Apache-2.0. Thank you for shipping it openly; this repo would not exist otherwise. If you use this, please cite and credit them.

License

Apache-2.0, to match the underlying Audio8 model.

---

Built and open-sourced by ScrappyLabs — we build our own hardware, our own render, and our own voices, and we open the reference paths when they're the kind of thing we'd have wanted to find. If this saved you an afternoon, that's the point.