gepard-vLLM

GEnerative Prosody-aware Autoregressive text-to-speech model for Realtime Dialogues

A streaming text-to-speech model for realtime spoken conversation. GEPARD generates speech autoregressively with an LLM backbone — text and audio embeddings are trained together in a single model — and decodes it to a waveform with an FSQ-based NanoCodec, streaming audio chunk-by-chunk as text arrives. The name evokes Gepard (/ˈɡeːpart/), German for cheetah — a nod to the model's low-latency, high-throughput streaming.

It's served on a vLLM backend and exposed through a Cartesia-compatible API. Because it speaks Cartesia's wire protocol, it drops into Pipecat and LiveKit Agents by pointing their Cartesia plugin's base URL at this server — no custom plugin. It supports zero-shot voice cloning, and cloned voices are stored in Postgres so they survive restarts and are shared across replicas.

• Model: Qwen3.5 backbone + 32 FSQ audio heads + a binary stop head
• Codec: nvidia/nemo-nano-codec-22khz-1.89kbps-21.5fps (22.05 kHz mono, ~21.5 fps)
• API (Cartesia-compatible):
  • POST /tts/bytes — batch synthesis (full audio bytes)
  • WebSocket /tts/websocket — streaming synthesis (sentence-by-sentence, barge-in)
  • POST /voices/clone — clone a voice from a reference clip
  • GET /voices, GET|DELETE /voices/{id} — manage voices
• Voice cloning: create a voice once → reference its uuid at synthesis; the speaker prefix is cached in-process, so per-request cost is one short prefill.

The LLM and the codec both run inside vLLM's EngineCore subprocess sharing one CUDA context; the main FastAPI process does no GPU work.

---

Hardware & CUDA

• NVIDIA GPU, CUDA 13+, compute capability sm_120 — Hopper (H100) and Blackwell (B200 / RTX 50-series) class chips.
• At the default gpu_memory_utilization=0.82 the engine reserves ~80 GB of VRAM. On smaller cards lower it via TTS_GPU_MEMORY_UTILIZATION (see Configuration).

If the host doesn't have the CUDA toolkit installed (bare-metal / non-Docker runs), install CUDA 13:

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt-get update
sudo apt-get -y install cuda-toolkit-13-2

Then add to ~/.bashrc and source it:

export PATH=/usr/local/cuda/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH

The Docker image bundles CUDA, so for the container path you only need the NVIDIA driver + NVIDIA Container Toolkit on the host — not the toolkit above.

---

Voice store (Postgres)

Voices live in Postgres (Supabase, Neon, RDS, …). DATABASE_URL is required — the server fails fast on startup if it's unset. Create the table once (Supabase SQL editor or psql):

create table voices (
  id          uuid primary key default gen_random_uuid(),
  name        text not null,
  language    text,
  created_at  timestamptz not null default now(),
  prefix      bytea not null
);

Connection string notes:

• Use a session-mode connection (Supabase "Session pooler", or a direct connection on an IPv6-capable host) and include ?sslmode=require.
• The server connects as the table-owner role, so Row Level Security with no policies does not block it (the owner bypasses RLS). RLS only affects the PostgREST API roles, which this server doesn't use.

export DATABASE_URL='postgresql://USER:PASSWORD@HOST:5432/postgres?sslmode=require'

A voice is a ~32 KB speaker prefix; a read-through LRU cache keeps hot voices in memory, so the DB is hit at most once per voice per replica, never per synthesis.

---

Run locally

# One-time: venv + CUDA torch + vLLM + NeMo + transformers + asyncpg.
chmod +x ./setup.sh
HF_TOKEN=hf_xxx bash setup.sh        # HF token needs access to the gated model

export DATABASE_URL='postgresql://…?sslmode=require'
uv run python server.py              # uvicorn on :8000

Ready when the log shows Voice store connected (Postgres) then GEPARD pipeline ready! (first launch ~60 s for vLLM compile + codec restore + CUDA-graph capture; warm launches ~25 s).

Health check: curl -s http://localhost:8000/health

Stopping

All GPU memory is held by the EngineCore subprocess:

pkill -f "server.py"
pkill -f "VLLM::EngineCore"                                 # exits 1 if already dead — fine
nvidia-smi --query-gpu=memory.used --format=csv,noheader    # expect 0 MiB
pgrep -f "VLLM::EngineCore" | xargs -r kill -9              # if it refuses to exit

---

Run with Docker

The image bundles Python, CUDA libs, vLLM, NeMo and the app. The model itself is not baked in — it's pulled from Hugging Face on first start using HF_TOKEN (mount a volume for the HF cache so it's downloaded only once).

1. Build

Build on a CUDA-13 GPU host so uv pip install vllm --torch-backend=auto resolves the cu13 wheels. On a GPU-less CI box, edit the Dockerfile to pin an explicit backend (--torch-backend=cu130). The image is linux/amd64 only (CUDA); don't build it on Apple Silicon without --platform linux/amd64 + emulation (slow — prefer a real x86 GPU host).

Set your Docker Hub namespace once, then build with two tags — a version and latest — so deployments can pin an immutable tag while latest tracks newest:

export DOCKERHUB_USER=your_dockerhub_user
export VERSION=2.0.0                                   # bump per release

docker build \
  -t "$DOCKERHUB_USER/gepard-vllm:$VERSION" \
  -t "$DOCKERHUB_USER/gepard-vllm:latest" \
  .

The model is not baked into the image — it downloads at first run via HF_TOKEN — so no secrets are needed at build time and the image is safe to push to a public repo.

2. Push to Docker Hub

docker login                                            # Docker Hub user + access token
docker push "$DOCKERHUB_USER/gepard-vllm:$VERSION"
docker push "$DOCKERHUB_USER/gepard-vllm:latest"

Use a Docker Hub access token (Account Settings → Security), not your password — docker login accepts it as the password and it can be revoked. The repo must exist (or be auto-created on first push); make it private if you don't want the image public.

Verify the pushed tags:

docker pull "$DOCKERHUB_USER/gepard-vllm:$VERSION"         # from another host, confirms it's up

3. Run on a GPU machine

Requires the NVIDIA driver and the NVIDIA Container Toolkit on the host.

docker pull $DOCKERHUB_USER/gepard-vllm:latest

docker run --rm --gpus all -p 8000:8000 \
  -e HF_TOKEN='hf_xxx' \
  -e DATABASE_URL='postgresql://USER:PASSWORD@HOST:5432/postgres?sslmode=require' \
  -v hf-cache:/root/.cache/huggingface \
  $DOCKERHUB_USER/gepard-vllm:latest

• --gpus all exposes the GPU; the server needs ~80 GB VRAM at defaults (tune with -e TTS_GPU_MEMORY_UTILIZATION=…).
• -v hf-cache:/root/.cache/huggingface persists the model download across runs.
• First start downloads the model (minutes) then logs GEPARD pipeline ready!.

Or with Docker Compose

Put your secrets in a .env (see .env.example) and let docker-compose.yml wire up the GPU, port, and HF cache:

cp .env.example .env      # fill in DATABASE_URL + HF_TOKEN
docker compose up         # builds locally; set `image:` in the file to pull instead

The real .env is git/Docker-ignored, so secrets stay out of the image.

---

Configuration

Runtime env vars:

config.py is split into two sections: runtime tunables (every TTS_* var above — change in .env and restart, no rebuild) and locked constants (token IDs, head/FSQ layout, speaker-prefix dims, text-repetition layout) that are intrinsic to the checkpoint and must not be changed without retraining.

Sampling temperature is pinned to config.TEMPERATURE (vLLM samples head 0, the model samples heads 1–31 internally; they must match). Automatic prefix caching is intentionally disabled — the speaker prefix is injected as per-request embeddings, not token IDs, so caching KV by token IDs would serve the wrong voice on a repeated prompt.

---

API reference

POST /voices/clone — clone a voice (multipart/form-data)

curl -sS -X POST http://localhost:8000/voices/clone \
  -F clip=@ref.wav -F name="My Voice" -F language=en
# -> {"id":"<uuid>","name":"My Voice","language":"en","created_at":"…", ...}

Postgres assigns the id (a uuid). Use that uuid as voice.id at synthesis. 3–10 s reference clips work best.

POST /tts/bytes — batch synthesis

curl -sS -X POST http://localhost:8000/tts/bytes \
  -H 'Content-Type: application/json' \
  -d '{
        "transcript": "Hello from a cloned voice.",
        "voice": {"mode": "id", "id": "<uuid>"},
        "output_format": {"container": "wav", "encoding": "pcm_s16le", "sample_rate": 22050}
      }' \
  -o out.wav

• voice: {"mode":"id","id":"<uuid>"}. Omit it (or use "default") for the model's built-in voice. Unknown id → 404.
• output_format.container: raw | wav; encoding: pcm_s16le | pcm_f32le. sample_rate is resampled from the model's native 22050 Hz.

WebSocket /tts/websocket — streaming synthesis

Cartesia wire protocol. Client sends one transcript per sentence with continue:true, then continue:false to end the turn; server replies with {"type":"chunk","data":"<base64 pcm>"} chunks then {"type":"done"}. A {"cancel":true,"context_id":…} message interrupts that context (barge-in). Concurrent contexts are multiplexed on one socket.

Voice management

curl -s http://localhost:8000/voices                 # list
curl -s http://localhost:8000/voices/<uuid>          # metadata
curl -X DELETE http://localhost:8000/voices/<uuid>   # delete

GET /health

{"status":"healthy","tts_initialized":true,"voice_store":true}

---

Pipecat integration

pip install "pipecat-ai[cartesia]", then point the Cartesia service's base URL at this server and use the cloned voice's uuid:

# Streaming (WebSocket)
from pipecat.services.cartesia.tts import CartesiaTTSService
tts = CartesiaTTSService(
    api_key="ignored",
    url="ws://YOUR_HOST:8000/tts/websocket",
    voice_id="<uuid>",
    sample_rate=22050,
)

# Batch (HTTP)
from pipecat.services.cartesia.tts import CartesiaHTTPTTSService
tts = CartesiaHTTPTTSService(
    api_key="ignored",
    base_url="http://YOUR_HOST:8000",
    voice_id="<uuid>",
    sample_rate=22050,
)

LiveKit integration

pip install "livekit-plugins-cartesia", then:

from livekit.plugins import cartesia
tts = cartesia.TTS(
    api_key="ignored",
    base_url="http://YOUR_HOST:8000",   # http -> ws; the plugin appends /tts/websocket
    voice="<uuid>",
    sample_rate=22050,
)

---

Notes / behavior

• Sentence streaming. Both Pipecat and LiveKit aggregate the LLM stream to sentences before sending, and the server synthesizes each as it arrives — so the first sentence renders while later ones are still streaming in (low TTFB), rather than waiting for the whole turn.
• Barge-in. Pipecat sends Cartesia's {"cancel":true,"context_id":…} on interruption; the server cancels only that context's in-flight generation (freeing its slot) and leaves other contexts running. LiveKit handles barge-in client-side and tears down the stream.
• Prosody. Each sentence is an independent generation, so prosody resets at sentence boundaries (the standard streaming-TTS trade-off). Voice timbre is preserved across sentences (same speaker prefix).

---

Production hardening (TODO)

This server is configured to run behind an API gateway for testing. Before wider or untrusted exposure, address:

• Authentication. No endpoint checks credentials — /tts/*, POST /voices/clone, and DELETE /voices/{id} are all open, so anyone who can reach the port can burn GPU and clone/delete voices in the shared store. Add an API-key middleware (validate a header against an env secret) on every route except /health. The Cartesia clients already send api_key="ignored" — turn that into a required key.
• Gate /debug/arm_profile. It's registered unconditionally and runs the torch profiler / writes a trace to the worker's /tmp with no auth. Only mount it when a TTS_PROFILE-style env flag is set, or drop it from prod builds.
• Lock down CORS. allow_origins=["*"] combined with allow_credentials=True (server.py) is invalid + insecure — restrict to known origins (or drop credentials).
• Cap upload size. POST /voices/clone does await clip.read() with no bound, so a large body is an easy OOM. Enforce a max content-length.
• Pin vLLM. Both Dockerfile and setup.sh install vllm unpinned. The server depends on a custom ModelRegistry hook and internal worker patching, so an unpinned rebuild can silently pull a vLLM that breaks startup. Pin vllm==<known-good> (ideally lockfile the rest).

---

Troubleshooting

• Server exits at startup with "DATABASE_URL is required" — set it (and include ?sslmode=require).
• Cloned voice not applied — pass the uuid returned by /voices/clone as voice.id, not the name.
• Free memory on device … is less than desired — an orphaned EngineCore still holds the GPU; see Stopping.
• Container can't see the GPU — install the NVIDIA Container Toolkit and run with --gpus all.