Serverless TTS benchmark console

This project sends one text prompt to the same Audio8 TTS model on Modal and Beam, returns WAV audio from each provider, and displays comparable latency instrumentation. It also includes an independent Beam runner for ResembleAI/chatterbox.

The model is configured as Audio8/Audio8-TTS-Preview-0.6b. It is a 0.6B multilingual TTS checkpoint with a bundled 44.1 kHz codec. The model card recommends Python 3.10+ and a CUDA-capable GPU, and documents native Transformers inference with trust_remote_code=True.

What is included

• Next.js comparison console at /.
• Server-only /api/synthesize route that calls Modal and Beam concurrently.
• Server-only /api/chatterbox route for the Chatterbox Beam endpoint.
• Shared Audio8 runtime used by both deployment entrypoints.
• Dedicated Chatterbox runtime with the same timing and WAV instrumentation.
• Modal class deployment with @modal.enter, model warmup, HF Volume caching, CPU/GPU memory snapshot support, and configurable warm containers.
• Beam Audio8 and Chatterbox endpoints with on_start, persistent Hugging Face caches, checkpoint restore, and keep-warm behavior.
• Modal Audio8 defaults to A100. Both Beam models default to RTX 4090 because Beam checkpoint restore supports RTX 4090 and it is currently available to this workspace.
• Beam uses a CUDA 12.8 / PyTorch 2.7 image for the RTX 4090 deployments.
• Per-provider metrics:
  • cold-start flag;
  • container uptime before the request;
  • container start to audio-ready time;
  • model load or checkpoint-restore initialization time;
  • generation and decode time;
  • WAV encoding time;
  • server total;
  • app round trip;
  • approximate network overhead;
  • audio duration, sample rate, and payload size.

Run the console locally

npm install
cp .env.example .env.local
npm run dev

Open http://localhost:3000. Until provider URLs and tokens are configured, the UI intentionally shows each provider as unavailable instead of silently returning fake audio.

Configure provider credentials

Set these values in .env.local or the deployment platform's secret store:

AUDIO8_MODEL_ID=Audio8/Audio8-TTS-Preview-0.6b
CHATTERBOX_MODEL_ID=ResembleAI/chatterbox
MODAL_GPU=A100
BEAM_GPU=RTX4090
CHATTERBOX_BEAM_GPU=RTX4090
BEAM_CHECKPOINT_ENABLED=1
CHATTERBOX_BEAM_CHECKPOINT_ENABLED=1
BEAM_MIN_CONTAINERS=0
BEAM_MAX_CONTAINERS=3
BEAM_TASKS_PER_CONTAINER=1
CHATTERBOX_BEAM_MIN_CONTAINERS=0
CHATTERBOX_BEAM_MAX_CONTAINERS=3
CHATTERBOX_BEAM_TASKS_PER_CONTAINER=1
MODAL_ENDPOINT_URL=https://...
MODAL_PROXY_TOKEN=wk-<id>.ws-<secret>
BEAM_ENDPOINT_URL=https://...
CHATTERBOX_BEAM_ENDPOINT_URL=https://...
BEAM_TOKEN=...

The browser only calls /api/synthesize; none of these credentials are exposed to client JavaScript.

Deploy Modal

Install and authenticate with the Modal CLI, then deploy from the repository root:

python3 -m pip install modal
modal setup
modal deploy providers/modal/app.py

The endpoint URL printed by Modal becomes MODAL_ENDPOINT_URL. The runtime expects the combined Modal proxy bearer token format wk-<id>.ws-<secret> as MODAL_PROXY_TOKEN.

Useful optional environment variables when deploying:

MODAL_GPU=A100
MODAL_ENABLE_GPU_SNAPSHOT=1
MODAL_MIN_CONTAINERS=0
MODAL_MAX_CONTAINERS=3
MODAL_SCALEDOWN_WINDOW_SECONDS=300

If GPU memory snapshots are incompatible with the final model image or GPU, redeploy with MODAL_ENABLE_GPU_SNAPSHOT=0; CPU memory snapshots and warm-container controls remain available.

Deploy Beam

Install and authenticate with Beam, then deploy from the repository root:

python3 -m pip install beam-client
beam configure default --token "$BEAM_TOKEN"
beam deploy providers/beam/app.py:predict
beam deploy providers/beam/chatterbox_app.py:predict

The Audio8 endpoint URL becomes BEAM_ENDPOINT_URL; the Chatterbox endpoint URL becomes CHATTERBOX_BEAM_ENDPOINT_URL. Both endpoints are private and the app sends the same Beam bearer token server-side.

Beam checkpoint restore is enabled for both RTX 4090 endpoints. Beam captures the container after on_start has loaded and warmed the model, then restores that state for later scale-from-zero boots. The initial capture can take up to three minutes and distribution can take up to five minutes; redeploy after changing the GPU or model image. Set the two *_MIN_CONTAINERS values to 1 when the goal is near-zero latency at the cost of continuously billed idle GPUs.

If Beam reports that no RTX 4090 capacity is available, the deployment may still be created but requests will fail before a container starts. Reserve an RTX 4090 pool with beam machine reserve --gpu RTX4090 --ttl 2h --yes only when the spend is acceptable, or use another supported GPU with available capacity.

Chatterbox installs its runtime dependencies without the package’s older pinned PyTorch 2.6 build. The endpoint instead uses PyTorch 2.7.1 with CUDA 12.8, which supports the RTX 4090 deployment.

Metric definitions

The provider containers return timings from a monotonic clock. coldStart is true for the first user request handled by a newly initialized container. containerStartToAudioMs measures from the provider process start to WAV bytes being ready. modelLoadMs measures model initialization during container startup; on snapshot/checkpoint restores it represents the initialization cost captured for the restored deployment. gatewayRoundTripMs includes the network path between the Next.js server and the provider. networkOverheadMs is an approximation: gateway round trip minus provider-reported server time.

The first deployment should be benchmarked with at least two runs per provider: one after scale-to-zero and one while the container is warm. Provider scheduling, GPU availability, and network distance can change these measurements.