Streamable TTS API

Streaming Text-to-Speech and Voice Conversion API with dynamic batching, CUDA Graphs, and torch.compile support.

Setup

1. Start vLLM backend

TTS_MODEL=Scicom-intl/Multilingual-Expressive-TTS-1.7B GPU_MEM_UTIL=0.7 \
docker compose -f vllm.yaml up --detach

Set TTS_API=http://tts-engine:9093 in your .env to point to the vLLM backend.

2. Configure environment

Copy .env_example to .env and adjust as needed. See app/env.py for all available variables:

3. Run the API

GPU:

docker compose up --build

CPU:

docker compose -f docker-compose-cpu.yaml up --build

Tracing (Loki + Tempo)

The app calls wan.patch() at startup, which always gives it JSON logs carrying the active trace id, a request log line per request, Prometheus metrics at /metrics, health probes and Scalar docs at /scalar. That library owns SERVICE_NAME, OTLP_ENDPOINT, TRACING_SAMPLE and the rest of the OTLP/log config.

ENABLE_TRACING_SPANS (on by default) adds this repo's own spans on the serving hot path (app/tracing.py), which answer where a request's time actually went:

POST /v1/audio/speech                 (fastapi instrumentation)
├── tts.normalize                     normalizer.mode=rule|llm, chars in/out
│   ├── normalize.llm                 the LLM normalizer call (mode="llm")
│   └── normalize.rule                the rule-based pipeline
└── tts.stream                        tts.ttfb_s, tts.lm_wait_s, tts.decode_wait_s,
    │                                 tts.decodes, tts.chunks, tts.audio_bytes
    ├── lm.generate                   the vLLM SSE stream, lm.deltas
    │   ├── lm.connect                POST → response headers
    │   └── lm.first_token            headers → first speech token (prefill)
    ├── tts.chunk (index=0)           one emitted audio chunk
    │   └── codec.decode              codec.tokens
    │       ├── codec.batch_wait      queued → picked up by the batch collector
    │       ├── codec.batch_prep      batch formed → H2D copy issued (batch thread)
    │       ├── codec.compute_wait    H2D issued → compute thread starts
    │       └── codec.gpu_decode      graph replay + D2H, batch.size, codec.cuda_graph
    └── tts.chunk (index=1) ...

So: dynamic batching = codec.batch_wait + codec.batch_prep + codec.compute_wait, waiting on vLLM = lm.connect + lm.first_token and the tts.lm_wait_s aggregate (how long the stitcher had no tokens left to decode), decoding speech tokens = codec.gpu_decode, or tts.decode_wait_s for the whole per-request wait. Voice conversion adds vc.load_audio, codec.encode, codec.encode_wait and codec.encode_gpu.

Two things make this safe to leave in the code path:

• Removable, not just cheap, and off unless someone is listening. Spans are built only when ENABLE_TRACING_SPANS is on and an exporter is configured and opentelemetry is importable. Fail any of the three and every helper is a shared contextlib.nullcontext() or a function returning None before doing anything — no tracer lookup, no time_ns(), no per-decode dict. The decode loop is GIL-bound, so tracing has to be genuinely absent when off.
• Explicit parents. A decode batch is built from N different requests, so the batching threads cannot use the ambient span context; each queued item carries its request's context plus the timestamp of the previous hop, and each stage is recorded after the fact with those timestamps.

Under real load, sample: TRACING_SAMPLE=0.05 keeps the trace volume (and the ~5 extra spans per decode) sane.

OTLP_ENDPOINT is part of the switch

ENABLE_TRACING_SPANS on its own does nothing: with no exporter configured the hot-path spans are not built at all, and the app logs why at startup rather than leaving you to wonder where the spans went:

hot-path spans requested but no span exporter is configured, so they are disabled rather
than built and dropped. Set OTLP_ENDPOINT (or ENABLE_CONSOLE_SPAN_EXPORTER=true) to
collect them, or TRACING_SPANS_REQUIRE_EXPORTER=false if a processor is installed in code.

The reason is that an OpenTelemetry SDK with no span processor attached still builds every span, stores its attributes, and then drops it. Measured over 3000 iterations of one request's worth of spans (24 spans with the real attribute sets, Apple M-series — indicative, not H100 numbers):

Any one of these opens the gate, so a console-exporter debug session or an auto-instrumented deployment is not silently un-traced: OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, JAEGER_HOST, ENABLE_CONSOLE_SPAN_EXPORTER=true. If you attach a processor in code instead, set TRACING_SPANS_REQUIRE_EXPORTER=false.

What you give up when no exporter is configured is the stage-level spanID on log lines: traceID and the request's own spanID still come from the FastAPI instrumentation (as does the X-Trace-Id response header), but with no hot-path spans every line of a request carries the same span id again.

One upstream default is turned off here. The OpenTelemetry ASGI instrumentation opens a span per ASGI message, and a streaming response polls the receive channel as it reads the LM stream — measured on an H20, one request produced 505 empty POST /v1/audio/speech http receive spans next to 19 real ones, which is both useless in the flame graph and a large multiple on Tempo's ingest. Two fixes, and the same shape of request now emits 25 spans and zero ASGI-message spans:

• DISCONNECT_POLL_S (0.25 s) bounds how often the LM reader asks whether the client disconnected. Each check is a real ASGI receive, and aiohttp yields two lines per SSE event, so the old per-line poll cost ~2 receives per speech token — event-loop time spent whether or not tracing is on. This alone took 505 noise spans to 9.
• _suppress_asgi_message_spans() in app/main.py removes the remainder, including the per-chunk http send spans that scale with utterance length. Set TRACE_ASGI_MESSAGE_SPANS=true to get the upstream behaviour back.

# a full Tempo + Loki + Alloy + Prometheus + Grafana stack to point it at
git clone https://github.com/Scicom-AI-Enterprise-Organization/wan
docker compose -f wan/grafana/docker-compose.yaml up -d \
  tempo loki alloy prometheus grafana

# then, in .env
ENABLE_TRACING_SPANS=true
OTLP_ENDPOINT=http://localhost:4327
SERVICE_NAME=tts-api

ENABLE_CONSOLE_SPAN_EXPORTER=true prints spans to stdout instead, which is enough to check the tree without a backend.

API Endpoints

GET /v1/audio/speaker

Returns the list of available speaker voices.

POST /v1/audio/normalize — Text Normalization

Normalizes text for TTS input. Strips markdown/HTML, then normalizes with one of two engines selected by mode:

• rule (default) — the built-in rule-based pipeline; optionally applies Malaysian text normalization (email, URL, phone, IC, money, time, units, etc.) when normalize_malaysian is set.
• llm — sends the text to an OpenAI-compatible LLM (OPENAI_BASE_URL / OPENAI_MODEL_NAME) with a multilingual few-shot prompt (app/prompt.py). The reply is constrained to {"normalized": "..."} via response_format json_schema (vLLM guided decoding), so the model can only return the normalized text. Returns 400 if OPENAI_* is not configured, 502 if the LLM call fails. normalize_malaysian is ignored in this mode.

Parameters:

Example:

curl -X POST 'http://localhost:9091/v1/audio/normalize' \
  -H 'Content-Type: application/json' \
  -d '{
    "input": "**Harga** rumah RM500,000. Hubungi 012-1234567 atau email husein.zol05@gmail.com",
    "normalize_malaysian": true
  }'

# LLM-based normalization
curl -X POST 'http://localhost:9091/v1/audio/normalize' \
  -H 'Content-Type: application/json' \
  -d '{
    "input": "baki saya tinggal RM1,250.50 dan IC saya 960314875079",
    "mode": "llm"
  }'
# {"output":"baki saya tinggal seribu dua ratus lima puluh ringgit lima puluh sen dan IC saya
#   sembilan enam kosong tiga satu empat lapan tujuh lima kosong tujuh sembilan.","mode":"llm"}

In llm mode the LLM is only called when the text contains something it could rewrite (a digit, a symbol, an ALL-CAPS or dotted token, a known abbreviation). Plain sentences such as "Hello there, how can I help you today?" come back unchanged from the LLM anyway, so they skip the ~0.55 s round trip and take the same pre/post cleanup path (LLM_NORMALIZER_SKIP_PLAIN, default on). On a TTS request that round trip sits entirely in front of the first audio byte.

mode: "spoken" is a rule-based replica of the LLM normalizer (app/spoken_normalizer/, pure Python, ~25 µs): numbers, money (RM/sen, dollars), IC and phone numbers digit by digit, dates, times, percentages, decimals, ordinals, units, emails, URLs, ids and the abbreviations the LLM expands, verbalized in the sentence's language, Malay/English by marker words, Mandarin and Tamil by script. It was built against the LLM's own outputs on bench/normalizer_corpus.py (bench/results/normalizer_truth.jsonl, 300 sentences) and agrees with them verbatim on 85% of sentences (English 91%, Malay 91%, Mandarin 90%, Tamil 72%, Malay/English code-switch 67%; most of the rest are LLM slips such as answering a Tamil sentence in English), never leaves a digit unread, and is what mode: "llm" falls back to. In code-switched sentences each number is read in the language of its neighbouring words. Coverage matrix and methodology: bench/NORMALIZER.md. bench/normalizer_agreement.py re-scores it; bench/normalizer_truth.py extends the ground truth.

curl -X POST 'http://localhost:9091/v1/audio/normalize' -H 'Content-Type: application/json' \
  -d '{"input": "Your total is RM1,250.50 and the meeting is at 3pm on 12/9/2026.", "mode": "spoken"}'
# {"output":"Your total is one thousand two hundred fifty ringgit fifty sen and the meeting is at
#   three p m on the twelfth of September twenty twenty six.","mode":"spoken"}

POST /v1/audio/speech — Text-to-Speech

Accepts JSON body.

Parameters:

Example:

curl -X POST 'http://localhost:9091/v1/audio/speech' \
  -H 'Content-Type: application/json' \
  -d '{
    "input": "Hello! How can I help you? can I get your passport number sir.",
    "voice": "husein",
    "model": "TTS-model",
    "response_format": "wav",
    "temperature": 0.7,
    "stream": true,
    "playback_speed": 0.5,
    "playback_overlap_speed": 0.1,
    "speaking_rate": 1.2,
    "normalize_malaysian": true
  }' \
  --output output.wav

Speaking rate

speaking_rate (alias speed) changes how fast the voice talks without changing its pitch. The LM has no rate control and speech tokens are a fixed 50 Hz, so this is done after NeuCodec decode, on the PCM stream, with WSOLA (waveform-similarity overlap-add — the SoundTouch-style tempo change, app/timestretch.py): the audio is copied out in ~40 ms blocks whose input hop is rate× the output hop, so whole pitch periods are dropped (faster) or repeated (slower) while the samples inside each block are untouched; each block's start is searched ±7.5 ms for the best waveform match to the previous block's tail and the two are crossfaded over 8 ms. Plain resampling would shorten every period too (chipmunk effect). It is a stage on the stitched stream (after crossfade + loudness normalization, before the pcm/wav/SSE layers), stateful across chunks with ~55 ms of lookahead — so it works for streaming, adds no measurable latency to the 2 s first chunk, and leaves the token/decode/batching pipeline untouched. 1.0 bypasses it entirely. Range 0.5–2.0 (422 outside): beyond that WSOLA on speech starts to buzz / drop consonants.

curl -X POST 'http://localhost:9091/v1/audio/speech' -H 'Content-Type: application/json' \
  -d '{"input": "Hello there, how can I help you?", "voice": "husein", "speaking_rate": 1.3,
       "response_format": "wav", "stream": false}' --output fast.wav

Interleaved generation

An agent that streams (LiveKit's StreamAdapter, for one) does not send a reply as one request: it cuts the text into sentence-sized chunks and synthesizes each with its own /v1/audio/speech call. The request at T+1 knows nothing about T, so the LM starts cold — pitch register, pace and energy reset at every join and the reply sounds stitched.

Pass the same interleave_id on the chunks of one reply and each request is prompted with the previous turns' text and the speech tokens the LM generated for them, in the interleaved document format the model was trained on:

<|im_start|>husein: hello my name is husein,<|speech_start|><|s_1834|>…<|s_77|><|im_end|>
<|im_start|>husein: i like to eat chicken rice.<|speech_start|>      ← the LM continues here

The history is prefill only — nothing extra is decoded or streamed, and the codec path (the bottleneck) is untouched. The last 5 turns are retained per id (MAX_RETAIN_INTERLEAVE, per-request max_retain_interleave), also bounded by INTERLEAVE_MAX_S seconds of speech and by the LM window. Turns live in /dev/shm, shared by every uvicorn worker on the host, and expire after INTERLEAVE_TTL_S idle seconds. A voice switch on the same id starts cold. Design and configuration: INTERLEAVE.md.

This requires an LM trained on interleaved documents, and no open-source TTS model is. The packing (pack_stage1.py --interleave_style full) is an in-house recipe and the checkpoints that have it are private model repos — so the benefit comes from the model, not from the prompt shape. Measured on a private interleave-trained checkpoint, it cuts the jump in pitch register and loudness between consecutive chunks by ~25% at no latency cost (bench/INTERLEAVE_AB.md); on a model packed without it the same prompt bought nothing and made the LM cut 7.5% of chunks short. Serving any other model — an open-source TTS LM, or an older in-house one — set INTERLEAVE_STORE=off, and check what your vLLM actually loads before relying on the feature.

curl -s -D - -X POST localhost:9091/v1/audio/speech -H 'Content-Type: application/json' \
  -d '{"input":"hello my name is husein,","voice":"husein","interleave_id":"room-42",
       "response_format":"wav","stream":false}' -o c1.wav
curl -s -D - -X POST localhost:9091/v1/audio/speech -H 'Content-Type: application/json' \
  -d '{"input":"i like to eat chicken rice.","voice":"husein","interleave_id":"room-42",
       "response_format":"wav","stream":false}' -o c2.wav
#   X-Interleave-Turns: 1        ← chunk 1 was in the prompt
#   X-Interleave-Tokens: 137     ← its speech tokens (2.7 s)

curl -s localhost:9091/v1/audio/interleave/room-42            # retained turns
curl -s -X DELETE localhost:9091/v1/audio/interleave/room-42  # forget now (else TTL)

For LiveKit, the openai.TTS plugin cannot add body fields but accepts a pre-built client, so pass the id as a default header — one id per room:

tts_client = openai_sdk.AsyncClient(api_key="unused", base_url=TTS_BASE_URL,
                                    default_headers={"X-Interleave-Id": ctx.room.name})
session = AgentSession(tts=openai.TTS(model=..., voice=..., client=tts_client,
                                      response_format="pcm"))

GET / DELETE /v1/audio/interleave/{id}

Inspect (GET: retained turns, their text and token counts, the caps and TTL) or forget (DELETE) one interleave id. Both return 400 when INTERLEAVE_STORE=off.

POST /v1/audio/vc — Voice Conversion

Accepts multipart form data. Clones the reference voice and generates speech for new text.

Parameters:

Example:

curl -X POST 'http://localhost:9091/v1/audio/vc' \
  -H 'Content-Type: multipart/form-data' \
  --form 'reference_audio=@jenny.wav' \
  --form 'reference_text=I wonder if I shall ever be happy enough to have real lace on my clothes and bows on my caps.' \
  --form 'generate_text=Ye encik, apa yang saya boleh tolong?' \
  --form 'model=TTS-model' \
  --form 'response_format=wav' \
  --form 'temperature=0.7' \
  --form 'repetition_penalty=1.15' \
  --form 'max_tokens=3072' \
  --form 'stream=true' \
  --form 'playback_speed=1.5' \
  --form 'playback_overlap_speed=0.2' \
  --form 'speaking_rate=1.0' \
  --output vc.wav

Gradio Demo

gradio_app.py is a simple UI for poking at a running API instance (local or remote) without curl — tabs for text-to-speech, text normalization, and voice conversion.

pip install gradio requests
TTS_API_BASE_URL=http://localhost:9091 python gradio_app.py   # defaults to localhost:9091

Open the printed local URL, set the API base URL if needed, and click Check connection to list available speakers.

Unit Tests

Running tests

pip install pytest requests

# or with uv (no install needed), e.g. the LLM normalizer tests:
uv run --with aiohttp --with pytest -- pytest tests/test_llm_normalizer.py -v
# include the live LLM tests (hits OPENAI_BASE_URL):
set -a; source .env; set +a
uv run --with aiohttp --with pytest -- pytest tests/test_llm_normalizer.py -v

# run all tests (unit + integration against live API)
python -m pytest tests/ -v

# run only markdown sanitization tests (no dependencies beyond app/rules.py)
python -m pytest tests/test_sanitize_markdown.py -v

# run the tracing helper tests (no GPU; add opentelemetry-sdk for the enabled-path tests)
uv run --with pytest --with opentelemetry-sdk -- pytest tests/test_tracing.py -v

# run normalizer tests (requires app/normalizer dependencies: dateparser, unidecode, numpy)
python -m pytest tests/test_normalizer.py -v

# run Malaysian rules stress tests
python -m pytest tests/test_malaysian_rules.py -v

# run multilingual tests
python -m pytest tests/test_multilingual.py -v

# run TTS & VC integration tests against a live API (default: http://localhost:9091)
python -m pytest tests/test_tts_vc_api.py -v

# run TTS & VC tests against a different URL
TTS_TEST_URL=http://localhost:8080 python -m pytest tests/test_tts_vc_api.py -v

# run normalize API integration tests (requires in-process app import)
python -m pytest tests/test_normalize_api.py -v

Test files

623 passed, 35 skipped on a full run with a live API and OPENAI_* configured.

Benchmark — H100 SXM vs H200 SXM

End-to-end throughput/latency of the optimized stack on a single GPU, measured on RunPod secure cloud (June 2026). Both pods run the same colocated layout — vLLM 0.16.0 serving the Qwen3-1.7B TTS LM plus the vendored NeuCodec decoder on one GPU — with the optimized configuration: dynamic batching + CUDA graphs + 4 uvicorn workers under NVIDIA MPS, bf16 throughout (no quantization), torch 2.9.1 / CUDA 12.8. vLLM --gpu-memory-utilization 0.4 --max-num-seqs 64. Driver 580.126.09 (H100) / 550.127.05 (H200).

Load generator: bench/bench.py, non-streaming, concurrency 1/4/16/50, per-conc-mult 5, temperature 0.6, max_tokens 1024, over the fixed 16-sentence eval set (~4.5–4.9 s audio/request; English, Malay, code-switch). Median of 5 runs per level; 0 request errors at every level. Throughput is the meaningful TTS metric: audio-seconds produced per wall-second (values > the eval clip length mean the server emits audio faster than real time in aggregate).

Throughput (audio-seconds produced per wall-second)

Latency & real-time factor (RTF)

Takeaways

• H200 wins at every concurrency, with the largest margins under load — 1.86× at C=16 and 1.41× at C=50 — where the bandwidth-bound NeuCodec decode dominates. The H200's higher HBM bandwidth (≈4.8 TB/s vs ≈3.35 TB/s) and 141 GB capacity give the colocated vLLM + 4 codec workers more headroom.
• Both run faster than real time at all concurrencies (RTF p50 < 1). Single-request latency is ~0.6 s (H100) / ~0.5 s (H200) for ~4.5 s of audio (RTF ≈ 0.15 / 0.12).
• H200 is also far more stable. Across the 5 runs the H200 throughput was tight (C=16: 80.2–83.5; C=50: 141–145), while the H100 showed large run-to-run swings at high concurrency (C=16: 32–60; C=50: 95–116) — transient MPS/codec contention stalls that the H200's extra bandwidth and memory absorb. The table reports medians; raw per-run JSON is in bench/results/runpod_h100_h200.json.

The full optimization writeup (baseline → CUDA graphs → multi-worker + MPS, and a documented negative result on source-level micro-optimizations) is in bench/OPTIMIZATION.md. To reproduce, see the ready scripts in bench/deploy/ (setup_pod.sh, start_vllm.sh, start_app.sh).

LiveKit agent stress test & loudness consistency

bench/livekit/ drives the API through a real LiveKit agent (text in → agent session.say() → TTS → WebRTC audio out) at concurrency 1/4/8: 0 errors, TTFB p50 ≈ 1.2 s with the LLM normalizer in the path. It also quantified utterance-to-utterance loudness variance — the LM's sampled speech tokens carry loudness, so identical text at temperature 0.6–0.7 spans 3–10 dB active-RMS (and different voices sit ~5 dB apart in natural level), and roughly half of hot utterances clip at full scale. STREAM_NORMALIZE=true (default; per-request stream_normalize) collapses the spread to < 2 dB with no added latency: one static gain per utterance (locked after the first ~1 s of voiced audio — no mid-utterance drift), boost capped by running-peak headroom, and a tanh soft-knee limiter instead of a hard clip. Details and before/after numbers in bench/livekit/README.md.

Streaming decode quality itself is near one-shot: the stitcher decodes growing windows (first = playback_speed×50 tokens, ×STREAM_CHUNK_GROWTH per step up to STREAM_MAX_CHUNK_S) with STREAM_PAST_CONTEXT_S of already-generated past tokens included in every window and sliced off after decode (free, unlike future context). Measured streamed-vs-one-shot envelope gap on identical tokens: 0.03–0.04 dB median (worst 0.3–0.5 dB, first window only) vs 0.25–0.55 dB median with the old fixed 1.5 s windows. For offline generation (stream: false), "playback_speed": 10 decodes the whole utterance in one window.

Huawei Ascend 910B3 NPU — support & the precision quality gap

The full stack runs on Huawei Ascend 910B3 (CANN 8.5.2): the vLLM LM via vllm-ascend on NPU 0, and the NeuCodec decoder via torch_npu on NPU 1. The app is unchanged apart from device selection — set DEVICE=npu and it uses a torch.npu stream shim, skipping CUDA graphs/streams automatically (see app/main.py). Pin each stage to a chip with ASCEND_RT_VISIBLE_DEVICES (0 for vLLM, 1 for the codec).

Working install (Python 3.11 — vLLM 0.11 uses py3.10 syntax that breaks on 3.9):

# codec app venv (py3.9 ok): torch 2.8.0 + torch_npu 2.8.0.post5  → see requirements-npu.txt
# vLLM venv (py3.11): install vllm first, then vllm-ascend (it pins torch back to 2.7.1)
uv venv /root/vllm-venv311 --python 3.11 && source /root/vllm-venv311/bin/activate
uv pip install vllm==0.11.0
uv pip install vllm-ascend==0.11.0 "setuptools<81"   # torch 2.7.1 + torch-npu 2.7.1; <81 keeps pkg_resources
source /usr/local/Ascend/ascend-toolkit/set_env.sh && source /usr/local/Ascend/nnal/atb/set_env.sh

The NeuCodec decode runs cleanly on the NPU (bit-identical to CUDA — the vendored RoPE in app/neucodec/_rope.py removed the torchtune/torchao dependency, which is what made py3.9 + NPU viable). Encode currently must run on CPU (the encoder's alias-free resample mis-shapes on NPU); this only affects VC/token-extraction, not TTS decode.

910B3 vs H100 — measured (single-stream, 16-sentence eval set, bf16, temp 0.6)

⚠️ Open issue — the bf16 quality gap. With identical decode and identical sampling settings, the Ascend LM produces measurably less-natural speech tokens (−0.6 MOS) than the H100, despite comparable intelligibility. The gap is in the vllm-ascend LM path, i.e. the 910B3's bf16 compute kernels — not the codec. Mitigations tested and their MOS:

Config	MOS	Result
Ascend bf16 + ACL graphs (default)	2.60	baseline
Ascend bf16 + --enforce-eager	2.45	no help — ACL graph capture is not the cause
Ascend --dtype float16 + eager	1.73	much worse — keep bf16
H100 bf16	3.21	target

Neither disabling graph capture nor switching precision closes it, so for this TTS model the 910B3 is both slower and less natural than H100 at bf16. MOS measured with faster-UTMOSv2.

Root cause — isolated (not fixable via config): the gap is entirely in the LM tokens the vllm-ascend bf16 forward produces, not the codec:

• Codec decode is bit-identical on NPU vs CPU — decoding the same tokens on both gives MAE = 0.00000 on all 16 clips. Rules out the decoder.
• Sampling is already fp32 (AscendSampler softmax dtype=float32) and matmul HF32 is off (torch.npu.matmul.allow_hf32 == False) — both full precision. Rules out sampler + matmul.
• fp32 is impossible on this stack: --dtype float32 crashes the engine — the Ascend paged-KV attention op (ReshapeCacheOperation, ERR00100) only supports bf16/fp16. So the fused attention kernel is locked to bf16, and its internal accumulation is what diverges from the H100's bf16 attention, shifting sampled tokens toward less-natural speech.

There is no serving-flag fix today. Real fixes require a higher-precision (fp32-accumulate) attention kernel from Huawei/vllm-ascend, a different attention backend, or fine-tuning the model to the 910B3 bf16 numerics. Tracked for upstream.