Qwen3-TTS Streaming — Arabic Edition

Streaming inference for Qwen3-TTS with Arabic language support, an OpenAI-compatible server, and Docker deployment for NVIDIA DGX Spark.

The official team mentions "Extreme Low-Latency Streaming Generation" in their paper but the streaming code was never released. This fork adds real streaming, Arabic fine-tuning, and a production-ready server on top of dffdeeq/Qwen3-TTS-streaming.

What's added over upstream:

• Real-time PCM streaming via stream_generate_pcm
• ~6x inference speedup vs upstream qwen-tts
• OpenAI-compatible /v1/audio/speech SSE server
• Arabic language fine-tuning and inference
• Docker support for NVIDIA DGX Spark (ARM64 / Grace Blackwell)

Based on dffdeeq/Qwen3-TTS-streaming and QwenLM/Qwen3-TTS.

---

Arabic Support

The base Qwen3-TTS-12Hz model supports a fixed set of languages defined in the model config. Arabic is not included by default. This fork adds Arabic as a first-class language through targeted changes to both the fine-tuning pipeline and the inference server.

How it works

1. Language embedding registration

The model's talker component uses a codec_embedding lookup table to condition generation on the input language. Each language maps to a token ID in config.talker_config.codec_language_id.

During fine-tuning, Arabic is registered at token ID 2072 with a warm start — its embedding is initialised as the mean of all existing language embeddings rather than random noise:

ARABIC_LANG_ID = 2072
config.talker_config.codec_language_id['arabic'] = ARABIC_LANG_ID

codec_emb = model.talker.model.codec_embedding
existing_ids = [v for k, v in config.talker_config.codec_language_id.items() if k != 'arabic']
avg = codec_emb.weight[existing_ids].float().mean(0)
codec_emb.weight[ARABIC_LANG_ID] = avg

This gives Arabic a sensible starting point and significantly speeds up convergence.

2. Language-conditioned codec prefix (4-token think block)

The upstream fine-tuning code uses a 3-token codec prefix (no-think, bos, eos). To carry language information into every generation step, this fork replaces it with a 4-token language-conditioned think block:

pos 3: codec_think_id
pos 4: codec_think_bos_id
pos 5: lang_id          ← Arabic token 2072 for Arabic samples
pos 6: codec_think_eos_id
pos 7: speaker embedding (shifted from pos 6 in upstream)

This means every forward pass explicitly encodes which language is being generated, allowing the model to switch cleanly between Arabic and other languages within the same checkpoint.

The collate_fn in dataset.py builds this sequence automatically per sample. The max_length buffer is +9 instead of the upstream +8 to accommodate the extra token.

3. Automatic language detection

If a training sample or inference request does not specify a language, it is auto-detected by scanning for Arabic Unicode characters (U+0600–U+06FF):

def _detect_language(self, text: str) -> str:
    for c in text:
        if '\u0600' <= c <= '\u06FF':
            return 'arabic'
    return 'english'

This means mixed datasets work without adding "language" fields to every JSONL entry.

4. Checkpoint self-containment

When a checkpoint is saved, the Arabic language ID is written back into config.json under codec_language_id so the saved model is fully self-contained and loads correctly without any external configuration:

"codec_language_id": {
  "chinese": ..., "english": ..., ...,
  "arabic": 2072
}

5. Inference server passthrough

The OpenAI-compatible server accepts a language field in the request body and passes it directly to the model's generation pipeline. For fine-tuned CustomVoice models, Arabic text is routed through the correct language embedding automatically:

curl http://localhost:8000/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
    "model": "tts-1",
    "input": "مرحبًا، كيف حالك؟",
    "voice": "my_arabic_speaker",
    "language": "arabic",
    "stream": true
  }'

---

Benchmark (RTX 5090)

Non-streaming (full inference)

Streaming

---

Installation (Python 3.12)

1. Install SOX

Linux:

sudo apt install sox libsox-fmt-all

Windows: Download from https://sourceforge.net/projects/sox/ and add to PATH.

2. Create environment

conda create -n qwen3-tts python=3.12 -y
conda activate qwen3-tts

3. Install dependencies

Linux:

pip install torch==2.9.1 torchaudio==2.9.1 --index-url https://download.pytorch.org/whl/cu130
pip install https://github.com/mjun0812/flash-attention-prebuild-wheels/releases/download/v0.6.8/flash_attn-2.8.3%2Bcu130torch2.9-cp312-cp312-linux_x86_64.whl

Windows:

pip install torch torchaudio --index-url https://download.pytorch.org/whl/cu130
pip install https://github.com/mjun0812/flash-attention-prebuild-wheels/releases/download/v0.7.12/flash_attn-2.8.3%2Bcu130torch2.10-cp312-cp312-win_amd64.whl
pip install -U "triton-windows<3.7"

4. Install package

git clone https://github.com/VadzimBelski-ScienceSoft/Qwen3-TTS-streaming-arabic.git
cd Qwen3-TTS-streaming-arabic
pip install -e .

---

Running Natively

Linux (x86-64 / CUDA)

# 1. Install SOX
sudo apt install sox libsox-fmt-all

# 2. Create environment
conda create -n qwen3-tts python=3.12 -y
conda activate qwen3-tts

# 3. Install PyTorch with CUDA 13.0 support
pip install torch==2.9.1 torchaudio==2.9.1 --index-url https://download.pytorch.org/whl/cu130

# Optional: flash-attention for faster inference
pip install https://github.com/mjun0812/flash-attention-prebuild-wheels/releases/download/v0.6.8/flash_attn-2.8.3%2Bcu130torch2.9-cp312-cp312-linux_x86_64.whl

# 4. Install the package
pip install -e ".[server]"

# 5. Start the server
python server.py --model Qwen/Qwen3-TTS-12Hz-1.7B-Base --device cuda:0 --port 8000

# Or with a fine-tuned checkpoint
python server.py --model ./output/checkpoint-epoch-9 --device cuda:0 --port 8000

macOS — Apple Silicon (M1/M2/M3/M4, MPS)

PyTorch supports Metal (MPS) on Apple Silicon for GPU-accelerated inference without CUDA.

# 1. Install SOX
brew install sox

# 2. Create environment
conda create -n qwen3-tts python=3.12 -y
conda activate qwen3-tts

# 3. Install PyTorch (the default PyPI build includes MPS support)
pip install torch torchaudio

# 4. Install the package
pip install -e ".[server]"

# 5. Start the server using the MPS backend
python server.py --model Qwen/Qwen3-TTS-12Hz-1.7B-Base --device mps --dtype float32 --attn-impl sdpa --port 8000

Note: --dtype float32 is required on MPS — bfloat16 is not fully supported. --attn-impl sdpa is the correct attention backend for MPS (flash-attn is CUDA-only).

macOS — Intel (CPU only)

brew install sox
conda create -n qwen3-tts python=3.12 -y && conda activate qwen3-tts
pip install torch torchaudio
pip install -e ".[server]"
python server.py --model Qwen/Qwen3-TTS-12Hz-1.7B-Base --device cpu --dtype float32 --attn-impl sdpa --port 8000

---

Streaming Parameters

See examples/ for usage:

• test_streaming_optimized.py
• test_optimized_no_streaming.py

---

OpenAI-Compatible Server

The server exposes a /v1/audio/speech endpoint compatible with the OpenAI TTS API, streaming PCM audio as Server-Sent Events.

Start the server

pip install -e ".[server]"

# Fine-tuned Arabic model
python server.py --model ./output/checkpoint-epoch-9 --host 0.0.0.0 --port 8000

# Or load from HuggingFace Hub
python server.py --model youruser/qwen3-arabic-tts --host 0.0.0.0 --port 8000

Arabic request

curl http://localhost:8000/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
    "model": "tts-1",
    "input": "مرحبًا، كيف حالك؟",
    "voice": "my_arabic_speaker",
    "language": "arabic",
    "stream": true
  }'

Request fields

Server arguments

Supported endpoints

Model types

---

Docker (DGX Spark / ARM64)

The Dockerfile uses the NVIDIA NGC PyTorch base image (nvcr.io/nvidia/pytorch:25.10-py3), which ships with CUDA 13, cuDNN, and PyTorch pre-installed for Grace Blackwell ARM64.

Build

./build.sh

Custom base image:

BASE_IMAGE=nvcr.io/nvidia/pytorch:24.08-py3 ./build.sh

Run — HuggingFace Hub model

MODEL=youruser/qwen3-arabic-tts ./run.sh

# Private model
HF_TOKEN=hf_xxx MODEL=youruser/private-model ./run.sh

The model is downloaded on first start and cached in a named Docker volume (qwen3-tts-hf-cache). Restarts reuse the cache.

Run — local checkpoint

MODEL=/path/to/checkpoint-epoch-9 ./run.sh

run.sh detects the local path, auto-mounts it to /model, and sets MODEL=/model.

Docker Compose

cat > .env <<EOF
MODEL=youruser/qwen3-arabic-tts
HF_TOKEN=hf_xxx
PORT=8000
ATTN_IMPL=sdpa
EOF

docker compose up -d

For a local checkpoint, uncomment the volume line in docker-compose.yml and set MODEL=/model.

---

Fine-tuning

See finetuning/README.md for the complete guide, including:

• JSONL format with optional language field
• Arabic warm-start embedding initialisation
• --resume_checkpoint for continuing training
• --use_wandb for experiment tracking

---

Why This Exists

From the official Qwen3-TTS README:

Now only offline inference is supported. Online serving will be supported later.

This fork adds streaming, Arabic, and a production server now.