Qwen3-ASR-0.6B Streaming (causal audio tower)

A streaming/causal drop-in audio tower for Qwen/Qwen3-ASR-0.6B: the pretrained (offline, fully bidirectional) audio encoder, fine-tuned to run append-only: each ~2 s audio block is encoded exactly once with a causal KV cache and a bounded 15 s attention window, and is never re-encoded. New audio costs one encoder pass over the new block plus an incremental decoder update; per-chunk compute is constant in stream length, memory is bounded, and streams can run indefinitely.

It is built for the minimum-compute-per-chunk regime: many concurrent streams, energy-constrained or on-device serving, and sessions of unbounded length. Per audio second it spends about 3x less than the re-compute streaming backend, at constant cost as the stream grows, in exchange for some accuracy (see the results table for the honest comparison).

This repository contains only the fine-tuned audio tower (746 MB fp32 safetensors). The decoder, adapter and feature extractor are loaded unchanged from the base model at runtime.

Results

Long-form: 21 full MCIF/ACL conference talks (5 to 7 minutes each, 2 h total, accented scientific English), human references, Whisper text normalization:

Short-form through the same streaming stack: LibriSpeech test-clean 3.73 / test-other 7.34 at the shipped low-latency default.

Streaming WER contract

The WER numbers above use the same real-streaming replay: no right to rewrite the past, a 250 ms live-tail cut at every non-final update, and an end-of-stream flush. When forced-aligner word_alignments are present, the 250 ms cut uses those word timestamps; otherwise it falls back to a uniform text approximation.

Commit latency (measured)

Per-committed-word latency (word spoken to word committed, audio fed at 1.0x), replayed over the 21-talk corpus:

The 2.2 s point sits at the 1.92 s block-cadence floor. The unstable tail is additionally visible live (~2 s behind the audio head).

How it works

• Block-bidirectional causal execution: full attention within each fixed block (192 mel frames = 1.92 s; a 96-frame variant is also in-regime), causal per-layer KV across blocks, sliding 15 s left window, sinusoidal positions continuing monotonically. Latency = one block.
• Trained by self-distillation: the causal student matches the frozen offline tower's output embeddings (MSE + 0.5·cosine), audio only, no labels, with mixed block sizes (96/192) and position-offset augmentation (log-uniform up to 6000 steps ≈ 2 h) so positions extrapolate far beyond the 120 s table. Base phase: LibriSpeech 960 h (~6 H100-hours). This release adds a diverse-audio phase (People's Speech + LibriSpeech replay, ~1k h, 20k steps) that took long-form WER from 18.1 to 17.2: spontaneous/accented audio is the measured quality lever.
• Segmented long-form serving: segments roll at sentence punctuation (min ~12 s, 16 s cap) and the encoder chain resets per segment, which bounds drift; cutting at linguistic boundaries measured better than fixed-length cuts (18.1 vs 18.7 WER).
• The serving stack adds a rolling decoder KV (the [prompt + audio] prefix persists across updates) and lossless speculative re-decoding (the previous hypothesis is verified in one parallel pass), making the decoder side incremental too.

Usage (WhisperLiveKit)

pip install "whisperlivekit[qwen3-streaming]"

wlk --backend qwen3-streaming --language en \
    --qwen3-streaming-audio-backend causal \
    --qwen3-streaming-tower-checkpoint qfuxa/qwen3-asr-0.6b-streaming

The tower downloads automatically; the base model comes from Qwen/Qwen3-ASR-0.6B. Works on CUDA, Apple Silicon (MPS) and CPU. See WhisperLiveKit for the WebSocket server, web UI and the OpenAI-compatible REST endpoint.

Standalone (no server), via the runtime package qwen3-asr-causal:

pip install "qwen3-asr-causal[streaming]"
qwen3-asr-causal transcribe audio.wav --backend hf --language en

from qwen3_asr_causal import Qwen3StreamingASR

asr = Qwen3StreamingASR(
    lan="en",
    qwen3_streaming_audio_backend="causal",
    qwen3_streaming_tower_checkpoint="qfuxa/qwen3-asr-0.6b-streaming",
)
streamer = asr.build_streamer("en")
# feed mel chunks of any size; the encoder consumes fixed 1.92 s blocks

Limitations

• English only. The LibriSpeech distillation degraded other languages severely (FLEURS vs the windowed backend: fr 7.8 to 37.9 WER, de 12.6 to 49.8 WER, zh 11.4 to 85.7 CER). Use the windowed backend for non-English. The recipe is language-agnostic and cheap; multilingual versions can follow the same ~6 GPU-hour path.
• Validated for the 0.6B base only (the tower must match the base model).
• Structural latency is one block (~1.9 s; 0.96 s with block_frames=96 at a small quality cost).
• Long-form robustness relies on segment resets at sentence boundaries; the encoder chain is bounded to one segment (~12-16 s) by design.
• Accuracy is below standard streaming Qwen3-ASR under the same no-rewrite contract (17.6 vs 12.6 WER long-form live): you are trading accuracy for ~constant per-chunk compute.

Provenance & license

Derived from Qwen/Qwen3-ASR-0.6B (Apache 2.0, © Alibaba Cloud). Modifications: the audio tower weights were fine-tuned (embedding self-distillation, 143k steps total) to support block-causal streaming execution; everything else is unchanged. Released under Apache 2.0. The streaming runtime, benchmark suite and measured results live in Qwen3-ASR-causal; training/eval code and the experiment log are under experiments/qwen3-causal/ in that repository.