mlx-qwen3-asr

Run Qwen3-ASR — one of the strongest open-source speech recognition models — natively on Apple Silicon.

A ground-up reimplementation of the official PyTorch model using Apple's MLX framework. Same weights, benchmarked against official/reference outputs and ground-truth eval sets, optimized for Mac GPUs via Metal. No PyTorch dependency for core transcription.

Why this exists

Qwen3-ASR is one of the strongest open-source ASR models available, with benchmark results exceeding Whisper-large-v3 across multiple languages and datasets. It supports 30 languages plus 22 Chinese dialects. But the official implementation is PyTorch + NVIDIA CUDA — it doesn't use Apple GPUs.

This project rewrites every layer for MLX so the same model runs natively on M1/M2/M3/M4 hardware. Not a wrapper — a full reimplementation with correct interleaved MRoPE, per-chunk windowed encoder attention, and all the architectural details that matter for output quality.

What's included

• Full encoder-decoder pipeline — audio encoder (Conv2d stem + windowed transformer) and text decoder (Qwen3-style with interleaved MRoPE), reimplemented from scratch for MLX
• Whisper-compatible mel frontend — native log-mel spectrogram computation with cached filterbank and Hann window
• Both model sizes — 0.6B (fast, default) and 1.7B (higher accuracy)
• Long audio support — hours-long input split at low-energy points into 30-second chunks; no 30-second feature truncation, memory released between chunks
• Word-level timestamps — native MLX forced aligner (default, 2.6x faster than PyTorch alternative) with O(n log n) LIS-based timestamp correction
• Speaker diarization (optional) — offline speaker-labeled outputs via pyannote integration (--diarize)
• 4-bit and 8-bit quantization — 8-bit matches fp16 output; 4-bit is 1.7x faster on 10 s clips, with quality measured on 100 speaker-balanced samples
• Multiple output formats — txt, json, srt, vtt, tsv
• Built-in HTTP server — mlx-qwen3-asr serve exposes the pipeline over HTTP with async jobs, OpenAI API compatibility, and Bearer token auth
• Session API — explicit model/tokenizer ownership with no hidden global state
• Speculative decoding — experimental opt-in path (0.6B drafts for 1.7B target), parity-verified
• Streaming — windowed re-decode with text-prefix rollback (the official Qwen3-ASR recipe); final text within ~2pp of offline quality on the multilingual-100 and long-form lanes
• Native WAV fast-path — custom binary WAV parser bypasses ffmpeg for PCM/float WAV files
• 735 tests — every optimization is benchmark-gated with committed JSON artifacts
• Minimal dependencies — mlx, numpy, regex, huggingface-hub

Requirements

• Apple Silicon Mac (M1/M2/M3/M4) — this is an MLX project, Metal GPU required
• Python 3.10+
• ffmpeg — required for non-WAV audio formats (mp3, m4a, flac, mp4, etc.). WAV files work without ffmpeg via the native fast-path loader
• ~1.2 GB memory for 0.6B model (fp16), ~3.4 GB for 1.7B

Installation

Install from PyPI:

pip install mlx-qwen3-asr

For video and most non-WAV audio formats, install ffmpeg on your system:

brew install ffmpeg

Install with optional timestamp alignment extras (for Japanese/Korean tokenization parity):

pip install "mlx-qwen3-asr[aligner]"

Install with optional microphone capture support:

pip install "mlx-qwen3-asr[mic]"

Install with HTTP server support:

pip install "mlx-qwen3-asr[serve]"

Install with diarization extras:

pip install "mlx-qwen3-asr[diarize]"

Note: --diarize uses pyannote.audio 4.x and defaults to pyannote/speaker-diarization-community-1. Accept the model terms on Hugging Face and set a token:

export PYANNOTE_AUTH_TOKEN=hf_...

Core ASR does not require any Hugging Face token.

For development:

git clone https://github.com/moona3k/mlx-qwen3-asr.git
cd mlx-qwen3-asr
pip install -e ".[dev]"

Quick start

Python API

from mlx_qwen3_asr import transcribe

result = transcribe("audio.wav")
print(result.text)
print(result.language)

By default, transcribe() uses Qwen/Qwen3-ASR-0.6B for fast local usage on Mac. Use Qwen/Qwen3-ASR-1.7B when you want higher accuracy and can afford higher latency/memory.

With options:

result = transcribe(
    "meeting.mp3",
    model="Qwen/Qwen3-ASR-1.7B",
    language="English",
    return_chunks=True,
    on_progress=lambda e: print(e["event"], e.get("progress", 0.0)),
    verbose=True,
)
print(result.text)
print(result.chunks)

Session API (recommended for repeated calls)

The Session object owns model and tokenizer state explicitly — no hidden globals, no cache surprises:

from mlx_qwen3_asr import Session

session = Session(model="Qwen/Qwen3-ASR-0.6B")

# Fast repeated transcription — model stays loaded
for audio_file in audio_files:
    result = session.transcribe(audio_file)
    print(result.text)

Loading models explicitly

from mlx_qwen3_asr import load_model, load_audio, transcribe

model, config = load_model("Qwen/Qwen3-ASR-0.6B")
audio = load_audio("speech.wav")
result = transcribe(audio, model=model)

CLI

mlx-qwen3-asr audio.wav

Specify model, language, and output format:

mlx-qwen3-asr recording.mp3 --model Qwen/Qwen3-ASR-0.6B --language English -f srt -o output/

Word-level timestamps:

mlx-qwen3-asr audio.wav --timestamps

Speaker-labeled output (experimental, offline):

mlx-qwen3-asr meeting.wav --diarize --num-speakers 2 -f json

Multiple files with all output formats:

mlx-qwen3-asr *.wav -f all -o transcripts/ --verbose

Stdout/file behavior:

mlx-qwen3-asr audio.wav --stdout-only        # print only (no output file)
mlx-qwen3-asr audio.wav --quiet -o out/      # write files only (no stdout text)

Language discovery:

mlx-qwen3-asr --list-languages

Environment diagnostics (ffmpeg, optional diarization deps, token status):

mlx-qwen3-asr --doctor

Run mlx-qwen3-asr --help for the full list of options.

HTTP server

Serve transcriptions over HTTP. Two endpoint styles: an async job API and an OpenAI-compatible synchronous endpoint.

pip install "mlx-qwen3-asr[serve]"
mlx-qwen3-asr serve --api-key $(openssl rand -hex 16)

Submit audio and poll for results:

# Submit
curl -X POST http://localhost:8765/transcribe \
  -H "Authorization: Bearer YOUR_KEY" \
  -F "audio=@recording.wav"

# Poll
curl http://localhost:8765/jobs/JOB_ID \
  -H "Authorization: Bearer YOUR_KEY"

Or use the OpenAI-compatible endpoint with existing SDK code:

from openai import OpenAI

client = OpenAI(api_key="YOUR_KEY", base_url="http://localhost:8765/v1")
result = client.audio.transcriptions.create(
    model="Qwen/Qwen3-ASR-0.6B",
    file=open("recording.wav", "rb"),
)
print(result.text)

The async API is better for long audio (no HTTP timeout risk). The OpenAI endpoint blocks until done — simpler for short clips and SDK integration. The server also implements /v1/models for SDK clients that perform model discovery.

See docs/server/ for the full API spec, deployment guide, and architecture decision record. See examples/ for copy-paste workflows covering the OpenAI-compatible server, subtitles, meetings, scanner/noisy audio, and batch folders.

Performance on Apple Silicon

Measured on Apple M4 Pro (48 GB), macOS 26, v0.4.0. All numbers come from committed JSON artifacts under docs/benchmarks/; see docs/BENCHMARKS.md for the full breakdown and docs/benchmarks/2026-09-07-quality-matrix-refresh.md for the exact commands.

Latency (median of 10 runs)

English quality (LibriSpeech, 100 speaker-balanced samples per subset)

Quantization (0.6B, LibriSpeech, 100 speaker-balanced samples per subset)

8-bit reproduces fp16 output exactly on test-clean. 4-bit trades about +0.3pp (clean) to +1.4pp (other) WER for the lowest latency.

Multilingual quality (FLEURS, 10 languages x 10 samples)

The 1.7B delivers a 30% relative improvement at 1.9x the latency. Per-language tables are in docs/BENCHMARKS.md.

MLX vs PyTorch quality (0.6B, same 100 multilingual clips, v0.4.0)

68% of clips produce identical text; the rest differ by lexical or numeric surface form (10,000 vs zehntausend) or punctuation, not by quality. On LibriSpeech test-other the two are within 0.11pp WER, and on 80-second clips MLX scores 10.59% vs 17.99% because it chunks at pauses while the reference decodes the whole clip. The PyTorch reference runs on CPU on a Mac, so it is 7x to 20x slower here; that is a platform difference, not a like-for-like GPU comparison.

On the real-world mixed lane (AMI IHM meetings + Earnings22 chunked, n=200, measured February 2026), MLX was within 0.19pp WER of PyTorch (23.23% vs 23.04%).

Optimizations applied

• Preallocated KV cache with in-place slice writes and rollback-safe trimming
• Direct grouped-query fused attention via mx.fast.scaled_dot_product_attention (no explicit K/V head expansion)
• Hybrid encoder windowing — dense block-diagonal mask for short audio, segmented per-window execution for long contexts (up to 4.2x faster on long audio)
• Cached mel filterbank and Hann window — computed once, reused across calls
• Native WAV fast-path — custom binary parser bypasses ffmpeg process startup for PCM/float WAV files (up to 25% faster on quantized short clips)
• Native in-repo BPE tokenizer — no transformers dependency in runtime transcription path
• Cached model and tokenizer instances — repeated transcribe() calls skip reload overhead
• 4-bit / 8-bit quantization — 1.3x to 1.7x faster than fp16 with explicit per-profile quality reporting

Full benchmark report: docs/BENCHMARKS.md. Latest refresh snapshot: docs/benchmarks/2026-09-07-quality-matrix-refresh.md. All benchmark artifacts are committed under docs/benchmarks/ for reproducibility.

Model quality

Word error rates from the Qwen3-ASR technical report compared against current open-source and proprietary leaders (lower is better):

English benchmarks

Chinese + multilingual benchmarks

Robustness benchmarks

GPT-4o-Transcribe leads on clean English read speech (1.39 WER). Parakeet-TDT-0.6B is strong on English. But Qwen3-ASR dominates on Chinese, multilingual, noisy, and accented speech — and is the only open-source model competitive across all categories.

Parakeet numbers from model card. All other numbers from the Qwen3-ASR paper. Robustness benchmarks are Qwen3-ASR internal test sets.

Correctness validation

This implementation is validated against the official PyTorch model via multiple parity gates:

• MLX vs PyTorch head-to-head — on the current multilingual-100 artifact, MLX shows lower aggregate primary error than PyTorch (9.54% vs 10.34%)
• Token-level greedy parity — multilingual-100 parity artifacts show 67-68% exact text match and 64-66% exact token match across 10 languages (v0.4.0 and v0.4.1 runs); remaining diffs are mostly lexical/numeric surface-form differences. Encoder outputs sit within 0.003 mean absolute error of the fp32 PyTorch reference, with the last tail token halved to 0.005 by the v0.4.1 tail-padding fix
• Expanded parity suite — tested across LibriSpeech test-clean, test-other, synthetic long mixes, and noise variants (SNR 10dB, 5dB)
• Long-form head-to-head — on 10 multilingual clips (78-90s each) MLX scored lower error than the PyTorch reference (10.6% vs 18.0% primary error) because it chunks at pauses while the reference decodes each clip whole; full transcripts are not token-identical
• Mel spectrogram parity — custom MLX mel matches HuggingFace WhisperFeatureExtractor with MAE < 3e-7
• Native aligner parity — MLX forced aligner matches official qwen-asr backend with 100% text match rate, <6ms timing MAE, and 2.64x speed advantage on 50 LibriSpeech samples

Model variants

Both models use interleaved Multi-dimensional RoPE (MRoPE) with sections [24, 20, 20], 128-bin mel spectrograms, and the same tokenizer (vocabulary size 151,936).

# Default: 0.6B (fast, ~1.2 GB memory)
result = transcribe("audio.wav")

# Accuracy-first: 1.7B (~3.4 GB memory)
result = transcribe("audio.wav", model="Qwen/Qwen3-ASR-1.7B")

Timestamps

Word-level timestamps via forced alignment using a dedicated aligner model (Qwen/Qwen3-ForcedAligner-0.6B). This path is native MLX (no PyTorch backend bridge):

mlx-qwen3-asr audio.wav --timestamps

result = transcribe("audio.wav", return_timestamps=True)
for segment in result.segments:
    print(f"{segment['start']:.2f}s - {segment['end']:.2f}s: {segment['text']}")

SRT/VTT outputs are grouped into subtitle-friendly phrase segments (not one word per cue). When -f srt or -f vtt is requested in offline mode, timestamps are auto-enabled.

Measured parity (LibriSpeech test-clean, n=50):

The aligner uses O(n log n) LIS-based timestamp correction (Fenwick tree) for monotonicity repair, validated against the legacy O(n^2) implementation via randomized parity tests.

For Japanese/Korean timestamp alignment, install the [aligner] extra so nagisa/soynlp tokenization matches the official path.

Speaker diarization (optional)

Speaker attribution is available as an offline optional path powered by pyannote.audio:

result = transcribe("meeting.wav", diarize=True)
print(result.speaker_segments)

mlx-qwen3-asr meeting.wav --diarize -f json

Current status:

• The public API/CLI and output schema are stable.
• The diarization backend is pyannote.audio 4.x (installed via [diarize] extra).
• The default model is pyannote/speaker-diarization-community-1; accept its Hugging Face terms and configure PYANNOTE_AUTH_TOKEN (or HF_TOKEN).
• PYANNOTE_MODEL_ID can point to another pyannote pipeline or a local offline clone.
• --diarize auto-enables timestamps and is not supported in --streaming/--mic mode.
• --diarize-device {auto,cpu,mps,cuda} (Python: diarization_device) selects where the pyannote pipeline runs. The default auto prefers MPS, then CUDA, then CPU; on an M-series Mac this cuts the diarization stage from minutes to seconds with identical output. If the accelerator cannot run the pipeline, the run warns and falls back to CPU.
• Migration note (2026-02-15): legacy diarization window/hop controls were removed (diarization_window_sec, diarization_hop_sec, --diarization-window-sec, --diarization-hop-sec). Speaker-count controls remain (--num-speakers, --min-speakers, --max-speakers).

Diarization setup troubleshooting

1. Install optional diarization dependencies:

   pip install "mlx-qwen3-asr[diarize]"
   

2. Accept the default pyannote/speaker-diarization-community-1 model terms on Hugging Face and set a token:

   export PYANNOTE_AUTH_TOKEN=hf_...
   

3. Run a quick smoke test:

   mlx-qwen3-asr meeting.wav --diarize -f json
   

Common errors and fixes:

• requires optional dependency 'pyannote.audio': install [diarize] extra.
• requires PyTorch via pyannote dependencies: reinstall [diarize] extra in the active environment.
• Failed to initialize pyannote pipeline ...: accept model terms on Hugging Face, set PYANNOTE_AUTH_TOKEN (or HF_TOKEN), and inspect the Root cause: details.
• --streaming does not support --diarize / --mic does not support --diarize: use offline file transcription mode for diarization.

Quantization

Pre-quantized artifacts, validated with this runtime on 100 speaker-balanced LibriSpeech test-clean clips (mlx-qwen3-asr >= 0.4.3):

mlx-qwen3-asr audio.wav --model moona3k/mlx-qwen3-asr-0.6b-4bit

Each model card carries the recipe, the per-sample evaluation reference and a reproduce command. The mlx-community/Qwen3-ASR-* checkpoints also load (since 0.4.1 they run in float16 rather than being promoted to float32).

Convert your own:

python scripts/convert.py \
  --model Qwen/Qwen3-ASR-0.6B \
  --quantize 4 --encoder-bits 8 --group-size 64 \
  --output-dir ./qwen3-asr-4bit

mlx-qwen3-asr audio.wav --model ./qwen3-asr-4bit

The audio encoder carries most of the 4-bit quality loss (0.6B all-4-bit: 2.63% WER; with an 8-bit encoder: 2.37%), so --encoder-bits 8 is the recommended 4-bit recipe; 8-bit throughout is lossless on this lane. Speed: 4-bit is about 1.7x and 8-bit about 1.3x faster than fp16 on a 10 s clip.

Publish to HuggingFace (converts, load-checks and uploads; --from-dir uploads an already validated directory):

python scripts/publish_quantized.py \
  --source-model Qwen/Qwen3-ASR-0.6B \
  --repo-id YOUR_USER/mlx-qwen3-asr-0.6b-4bit \
  --bits 4

The token is read from HF_TOKEN, else from the gitignored file .secrets/hf_token at the repo root, else from huggingface-cli login.

Output formats

mlx-qwen3-asr audio.wav -f txt           # plain text
mlx-qwen3-asr audio.wav -f srt -o out/   # SRT subtitles
mlx-qwen3-asr audio.wav -f json          # structured JSON
mlx-qwen3-asr audio.wav -f vtt -o out/   # WebVTT
mlx-qwen3-asr *.wav -f all -o out/       # all formats at once

Supported: txt, json, srt, vtt, tsv. Subtitle formats (srt/vtt) require timestamp segments and are only supported in offline mode.

Supported languages

Qwen3-ASR officially lists 30 core languages:

Plus 22 Chinese dialects (Sichuan, Shanghai, Cantonese, and others), for 52 total language/dialect variants.

Print CLI-accepted aliases/codes:

mlx-qwen3-asr --list-languages

Experimental features

Speculative decoding

Uses the 0.6B model as a draft to accelerate 1.7B inference. Currently parity-safe but slower on tested workloads due to draft audio encoder overhead:

mlx-qwen3-asr audio.wav \
  --model Qwen/Qwen3-ASR-1.7B \
  --draft-model Qwen/Qwen3-ASR-0.6B \
  --num-draft-tokens 4

result = transcribe(
    "audio.wav",
    model="Qwen/Qwen3-ASR-1.7B",
    draft_model="Qwen/Qwen3-ASR-0.6B",
    num_draft_tokens=4,
)

Status: greedy parity verified, but 0.53-0.55x on short/10s clips. Not enabled by default until benchmark evidence shows net speed wins.

Domain vocabulary context

When transcribing specialized audio — earnings calls, medical dictation, legal proceedings — the model can confuse rare terms with more common homophones. The context parameter lets you provide a hint: a string of domain-specific words or phrases that gets injected into the system prompt, nudging the decoder toward the correct vocabulary.

This matches the official Qwen3-ASR context API. The format is space-separated terms:

# Finance: avoids "e-bit-da" → "EBITDA", "FX" not "effects", etc.
result = transcribe("earnings-call.wav", context="EBITDA non-GAAP FX hedging")

# Medical
result = transcribe("consult.wav", context="metformin HbA1c nephropathy")

# Also works with streaming
state = init_streaming(context="EBITDA non-GAAP FX hedging")

mlx-qwen3-asr earnings-call.wav --context "EBITDA non-GAAP FX hedging"

For batch transcription, pass a list of per-audio context strings:

results = transcribe_batch(
    [audio_en, audio_zh],
    context=["EBITDA non-GAAP", "交易 停滞"],
)

When omitted, the system prompt is empty (matching the official default) — no domain bias is applied.

Streaming

Near-real-time transcription following the official streaming recipe: each chunk decodes the accumulated window (bounded by max_context_sec) with the previous text, minus its last few tokens, forced as a prefix. Encoder output and decoder KV for the parts of the window that cannot change are reused across chunks, and when the window fills it is committed at a pause rather than mid-word. Partial text is stable and the final text tracks offline quality (multilingual-100 primary error 11.3% streaming vs 9.5% offline; long-form 11.9% vs 10.6%).

from mlx_qwen3_asr.streaming import (
    init_streaming,
    feed_audio,
    finish_streaming,
    streaming_metrics,
)

state = init_streaming(chunk_size_sec=2.0, max_context_sec=30.0)
for chunk in audio_chunks:
    state = feed_audio(chunk, state)
    print(state.text)
state = finish_streaming(state)
print(streaming_metrics(state))

CLI:

mlx-qwen3-asr --streaming --stream-finalization-mode accuracy audio.wav
# Optional: speech-aware boundary selection near chunk edges
mlx-qwen3-asr --streaming --stream-endpointing-mode energy audio.wav

Live microphone transcription:

mlx-qwen3-asr --mic
mlx-qwen3-asr --mic --language Japanese

Optional microphone flags: --mic-device, --mic-duration-sec, --mic-sample-rate.

• Ingests small PCM chunks (default 2s)
• Each chunk re-encodes the accumulated window and decodes from the previous text minus the last unfixed_token_num tokens, so per-chunk cost is bounded by the window, not by session length; encoder output and decoder KV for complete 8 s attention windows are reused across chunks (RTF 0.08 on 5-20 s clips, 0.08 on 75 s clips with a 30 s window)
• Bounded context window (default 30s): when it fills, its text is committed and a new window starts
• Prefix rollback controls (unfixed_chunk_num, unfixed_token_num)
• stable_text is monotonic by design: corrections that would shorten already-stable prefix text are intentionally not applied to the stable prefix (favoring stability over maximal editability in partial output)
• Optional speech-aware endpointing (endpointing_mode="energy") that selects low-energy boundaries near chunk edges
• finalization_mode and enable_tail_refine are accepted for compatibility; the window re-decode at finish covers what the former tail-refine pass did
• Input validation: handles int16 PCM normalization, non-1D arrays, empty input

API reference

transcribe(audio, *, model, draft_model, context, language, return_timestamps, diarize, diarization_num_speakers, diarization_min_speakers, diarization_max_speakers, diarization_device, return_chunks, forced_aligner, dtype, max_new_tokens, num_draft_tokens, verbose, on_progress)

Transcribe audio to text. Accepts a file path, numpy array, mx.array, or (array, sample_rate) tuple. Returns a TranscriptionResult.

max_new_tokens=None (default) uses a duration-aware per-chunk decode budget to avoid runaway generation on noisy inputs that do not emit EOS. Pass an integer to override the cap explicitly. If you use unusually long custom chunks and see truncated=True, pass a larger explicit value for that workload.

Additional Python entry points:

• transcribe_batch(audios, ...) and transcribe_batch_async(audios, ...)
• transcribe_async(audio, ...)

Session(model, *, dtype, tokenizer_model)

Explicit transcription session. Owns model and tokenizer state with no hidden globals.

• Offline: session.transcribe(audio, ...) with the same parameters as top-level transcribe.
• Async: await session.transcribe_async(audio, ...).
• Streaming: session.init_streaming(...), session.feed_audio(pcm, state), session.finish_streaming(state).
• Introspection: session.model_info (model id/path, dtype, vocab size, model-declared language codes).

streaming_metrics(state)

Return streaming diagnostics for a session state:

• partial_stability
• rewrite_rate
• finalization_delta_chars

load_model(name_or_path, *, dtype)

Load a Qwen3-ASR model and config from HuggingFace or local path. Returns (model, config).

load_audio(path_or_url)

Load and resample audio to mono 16 kHz. Returns an mx.array.

ForcedAligner(model_path, *, dtype, backend)

Word-level forced aligner. Native backend: mlx (default).

TranscriptionResult

Frozen dataclass:

• text (str) — transcribed text
• language (str) — detected or forced language (canonicalized names, e.g. English)
• segments (list[dict] | None) — word-level timestamps when requested: [{"text": "hello", "start": 0.5, "end": 0.8}, ...]
• chunks (list[dict] | None) — chunk-level transcript and generation metadata when return_chunks=True
• speaker_segments (list[dict] | None) — speaker-attributed spans when diarize=True: [{"speaker": "SPEAKER_00", "start": 0.0, "end": 2.0, "text": "..."}, ...]
• finish_reason (str | None) — aggregate decode stop reason: eos, repetition, length, or mixed
• truncated (bool) — true when any chunk exhausted its token budget before EOS/repetition

Quality gates

This project enforces parity with the official PyTorch implementation. No optimization lands without passing quality gates and committing benchmark artifacts.

# Unit tests (735 tests)
pytest -q

# Fast quality gate
python scripts/quality_gate.py --mode fast

# Release gate with token-level parity (downloads model weights)
RUN_REFERENCE_PARITY=1 python scripts/quality_gate.py --mode release

# Speaker-balanced WER evaluation (100 samples)
python scripts/eval_librispeech.py --subset test-clean --samples 100 --sampling speaker_round_robin

# Latency benchmark
python scripts/benchmark_asr.py tests/fixtures/test_speech.wav \
  --model Qwen/Qwen3-ASR-0.6B --runs 5 \
  --json-output docs/benchmarks/latest.json

Additional quality lanes available:

• Aligner parity: RUN_ALIGNER_PARITY=1 — validates MLX aligner against official backend
• Expanded parity suite: RUN_REFERENCE_PARITY_SUITE=1 — test-clean, test-other, long mixes, noise variants with Unicode-safe text comparison
• Multilingual parity: manifest-driven workflow via scripts/build_multilingual_manifest.py for cross-language validation
• Streaming manifest quality: RUN_STREAMING_MANIFEST_QUALITY_EVAL=1 with STREAMING_MANIFEST_QUALITY_EVAL_JSONL=... — multi-file streaming stability/rewrite/finalization lane via scripts/eval_streaming_manifest.py
• Real-world long-form quality: RUN_REALWORLD_LONGFORM_EVAL=1 on full-recording Earnings22 manifests
• Diarization quality: RUN_DIARIZATION_QUALITY_EVAL=1 with DIARIZATION_QUALITY_EVAL_JSONL=... — DER/JER lane via scripts/eval_diarization.py

See docs/QUALITY_GATE.md for full documentation. Evaluation coverage status and prioritized gaps are tracked in docs/EVAL_GAPS.md.

Architecture overview

Audio (16kHz mono)
  → 128-bin log-mel spectrogram (native MLX, Whisper-compatible)
  → Conv2d stem (3 layers, stride 2 each → 8x downsample)
  → Sinusoidal position embeddings
  → Windowed transformer encoder (18 or 24 layers, hybrid dense/segmented attention)
  → LayerNorm + GELU projection → audio features

Chat-template prompt (context is optional domain vocabulary, empty by default):
  <|im_start|>system\n{context}<|im_end|>
  <|im_start|>user\n<|audio_start|><|audio_pad|>*N<|audio_end|><|im_end|>
  <|im_start|>assistant\n

  → Token embedding (151,936 vocab)
  → Replace audio_pad positions with encoded audio features
  → Qwen3 text decoder (28 layers, interleaved MRoPE, SwiGLU, RMSNorm)
  → Autoregressive decode with preallocated KV cache
  → Parse output: "language English<asr_text>transcribed text here"

Key architectural details:

• Interleaved MRoPE — sections [24, 20, 20] with stride-3 frequency assignment across temporal, height, and width dimensions. This is the detail other MLX ports get wrong (using standard RoPE or chunked assignment).
• Audio encoder uses LayerNorm + bias — different from the text decoder which uses RMSNorm without bias.
• Q/K norms — RMSNorm applied per-head on queries and keys before attention (Qwen3 innovation).

Project structure

mlx_qwen3_asr/
├── transcribe.py         # Public pipeline: transcribe, batch, async, diarization glue
├── session.py            # Session API: explicit model/tokenizer ownership
├── streaming.py          # Windowed re-decode with text-prefix rollback
├── cli.py                # CLI (transcribe, serve, --mic, --doctor)
├── server.py             # HTTP server + OpenAI-compatible endpoint
├── audio.py              # Audio I/O, WAV fast path, mel spectrogram
├── chunking.py           # Energy-based long-audio splitting
├── encoder.py            # Audio encoder (Conv2d stem + windowed transformer)
├── decoder.py            # Text decoder (GQA, SwiGLU, KV cache)
├── mrope.py              # Interleaved MRoPE
├── attention.py          # Shared SDPA helper
├── model.py              # Qwen3ASRModel: audio-text fusion, prefill/step
├── generate.py           # Greedy + speculative decoding
├── forced_aligner.py     # Native MLX forced aligner + LIS correction
├── diarization.py        # Optional pyannote integration
├── tokenizer.py          # Native BPE tokenizer, language aliases, output parsing
├── load_models.py        # HF download, weight loading, model cache
├── convert.py            # Weight key remapping + Conv2d transpose
├── writers.py            # txt/json/srt/vtt/tsv writers, subtitle cue grouping
└── config.py             # Dataclass configs

tests/                    # 12,672 lines, 735 tests
scripts/                  # Benchmarks, evaluation, conversion, publishing
docs/                     # Architecture, decisions, benchmarks, roadmap
docs/benchmarks/          # 160+ committed artifacts for reproducibility

Development

git clone https://github.com/moona3k/mlx-qwen3-asr.git
cd mlx-qwen3-asr
pip install -e ".[dev]"
pytest -q                 # 735 tests

Acknowledgments

• Qwen team at Alibaba for the Qwen3-ASR model
• Apple MLX team for the MLX framework
• mlx-whisper for architecture patterns and inspiration
• Contributors: @ggshr9 (GPU diarization, --diarize-device), @Tadanobu0 (server inference-thread fix), @cms42 (GPU memory release between chunks)

License

Apache 2.0. See LICENSE for details.