mlx-r2t2-stt-swift

The MLXEngine stt package for Confucius4-R2T2 — NetEase Youdao's 160 ms, append-only streaming speech-to-text on the Qwen3-ASR-1.7B architecture (Chinese + English tuned, 30 languages, hotword / context biasing). The dictation-latency engine next to mlx-nemotron-stt-swift; the second .incremental provider on the live plane after mlx-vibevoice-asr-swift.

• Model: netease-youdao/Confucius4-R2T2 via mlx-community/Confucius4-R2T2-8bit (default, 2.5 GB) or -bf16 (4.1 GB).
• Core: qwen3-asr-mlx-swift (Qwen3ASR).
• Capability: stt — STTControls(attributesSpeakers: false, supportsContextBiasing: true, liveDiscipline: .incremental); conforms to LiveTranscribing (engine.transcribeLive).
• Licence: weights — NetEase Youdao Model Use License Agreement = SPDXLicense.neteaseYoudaoModelUse, allowlisted in engine v0.57.0 (AB-A-0087), so the package admits under the default .permissiveOnly with no advisory. §2.2 keeps a scale gate: a separate licence is required above 100 M monthly active users or RMB 1 B annual revenue. Port code MIT.

Install

.package(url: "https://github.com/xocialize/mlx-r2t2-stt-swift.git", from: "0.1.0"),
// …
.product(name: "MLXR2T2STT", package: "mlx-r2t2-stt-swift"),

Usage

let engine = MLXServeEngine()
_ = try await engine.register(R2T2STTPackage.registration, configuration: R2T2STTConfiguration())
_ = try await engine.prepare(.stt)

// One-shot (runs the same streaming loop over the file — bounded memory, any length)
let r = try await engine.run(STTRequest(audio: wav, language: "English", context: ["MLX", "Qwen3"]))

// Live (contract 1.39.0): push 16 kHz mono float; chunks carry NEW text only — append them.
// Hotwords work live too: STTSessionRequest(language: "English", context: ["MLX", "Qwen3"])
let session = try await engine.transcribeLive(STTSessionRequest(language: "English"))
Task { for try await c in session.updates { print(c.text, terminator: "") } }
_ = session.push(samples, sampleRate: 16000)
session.finish()

Committed text is never revised, so every chunk reports committedThrough == processedSeconds. The loop does hold back its newest, still-unstable tokens until a later chunk, so text for the last few hundred milliseconds of audio can arrive a chunk or two after that audio was pushed.

Memory

Declared from phys_footprint, measured with r2t2-gates val on an M5 Max (release build, the engine's default GPU-cache policy):

• Bounded in session length. The 16 s rolling window caps the prompt at about 400 tokens, so memory does not grow with the session. That is also why the package does not map STTSessionRequest.plannedDuration: there is nothing extra to reserve for a long plan, and the engine still ends a planned session itself.
• Most of the activation is the engine's buffer pool, not the model. R2T2's own MLX working set is about 1.1 GB above the weights, flat in clip length. Each chunk re-prefills a prompt of a new length, so freed buffers rarely fit the next request and the recycling pool fills to its cap (2 GiB under the default policy on a large Mac). With pooling off (GPUCacheConfiguration(limit: .bytes(0))), phys activation falls to about 0.06 GB, but runs are about 50 % slower. The pool is host policy, so the declaration includes it.
• After a run, about 2 GB of released buffers stay mapped in the Metal heap until memory pressure or process exit. No array is retained: MLX's active memory returns exactly to the weights. r2t2-gates mem prints the breakdown.

Gates

swift run -c release r2t2-gates g5  --model <dir> --wav clip.wav --language English   # engine path
swift run -c release r2t2-gates g6  --model <dir> --wav clip.wav --language English   # LIV-1..6
swift run -c release r2t2-gates val --model <dir> --wav clip.wav --quant int8         # phys footprint
swift run -c release r2t2-gates mem --model <dir> --wav clip.wav [--pool 0]           # memory breakdown (diagnostic)

val gates the declaration on whatever clip it is given. The stamped runs used a 6.7 s clip and continuous 163 s (Mandarin) and 182 s (English) streams, where the window has long since filled. Numerics are gated in the core (qwen3asr-gates g0–g4, whose goldens and clips ship in its repo). Stamps live in the core's PORTING-SPEC.md.