🎙️ X-ASR

Streaming-focused automatic speech recognition models based on icefall/k2, Zipformer, and sherpa-onnx.

_{Participating Institutions}

🌐 中文版

🤗 Hugging Face Hub | 🧩 ModelScope | 🪐 Hugging Face Space | 🎧 Online Demo | 🎙️ Local Live Demo | 🚀 Deployment Guide

📄 X-ASR-zh-en Technical Report: Coming Soon

🔍 Overview | 📅 Timeline | 📦 Model Releases | 🎙️ Applications | 📊 Evaluation | 🚀 Quick Start | 🗂️ Repository Layout

---

🔍 Overview

🧩 X-ASR

X-ASR is a series of automatic speech recognition models built with the icefall framework. The series focuses on streaming ASR and low-latency deployment, while also supporting offline recognition. This repository currently releases an initial batch of Chinese-English streaming ASR models, and the X-ASR series will be continuously maintained, updated, and scaled across languages, model architectures, and training data.

🤖 X-ASR-zh-en

X-ASR-zh-en is trained on approximately 1 million hours of open-source and collected speech data. It is designed as an offline-streaming unified transducer ASR model with the Zipformer architecture, supporting both offline decoding and true streaming decoding. The model provides multiple streaming chunk sizes: 160 ms, 480 ms, 960 ms, and 1920 ms, supports punctuation and casing, and can be conveniently deployed with sherpa-onnx.

📅 Timeline

📦 Model Releases

⭐ Highlights

🎙️ Applications

We welcome more experiments and real-world use cases built on top of X-ASR. The following downstream applications are based on X-ASR and have been synced into this repository.

🧪 Vibe-Coding Application with FireRedVAD

⬇️ Desktop Package Download

Vibe XASR · a local voice input method powered by X-ASR

⬇️  Download for macOS  →
_{Universal (Apple Silicon + Intel) · macOS 15.0+ · signed & notarized · auto-updates in-app}

Hold a hotkey, speak, and the text lands right at your cursor — 100% local & offline, your data never leaves the device. The X-ASR streaming engine turns Chinese & English speech (freely code-switched) into text in real time, system-wide.

Core features

• 🎙️ Three dictation modes — insert-on-finish · live streaming (types as you talk) · OnCall standby (floating window)
• 📋 Built-in pad & history — saved by date; copy / edit / export
• 📖 Personal dictionary — hotwords, homophone correction, replace rules
• ✨ AI Polish (Beta) — optional cloud LLM smooths wording & removes fillers (opt-in, off by default)
• 🔒 Privacy-first & auto-update — fully offline; one-click upgrades inside the app

_{🪟 A Windows build is also available (in Releases) — an early preview, not yet fully tested, kept in sync with the latest macOS features. Please report issues as you run into them.}

📊 Evaluation

The following results are for the current X-ASR-zh-en release. All results are reported with greedy search. Measurement: English results use WER (%), and Chinese results use CER (%); lower is better.

🧪 Public ASR Benchmarks

Note: Bold numbers indicate the best result among the listed modes for each benchmark column.

🏆 Public Benchmark Model Comparison

The following table compares representative ASR models on the same public benchmark columns. Ranks are computed by AVG across the five listed columns; lower is better. Parameter sizes are shown when provided by the source sheet.

🧭 Vertical-Domain Benchmarks

The following results report GigaSpeechBench vertical-domain performance for the current X-ASR-zh-en release. Values are WER/CER percentages; lower is better. Domain abbreviations follow the GigaSpeechBench vertical-domain labels.

CH

EN

🎧 Demo

A sherpa-onnx based online demo is available here:

• https://stream-asr.sjtuxlance.com/

Demo video:

Open demo video

🚀 Quick Start

This section shows how to build and run the sherpa-onnx WebSocket streaming server and the corresponding WebSocket client. For complete deployment arguments, model switching, runtime options, and production notes, see the deployment guide.

1. Clone or download model artifacts

This repository uses Git LFS for ONNX model artifacts and demo media. Install Git LFS before cloning or before pulling large files.

GitHub

Use GitHub when you want the full project repository, bilingual documentation, training references, deployment examples, and issue-tracking context.

git lfs install
git clone https://github.com/Gilgamesh-J/X-ASR.git
cd X-ASR
git lfs pull

Hugging Face

Use Hugging Face when you want the model artifact page and standard HF Hub download tooling.

hf download GilgameshWind/X-ASR-zh-en \
  --local-dir ./X-ASR-zh-en

ModelScope

Use ModelScope when you prefer the ModelScope mirror or Git LFS clone from ModelScope.

git lfs install
git clone https://www.modelscope.ai/Gilgamesh-J/X-ASR-zh-en.git
cd X-ASR-zh-en
git lfs pull

2. Prepare the sherpa-onnx runtime

If you cloned the full GitHub project, enter:

cd X-ASR/X-ASR-zh-en/deployment

If you downloaded from Hugging Face or cloned from ModelScope, enter:

cd X-ASR-zh-en/deployment

Then prepare the Python environment:

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

3. Start the WebSocket server

The server wraps sherpa_onnx.OnlineRecognizer and exposes a WebSocket endpoint. Each WebSocket connection keeps an independent recognizer session, so concurrent clients do not share decoding state. The example below starts the 160 ms streaming model on CPU and listens on ws://0.0.0.0:6666.

python infer_and_client/sherpa_streaming_server.py \
  --host 0.0.0.0 \
  --port 6666 \
  --tokens models/chunk-160ms-model/tokens.txt \
  --encoder models/chunk-160ms-model/encoder-160ms.onnx \
  --decoder models/chunk-160ms-model/decoder-160ms.onnx \
  --joiner models/chunk-160ms-model/joiner-160ms.onnx \
  --provider cpu \
  --sample-rate 16000 \
  --feature-dim 80 \
  --num-threads 1 \
  --decoding-method greedy_search \
  --model-type zipformer2 \
  --enable-endpoint-detection 0 \
  --text-format none

The --tokens, --encoder, --decoder, and --joiner files must come from the same model directory.

Optional: interactive tail-probe mode

The default mode decodes only from audio chunks received from the client. For interactive voice-input scenarios, you can enable the optional energy-based tail probe. After speech activity is observed and low-energy audio lasts long enough, the server appends a short silence segment and runs decoding again. This can help trailing partial results refresh after the user pauses.

python infer_and_client/sherpa_streaming_server.py \
  --host 0.0.0.0 \
  --port 6666 \
  --tokens models/chunk-160ms-model/tokens.txt \
  --encoder models/chunk-160ms-model/encoder-160ms.onnx \
  --decoder models/chunk-160ms-model/decoder-160ms.onnx \
  --joiner models/chunk-160ms-model/joiner-160ms.onnx \
  --provider cpu \
  --sample-rate 16000 \
  --feature-dim 80 \
  --num-threads 1 \
  --decoding-method greedy_search \
  --model-type zipformer2 \
  --enable-endpoint-detection 0 \
  --text-format none \
  --enable-energy-tail-probe 1 \
  --low-energy-rms 0.003 \
  --speech-rms 0.010 \
  --min-speech-ms 200 \
  --min-silence-ms 500 \
  --tail-probe-ms 500 \
  --tail-probe-cooldown-ms 1000

Tune the RMS and silence thresholds according to microphone gain, background noise, and frontend audio chunking behavior. Keep --enable-energy-tail-probe 0 for the most conservative server behavior.

4. Run the WebSocket client

Open another terminal:

cd X-ASR-zh-en/deployment
source .venv/bin/activate

python infer_and_client/sherpa_streaming_client.py \
  --server-uri ws://127.0.0.1:6666 \
  --wav /path/to/test.wav \
  --chunk-ms 100 \
  --simulate-realtime 1

The client loads a WAV file, converts or resamples it to 16 kHz mono int16 PCM, sends binary PCM chunks over WebSocket, and prints partial/final recognition results returned by the server. With --simulate-realtime 1, --chunk-ms 100 means one audio packet is sent roughly every 100 ms.

5. WebSocket protocol

The provided client and server use a minimal streaming protocol:

For detailed deployment instructions, see X-ASR-zh-en/deployment/README.md. For the script-level server/client guide and full parameter reference, see X-ASR-zh-en/deployment/infer_and_client/README.md.

🗂️ Repository Layout

X-ASR/
|-- README.md
|-- README_zh.md
|-- LICENSE
|-- assets/
|   |-- figures/
|   |   |-- demo-preview.png
|   |   `-- zipformer.png
|   |-- demos/
|   |   `-- demo.mov
|   `-- institutions/
|       |-- sjtu.png
|       |-- sii.png
|       |-- fudan.png
|       `-- hust.png
`-- X-ASR-zh-en/
    |-- deployment/
    |   |-- README.md
    |   |-- requirements.txt
    |   |-- infer_and_client/
    |   |   |-- README.md
    |   |   |-- sherpa_streaming_infer.py
    |   |   |-- sherpa_streaming_server.py
    |   |   `-- sherpa_streaming_client.py
    |   |-- x-asr-live-demo/
    |   |   |-- README.md
    |   |   |-- README_zh.md
    |   |   |-- live_asr.py
    |   |   |-- download_models.sh
    |   |   |-- requirements.txt
    |   |   `-- assets/
    |   `-- models/
    |       |-- README.md
    |       |-- chunk-160ms-model/
    |       |-- chunk-480ms-model/
    |       |-- chunk-960ms-model/
    |       `-- chunk-1920ms-model/
    `-- zipformer/
        |-- README.md
        |-- train.py
        |-- finetune.py
        |-- decode.py
        |-- streaming_decode.py
        |-- export.py
        |-- export-onnx.py
        |-- export-onnx-streaming.py
        |-- model.py
        |-- zipformer.py
        |-- data/
        |   |-- lang_5000/
        |   |   |-- bpe.model
        |   |   `-- tokens.txt
        |   `-- lang_5000_with_punctuation/
        |       |-- bpe_punc.model
        |       `-- tokens.txt
        `-- checkpoint/
            |-- pretrained.pt
            `-- fintuned_with_punctuation.pt

X-ASR-zh-en/deployment/ contains runnable sherpa-onnx deployment artifacts, including the WebSocket server/client path and the local live ASR application demo. X-ASR-zh-en/zipformer/ contains the icefall/Zipformer training, decoding, export recipe files, tokenizer/data files, and released PyTorch checkpoints for the model.

🤝 Contributing

We welcome feedback and contributions in the following areas:

• Deployment issues on different CPU/GPU environments
• Streaming latency and stability reports
• Evaluation results on new datasets or domains
• Requests for new languages or future releases
• Improvements to documentation and examples

When reporting deployment problems, please include the environment, command, input audio format, and error log.

📜 License

This project is released under the Apache-2.0 License.

🙏 Acknowledgements

This model series is trained with icefall and deployed with sherpa-onnx.

• icefall: https://github.com/k2-fsa/icefall
• sherpa-onnx: https://github.com/k2-fsa/sherpa-onnx