Rust Streaming ASR WebSocket Server

A high-performance WebSocket streaming ASR server built with Rust, implementing async I/O + parallel CPU architecture for speech recognition.

Architecture

This server follows the optimal pattern for streaming ASR:

• Async WebSocket Layer: Uses Tokio and Axum to handle thousands of concurrent WebSocket connections efficiently
• CPU-Bound ASR Processing: Dedicated thread pool using Rayon for parallel speech recognition inference
• Non-Blocking Design: Audio processing is offloaded to spawn_blocking threads, keeping the async runtime free for I/O

Features

• High Performance: Async I/O for WebSocket handling + parallel threads for CPU-intensive ASR
• Configurable: TOML-based configuration system
• WebSocket Streaming: Real-time audio streaming and transcription results
• Sherpa-ONNX Integration: Real sherpa-onnx with Estonian-English streaming zipformer model
• Whisper Support: OpenAI Whisper Large-v3 and Turbo models for offline transcription
• Parakeet EOU Support: NVIDIA Parakeet Realtime EOU 120M for low-latency English streaming
• FastConformer Support: NVIDIA FastConformer streaming models (80ms/1040ms latency)
• Parakeet TDT Support: NVIDIA Parakeet TDT 0.6B-v3 model for 25 European languages
• Automatic Punctuation: CT-Transformer punctuation model adds commas, periods, and capitalization
• Multi-Language: Estonian, English, Russian, Ukrainian, and 21 more European languages
• Multi-Format Audio: Support for Opus, FLAC, WAV, MP3, and AAC audio formats
• Safe & Fast: Built with Rust for memory safety and performance
• Production Ready: Powers the ASR backend at tekstiks.ee

Quick Start

Prerequisites

1. Install sherpa-onnx CLI tools (required for Parakeet TDT model):

   # Download latest sherpa-onnx
   wget https://github.com/k2-fsa/sherpa-onnx/releases/download/v1.12.14/sherpa-onnx-v1.12.14-linux-x64-static.tar.bz2
   tar xf sherpa-onnx-v1.12.14-linux-x64-static.tar.bz2
   
   # Install to user bin (no sudo required)
   cp sherpa-onnx-v1.12.14-linux-x64-static/bin/sherpa-onnx-offline ~/.cargo/bin/
   
   # Or install system-wide (requires sudo)
   sudo cp sherpa-onnx-v1.12.14-linux-x64-static/bin/sherpa-onnx-offline /usr/local/bin/
   

   Note: The Parakeet TDT model (25 European languages) requires the sherpa-onnx-offline CLI binary. Estonian and English streaming models work without this.

Running the Server

1. Start the server:

   cargo run
   

2. Connect via WebSocket: The server runs on ws://127.0.0.1:8080/ws

Offline Transcription (Whisper)

For batch transcription of audio files using Whisper models:

1. Convert audio to WAV format (if needed):

   ffmpeg -i audio.opus -ar 16000 -ac 1 audio.wav
   

2. Run transcription:

   # Ensure library path is set
   export LD_LIBRARY_PATH="/path/to/sherpa/lib:$LD_LIBRARY_PATH"
   
   # Run opus transcription binary
   cargo run --bin transcribe_opus -- input.opus
   

WebSocket Protocol

Client Messages

1. Start Session:

   {
     "type": "start",
     "sample_rate": 16000,
     "format": "pcm",
     "language": "en"
   }
   

   Available language values:

   • en - English (Parakeet EOU 120M, low-latency streaming)
   • et - Estonian/English (Zipformer bilingual)
   • fastconformer_en_1040ms - English (FastConformer 1040ms latency, high accuracy)
   • parakeet_tdt_v3 - Auto-detect (25 European languages)

2. Send Audio:

   {
     "type": "audio",
     "data": "base64-encoded-audio-data"
   }
   

3. Stop Session:

   {
     "type": "stop"
   }
   

4. Ping:

   {
     "type": "ping"
   }
   

Server Messages

1. Ready:

   {
     "type": "ready",
     "session_id": "uuid-here"
   }
   

2. Transcript:

   {
     "type": "transcript",
     "session_id": "uuid-here",
     "text": "transcribed text",
     "is_final": false,
     "confidence": 0.95
   }
   

3. Error:

   {
     "type": "error",
     "message": "error description"
   }
   

Real-Time Speaker Diarization

Optional per-session speaker diarization with known-speaker identification, compatible with offline-asr rosters. Enable it in config.toml ([diarization] enabled = true) and opt in per session:

{ "type": "start", "language": "en", "diarize": true }

Audio is VAD-segmented, each speech window is embedded with a ReDimNet2 ONNX model, and clusters are re-computed every few seconds — so speaker tags are revised retroactively as more audio arrives. Cluster centroids are matched against a roster.json produced by offline-asr enroll to auto-suggest names.

Additional server messages when diarization is active:

1. Speaker segment (provisional, sent as speech is detected):

   {
     "type": "speaker_segment",
     "session_id": "uuid",
     "segment_id": 4,
     "start_time": 7.3,
     "end_time": 8.9,
     "cluster": "S2",
     "name": "mari",
     "name_confidence": 0.84
   }
   

   start_time/end_time are seconds relative to the start of the session's audio. name/name_confidence are omitted when no roster speaker matches.

2. Speakers updated (whenever re-clustering changes any tag or name; a full snapshot — the client should replace its segment→speaker mapping):

   {
     "type": "speakers_updated",
     "session_id": "uuid",
     "revision": 3,
     "segments": [ { "segment_id": 0, "cluster": "S1" }, ... ],
     "clusters": [ { "cluster": "S1", "name": "mari", "name_confidence": 0.84 },
                   { "cluster": "S2" } ]
   }
   

   revision increases strictly; apply snapshots in order. A final update is always sent during session finalization.

transcript messages additionally carry audio_time (seconds of audio ingested when the result was emitted) so the UI can align transcript text with speaker segments.

Setup:

1. Export the embedding model (one-time, in offline-asr): python scripts/export_redimnet2_onnx.py -o <this repo>/models/speaker/redimnet2-b6-vb2-vox2-lm.onnx The script verifies torch↔ONNX equivalence (cosine > 0.999), so embeddings and rosters are interchangeable between the two systems. The roster's model_version must match [diarization] model_version exactly.
2. Optionally enroll known speakers with offline-asr enroll and point [speaker_id] roster_path at the resulting roster.json.
3. Calibrate the clustering threshold on your own recordings if needed: cargo run --release --bin diarize_wav -- meeting.wav --sweep (measured on 16 kHz test data: same-speaker cosine distance ≈ 0.1–0.2, cross-speaker ≈ 0.5; the default cut is 0.4).

Model Setup

IMPORTANT: ASR models are NOT included in this repository and must be downloaded separately.

Quick Setup

1. Download TTS models (included in Git LFS):

   git lfs install
   git lfs pull
   

2. Download ASR models (must be obtained separately):

   The following models are required but NOT included in the repository:

   Estonian-English Streaming Model

   # Download Estonian-English zipformer model
   # This model must be obtained from TalTechNLP or other sources
   # Place files in: models/streaming-zipformer-large.et-en/
   # Required files: encoder.onnx, decoder.onnx, joiner.onnx, tokens.txt
   

   English Streaming Model

   # Download English zipformer model from sherpa-onnx releases
   wget https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-streaming-zipformer-en-2023-06-26.tar.bz2
   tar xvf sherpa-onnx-streaming-zipformer-en-2023-06-26.tar.bz2
   mv sherpa-onnx-streaming-zipformer-en-2023-06-26 models/streaming-zipformer-en-2023-06-26
   

   Parakeet TDT v3 (25 European Languages)

   # Download Parakeet TDT model
   wget https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8.tar.bz2
   tar xvf sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8.tar.bz2
   mv sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8 models/
   

Included Models (Git LFS)

The following models ARE included via Git LFS:

• TTS Model: Estonian TTS (models/tts/vits-coqui-et-cv/model.onnx)
• VAD Model: Silero VAD (models/silero_vad.onnx) - currently disabled

Model Directory Structure

After setup, your models directory should look like:

models/
├── streaming-zipformer-large.et-en/          # Estonian-English (obtain separately)
│   ├── encoder.onnx
│   ├── decoder.onnx
│   ├── joiner.onnx
│   └── tokens.txt
├── parakeet-realtime-eou-120m-v1-onnx/       # Parakeet EOU (English streaming)
│   └── ...
├── fastconformer-en-80ms-transducer/         # FastConformer 80ms (with punctuation)
│   ├── encoder.onnx
│   ├── decoder.onnx
│   ├── joiner.onnx
│   └── tokens.txt
├── fastconformer-en-1040ms-transducer/       # FastConformer 1040ms (high accuracy)
│   ├── encoder.onnx
│   ├── decoder.onnx
│   ├── joiner.onnx
│   └── tokens.txt
├── sherpa-onnx-punct-ct-transformer-zh-en-vocab272727-2024-04-12/  # Punctuation model
│   └── model.onnx
├── sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8/  # Parakeet TDT (25 languages)
│   ├── encoder.int8.onnx
│   ├── decoder.int8.onnx
│   ├── joiner.int8.onnx
│   └── tokens.txt
├── tts/
│   └── vits-coqui-et-cv/                     # Estonian TTS (Git LFS)
│       ├── model.onnx
│       └── tokens.txt
└── silero_vad.onnx                           # VAD (Git LFS)

Whisper Models (Offline Recognition)

For offline transcription with OpenAI Whisper models:

# Download Whisper Large-v3 model
wget https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-whisper-large-v3.tar.bz2
tar xvf sherpa-onnx-whisper-large-v3.tar.bz2
mv sherpa-onnx-whisper-large-v3 models/whisper-large-v3

# Download Whisper Turbo model
wget https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-whisper-turbo.tar.bz2
tar xvf sherpa-onnx-whisper-turbo.tar.bz2
mv sherpa-onnx-whisper-turbo models/whisper-turbo

Note: Whisper models in sherpa-onnx process only the first 30 seconds of audio due to architectural limitations.

Punctuation Model

The server supports automatic punctuation restoration using a CT-Transformer model. This adds commas, periods, question marks, and proper capitalization to ASR output.

# Download CT-Transformer punctuation model (bilingual EN/ZH, ~294MB)
wget https://github.com/k2-fsa/sherpa-onnx/releases/download/punctuation-models/sherpa-onnx-punct-ct-transformer-zh-en-vocab272727-2024-04-12.tar.bz2
tar xvf sherpa-onnx-punct-ct-transformer-zh-en-vocab272727-2024-04-12.tar.bz2
mv sherpa-onnx-punct-ct-transformer-zh-en-vocab272727-2024-04-12 models/

Note: The English-only model (sherpa-onnx-online-punct-en-2024-08-06) uses a different architecture (CNN-BiLSTM) that is not compatible with the current sherpa-rs bindings. Use the CT-Transformer model instead.

Transducer Models (RNN-T)

Transducer models (also known as RNN-T or RNNT) are streaming ASR models that use an encoder-decoder-joiner architecture. They provide real-time transcription with good accuracy.

Model Types

The server supports two transducer model types:

1. Transducer - Standard sherpa-onnx transducer models (zipformer, FastConformer)
2. NemoTransducer - NVIDIA NeMo transducer models (Parakeet TDT) - requires CLI subprocess

Configuration

Transducer models require three ONNX files: encoder, decoder, and joiner. Configure them in config.toml:

[asr.models.my_transducer_model]
model_path = "./models/my-model/encoder.onnx"      # Primary path (encoder)
tokens_path = "./models/my-model/tokens.txt"
encoder_path = "./models/my-model/encoder.onnx"    # Explicit encoder path
decoder_path = "./models/my-model/decoder.onnx"    # RNN decoder
joiner_path = "./models/my-model/joiner.onnx"      # Joiner network
name = "My Transducer Model"
language_code = "en"
model_type = "Transducer"
sample_rate = 16000
punctuation = true  # Enable punctuation post-processing

Available Transducer Models

Estonian-English Zipformer (model_type = "Transducer"):

[asr.models.et]
model_path = "./models/streaming-zipformer-large.et-en/encoder.onnx"
tokens_path = "./models/streaming-zipformer-large.et-en/tokens.txt"
name = "Estonian/English Zipformer"
language_code = "et"
model_type = "Transducer"

FastConformer 1040ms (model_type = "Transducer"):

• High accuracy, 1040ms latency
• No built-in punctuation - enable punctuation = true for post-processing

[asr.models.fastconformer_transducer_en_1040ms]
model_path = "./models/fastconformer-en-1040ms-transducer/encoder.onnx"
tokens_path = "./models/fastconformer-en-1040ms-transducer/tokens.txt"
encoder_path = "./models/fastconformer-en-1040ms-transducer/encoder.onnx"
decoder_path = "./models/fastconformer-en-1040ms-transducer/decoder.onnx"
joiner_path = "./models/fastconformer-en-1040ms-transducer/joiner.onnx"
name = "FastConformer English Streaming 1040ms (RNNT)"
language_code = "en"
model_type = "Transducer"
sample_rate = 16000
punctuation = true

FastConformer 80ms with Punctuation (model_type = "Transducer"):

• Low latency (80ms)
• Has built-in punctuation & capitalization (_pc variant)

[asr.models.fastconformer_transducer_en_80ms]
model_path = "./models/fastconformer-en-80ms-transducer/encoder.onnx"
tokens_path = "./models/fastconformer-en-80ms-transducer/tokens.txt"
encoder_path = "./models/fastconformer-en-80ms-transducer/encoder.onnx"
decoder_path = "./models/fastconformer-en-80ms-transducer/decoder.onnx"
joiner_path = "./models/fastconformer-en-80ms-transducer/joiner.onnx"
name = "FastConformer English Streaming 80ms (RNNT)"
language_code = "en"
model_type = "Transducer"
sample_rate = 16000
punctuation = false  # Model has native punctuation

Parakeet TDT v3 (model_type = "NemoTransducer"):

• 25 European languages with auto-detection
• Requires sherpa-onnx-offline CLI binary (see Prerequisites)
• Uses offline/pseudo-streaming mode

[asr.models.parakeet_tdt_v3]
model_path = "./models/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8/encoder.int8.onnx"
tokens_path = "./models/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8/tokens.txt"
encoder_path = "./models/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8/encoder.int8.onnx"
decoder_path = "./models/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8/decoder.int8.onnx"
joiner_path = "./models/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8/joiner.int8.onnx"
name = "Parakeet TDT v3 (25 European languages)"
language_code = "auto"
model_type = "NemoTransducer"
sample_rate = 16000

Converting NeMo Models to ONNX

To convert NVIDIA NeMo models to sherpa-onnx format:

# Install dependencies
pip install nemo_toolkit[asr] onnx

# Use the export script (for FastConformer transducer)
python scripts/export_fastconformer_rnnt.py

# Or use sherpa-onnx's official conversion scripts
# See: https://github.com/k2-fsa/sherpa-onnx/blob/master/scripts/nemo/

The conversion produces three files:

• encoder.onnx - Audio feature encoder
• decoder.onnx - RNN decoder (prediction network)
• joiner.onnx - Joint network combining encoder and decoder outputs

FastConformer Models

NVIDIA FastConformer streaming models with different latency configurations:

# The 80ms model with punctuation is included in models/fastconformer-en-80ms-transducer/
# (converted from nvidia/stt_en_fastconformer_hybrid_medium_streaming_80ms_pc)

# The 1040ms model (higher accuracy, no built-in punctuation) is included in:
# models/fastconformer-en-1040ms-transducer/
# (converted from nvidia/stt_en_fastconformer_hybrid_large_streaming_1040ms)

Note: The 1040ms model doesn't have built-in punctuation support (NVIDIA doesn't offer a _pc variant for 1040ms). Enable the punctuation post-processor in config to add punctuation.

Parakeet EOU Model

NVIDIA Parakeet Realtime EOU 120M for low-latency English streaming:

# Download from HuggingFace or use the included model
# Place in: models/parakeet-realtime-eou-120m-v1-onnx/

Configuration

Create a config.toml file or set ASR_CONFIG environment variable:

bind_address = "127.0.0.1:8081"

[asr]
default_language = "en"
provider = "cpu"
num_threads = 4
sample_rate = 16000

# Punctuation post-processing (adds commas, periods, capitalization)
[asr.punctuation]
enabled = true
model_path = "./models/sherpa-onnx-punct-ct-transformer-zh-en-vocab272727-2024-04-12/model.onnx"
num_threads = 2

# Pseudo-streaming for offline models (Whisper, NeMo)
[asr.pseudo_streaming]
enabled = true
chunk_duration_seconds = 1.6
total_buffer_seconds = 4.0

# Estonian/English streaming model
[asr.models.et]
model_path = "./models/streaming-zipformer-large.et-en/encoder.onnx"
tokens_path = "./models/streaming-zipformer-large.et-en/tokens.txt"
name = "Estonian/English Zipformer"
language_code = "et"
model_type = "Transducer"

# English Parakeet EOU (low-latency streaming)
[asr.models.en]
model_path = "./models/parakeet-realtime-eou-120m-v1-onnx"
name = "Parakeet Realtime EOU 120M v1 (English)"
language_code = "en"
model_type = "ParakeetEOU"
sample_rate = 16000

# English FastConformer 1040ms (high accuracy streaming)
[asr.models.fastconformer_en_1040ms]
model_path = "./models/fastconformer-en-1040ms-transducer/encoder.onnx"
tokens_path = "./models/fastconformer-en-1040ms-transducer/tokens.txt"
name = "FastConformer English Streaming 1040ms"
language_code = "en"
model_type = "Transducer"
sample_rate = 16000

[audio]
buffer_size = 8192
max_chunk_size = 4096
silence_threshold = 0.01
min_speech_duration_ms = 500

[processing]
max_concurrent_sessions = 100
processing_threads = 4
session_timeout_seconds = 300

Project Structure

src/
├── main.rs              # Axum server setup and routing
├── lib.rs               # Library exports
├── config.rs            # Configuration management (models, punctuation, pseudo-streaming)
├── asr.rs               # ASR service with thread pool and punctuation integration
├── parakeet_backend.rs  # Parakeet EOU model backend
└── websocket.rs         # WebSocket message handling

Sherpa-ONNX Integration

Real sherpa-onnx is now integrated!

The server uses:

• Model: Estonian-English streaming zipformer from TalTechNLP
• Dependency: Git version of sherpa-rs (bypasses checksum issues)
• Architecture: Transducer model with encoder, decoder, and joiner
• Languages: Estonian and English speech recognition

Troubleshooting sherpa-rs

If you encounter checksum errors with the published sherpa-rs crate:

# Use git dependency instead of crates.io version
sherpa-rs = { git = "https://github.com/thewh1teagle/sherpa-rs.git", features = ["download-binaries"] }

Library Linking Issues (sherpa-onnx)

If you encounter runtime errors like libsherpa-onnx-c-api.so => not found or fatal runtime error: Rust cannot catch foreign exceptions, this indicates that the sherpa-onnx C library is not found at runtime.

Solution:

1. Locate the sherpa-onnx library: Find where the library was installed. Common locations:

   # Check if you have a working installation (e.g., from desktop-asr project)
   find /home/$USER -name "libsherpa-onnx-c-api.so" 2>/dev/null
   

2. Set library path before running:

   export LD_LIBRARY_PATH="/path/to/sherpa/lib:$LD_LIBRARY_PATH"
   cargo run
   

3. For permanent solution, copy to system location:

   sudo cp /path/to/libsherpa-onnx-c-api.so /usr/local/lib/
   sudo ldconfig
   

4. Example with working desktop-asr setup:

   export LD_LIBRARY_PATH="/home/aivo/dev2/desktop-asr/dist/lib:$LD_LIBRARY_PATH"
   cargo run
   

Note: This issue typically occurs after a cargo clean rebuild or when sherpa-rs downloads/builds new binaries but doesn't place them in the system library path.

Development

• cargo check - Type check
• cargo test - Run tests
• cargo run - Start server
• RUST_LOG=debug cargo run - Debug logging

Testing

1. Start the server: cargo run
2. Open the test client: Open client_test.html in your browser
3. Test the pipeline:
   • Click "Connect" to establish WebSocket connection
   • Click "Start Session" to begin ASR session
   • Click "Send Test Audio Data" to test transcription
   • Watch real-time results in the output panel

Performance Notes

• Concurrent Connections: Handles thousands of WebSocket connections
• CPU Utilization: Dedicated thread pool prevents blocking async runtime
• Memory Efficient: Rust's zero-cost abstractions and careful memory management
• Model Size: ~720MB Estonian-English model with high accuracy

License

MIT License