x-benchmark-tests

A collection of benchmarking scripts and experimental results for evaluating state-of-the-art Speech-to-Text (STT) models. This repository contains the raw tests used to determine production-grade performance across quality, speed, and concurrency.

Models Evaluated

• Whisper Turbo (faster-whisper)
• Qwen3-ASR-0.6B (Transformers & vLLM)
• SenseVoice Small
• Voxtral 4B
• Moonshine Medium (ONNX CPU)

Benchmark Categories

• Real-time Factor (RTFx): Measure processing speed vs. audio duration.
• Concurrency: Testing how many streams a single L4 GPU can handle before quality or latency degrades.
• TTFT (Time to First Token): Measuring latency from audio arrival to first transcript fragment.
• Multilingual Performance: Special focus on Hindi and English accuracy.
• Multi-worker Scaling: Evaluating throughput gains from running multiple model instances on the same GPU.

Key Results

• multiworker_results.txt: Detailed logs of multi-worker throughput tests.
• production_results.txt: Final verification of the chosen production configurations.

Usage

Most scripts are designed to be run against a set of reference audio files (e.g., hard_16k.wav, hindi_16k.wav).

Example:

python bench_whisper_cpu_pinned.py
python bench_qwen3_production.py

Hardware Note

All benchmarks were performed on NVIDIA L4 GPUs (AWS g6 instances) unless otherwise specified.