ASR EvalKit

A modular, extensible toolkit for evaluating Automatic Speech Recognition (ASR) models. It provides a unified Python API and a Command-Line Interface (CLI) to evaluate HuggingFace, NeMo, and vLLM-backed models across various datasets.

Key Features

• Unified Provider System: Evaluate transformers, NVIDIA NeMo, or vLLM models seamlessly.
• Dataset Support: Native support for HuggingFace datasets (with streaming) and local NeMo manifests.
• Extensible: Effortlessly add new models, metrics, or normalisers via clean abstract base classes.

---

Installation

1. Install uv (recommended)

uv is a fast Python package manager. Install it with:

curl -LsSf https://astral.sh/uv/install.sh | sh

2. Clone and create a virtual environment

git clone https://github.com/knoveleng/asr-evalkit.git
cd asr-evalkit

# Create a virtual environment named 'asr'
uv venv asr --python 3.12

# Activate it
source asr/bin/activate

3. Install dependencies

# Install the package in editable mode (reads from requirements.txt via setup.py)
uv pip install -e .

# Optional — install NeMo and dev tools
uv pip install -e ".[nemo,dev]"

Note: For vLLM-backed models (seallm_audio, qwen3_asr), install vLLM separately:

uv pip install vllm

---

Quick Start: Command-Line Interface (CLI)

The easiest way to run evaluations is via the asr-evalkit command.

1. Evaluate a HuggingFace Model on a HuggingFace Dataset

asr-evalkit \
  --evaluator whisper \
  --model openai/whisper-large-v3-turbo \
  --dataset openslr/librispeech_asr \
  --dataset-config clean \
  --dataset-split test \
  --streaming \
  --audio-column audio \
  --text-column text \
  --output-file results.json

2. Evaluate a NeMo Model on a HuggingFace Dataset

asr-evalkit \
  --evaluator canary \
  --model nvidia/canary-1b \
  --dataset openslr/librispeech_asr \
  --dataset-config clean \
  --dataset-split test \
  --streaming \
  --audio-column audio \
  --text-column text \
  --output-file canary_results.json

3. Evaluate a NeMo Model on a Local Manifest

asr-evalkit \
  --evaluator nemotron_asr \
  --model nvidia/nemotron-3.5-asr-streaming-0.6b \
  --dataset path/to/manifest.jsonl \
  --dataset-source nemo \
  --language en \
  --max-samples 100 \
  --output-file nemotron_results.json

---

Example Scripts

The scripts/ directory contains ready-to-use examples you can copy and adapt:

Running the batch benchmark script:

# Run with defaults (10 samples per dataset, results → results/benchmarks/)
bash scripts/evaluate_benchmarks.sh

# Override sample count and output directory
MAX_SAMPLES=100 OUTPUT_DIR=results/full bash scripts/evaluate_benchmarks.sh

Tip: evaluate_benchmarks.sh is designed to be your personal run script — it is listed in .gitignore so you can freely customise it (add models, change datasets, tweak flags) without affecting the repo.

---

Quick Start: Python API

The Python API offers the exact same capabilities as the CLI but allows integration into your own scripts.

1. HuggingFace Model & Dataset

from asr_evalkit.evaluators.providers import HuggingFaceProvider
from asr_evalkit.config import ModelConfig, NormalizerConfig
from asr_evalkit.datasets import HuggingFaceDataset
from asr_evalkit.runner import Runner

# Configure Provider (Handles weights + precision)
provider = HuggingFaceProvider(
    model_id="openai/whisper-large-v3-turbo",
    device="cuda",
    use_fp16=True
)

# Set up Runner
model_cfg = ModelConfig(evaluator="whisper", provider=provider)
runner = Runner(model=model_cfg)

# Configure Dataset
hf_dataset = HuggingFaceDataset(
    "openslr/librispeech_asr",
    config="clean",
    split="test",
    audio_column="audio",
    text_column="text",
    streaming=True,
    max_samples=10
)

# Run Evaluation
# Note: Default behaviour (lowercase + Unicode NFC + remove punctuation + remove extra whitespace)
results = runner.run(dataset=hf_dataset, normalizer=NormalizerConfig())

print(f"WER: {results['wer']*100:.2f}%")

2. NeMo Model on a Local Manifest

from asr_evalkit.evaluators.providers import NeMoProvider
from asr_evalkit.config import ModelConfig
from asr_evalkit.datasets import NeMoDataset
from asr_evalkit.runner import Runner

# Use NeMo provider instead
provider = NeMoProvider(model_id="nvidia/parakeet-tdt-0.6b-v3", device="cuda")
runner = Runner(model=ModelConfig(evaluator="parakeet", provider=provider))

# Load a local NeMo dataset manifest
nemo_dataset = NeMoDataset(
    "path/to/test_manifest.jsonl",
    audio_column="audio_filepath",
    text_column="text",
    max_samples=10
)

results = runner.run(dataset=nemo_dataset)

---

Supported Evaluators & Providers

To see a list of all currently supported models natively built into the package, run:

asr-evalkit --list-evaluators

---

Benchmark Results

Architecture & Customisation

ASR EvalKit is structured around a strict provider/evaluator separation to maintain clean code and easy API mapping.

If you need to customise the toolkit, please refer to the detailed guides in the docs/ folder:

• Architecture Overview — Understand how Providers, Evaluators, and Runners interact.
• Adding a Custom Evaluator — Step-by-step guide to adding support for a new model architecture.
• Adding a Custom Provider — How to support a new backend (e.g. ONNX Runtime, TensorRT-LLM).
• Dataset Configurations — Details on HF streaming vs. NeMo manifests.
• Metrics & Normalisers — How to write custom scoring metrics or ground-truth normalisers.
• CLI Reference — Full command-line options.
• Troubleshooting — Common runtime errors and fixes (vLLM, CUDA, torch, peft).

---

⚠️ Known Issues

AssertionError: duplicate template name when running outside a virtual environment

If you see this error when using vLLM-backed models (e.g. seallm_audio, qwen3_asr) with the system Python, your torch installation has stale kernel files from a previous upgrade. See Troubleshooting → duplicate template name for the one-line fix.

---

Citation

If you use ASR EvalKit in your research, please cite:

@software{dang2026asrevalkit,
  author       = {Quy-Anh Dang},
  title        = {ASR EvalKit: A Modular Toolkit for Evaluating Automatic Speech Recognition Models},
  year         = {2026},
  url          = {https://github.com/knoveleng/asr-evalkit},
}