ASR Consilium

A repository for Automatic Speech Recognition (ASR) that ensembles multiple open-source models to achieve SOTA quality of recognition. Useful if you need to get the maximum quality of recognition despite the computational time.

Usage

pip install asr-consilium 

or copy folder asr_consilium into your project. After:

from asr_consilium import inference as asr_inference

jsonl_dataset = 'markdown.jsonl' # Dataset in jsonl format
out_file = 'result.jsonl' # File where results will be stored in JSONL format
asr_inference(
    jsonl_file=jsonl_dataset, # input dataset for processing
    out_file=out_file, # Outputs file
    batch_size=16, # Batch size
    model_list=None, # Models for ensemble - see usage below
    weights=None, # Weights for models
    language='English', # Language
    normalize=True, # Normalize texts before ensemble
    char_level=False, # ensemble on char or word level
    ensemble_type='median_extended', # Type of ensemble ('greedy', 'median', 'median_extended')
    skip_existed=True, # if some model already calculated the results they will not recalculate
)

Or with command line:

python3 inference.py --input_data "samples/markdown.jsonl" --output "result.jsonl" --batch_size 16

Supported models

• The Cohere Transcribe (03-2026) model requires a newer version of the transformers library (>= 5.4), as well as granted access to the model itself.

Requirements

• Python >= 3.12 (primarily due to compatibility limitations with the nemo_toolkit[asr,tts] package; all other components are compatible with Python 3.10).

Format of markdown.jsonl

markdown.jsonl must contain 2 fields:

• 'audio' - name of audio file in wav format and 16000 Hz sample rate. It must be in the same folder with markdown.jsonl file.
• 'duration' - duration of audio in seconds (needed for sorting and for tokens approximation)
• 'text' (optional) - needed for validation. If you have real speech text. Code will automatically calculate WER and CER metrics for your data.

...
{"audio": "483_0.wav", "duration": 2.079875}
{"audio": "461_1.wav", "duration": 9.6599375}
{"audio": "243_2.wav", "duration": 7.3400625}
...

You can find sample here

Choose models for ensembling

List of models and their weights are chosen with parameters model_list and weights. You may leave it None. So default list of models and default weights will be used.

Default models and weights

model_list = [
    'nvidia/parakeet-tdt-0.6b-v2',
    'nvidia/parakeet-tdt-0.6b-v3',
    'Qwen/Qwen3-ASR-1.7B',
    'nvidia/canary-qwen-2.5b',
    'ibm-granite/granite-speech-3.3-8b',
    'ibm-granite/granite-4.0-1b-speech',
    'ibm-granite/granite-speech-4.1-2b',
    'ZFTurbo/Phi-4-multimodal-instruct',
]

weights = [4.5, 4.2, 8.4, 9.8, 8.7, 3.5, 8.9, 9.4]

Results for different datasets

Note: WER/CER metric calculation slightly different from Open ASR Leaderboard

WER table (English language)

CER Table (English language)

• Tables for the French, German, Spanish, Italian, Portuguese, and Russian languages are available in a separate document.