Speech-to-Text Benchmark

Made in Vancouver, Canada by Picovoice

This repo is a minimalist and extensible framework for benchmarking different speech-to-text engines.

Table of Contents

• Data
• Metrics
• Engines
• Usage
• Results

Data

• LibriSpeech
• TED-LIUM
• Common Voice
• Multilingual LibriSpeech
• VoxPopuli
• Fleurs (Download instructions)
• JSUT-BASIC (Download instructions)
• Zeroth Korean
• Pansori-TEDxKR

Metrics

Word Error Rate

Word error rate (WER) is the ratio of edit distance between words in a reference transcript and the words in the output of the speech-to-text engine to the number of words in the reference transcript.

Character Error Rate

Character error rate (CER) is the ratio of edit distance between characters in a reference transcript and the characters in the output of the speech-to-text engine to the number of characters in the reference transcript. CER is reported for Korean and Japanese following industry standards.

Punctuation Error Rate

Punctuation Error Rate (PER) is the ratio of punctuation-specific errors between a reference transcript and the output of a speech-to-text engine to the number of punctuation-related operations in the reference transcript (more details in Section 3 of Meister et al.). We report PER results for periods (.) and question marks (?).

Core-Hour

The Core-Hour metric is used to evaluate the computational efficiency of the speech-to-text engine, indicating the number of CPU hours required to process one hundred hours of audio. A speech-to-text engine with lower Core-Hour is more computationally efficient. We omit this metric for cloud-based engines.

Word Emission Latency

Word emission latency is used to evaluate the responsiveness of streaming speech-to-text engines. It measures the average delay from the point a word has finished being spoken to when its transcription is emitted by the engine. We measure this metric only for streaming engines.

Model Size

The aggregate size of models (acoustic and language), in MB. We omit this metric for cloud-based engines.

Engines

• Amazon Transcribe
• Azure Speech-to-Text
• Google Speech-to-Text
• IBM Watson Speech-to-Text
• Moonshine
• Nemotron 3.5 ASR Streaming
• OpenAI Whisper
• Vosk
• Whisper.cpp
• Picovoice Cheetah
• Picovoice Leopard

Usage

This benchmark has been developed and tested on Ubuntu 22.04 with Python 3.12.

• Install FFmpeg
• Download datasets.
• Install the requirements:

pip3 install -r requirements.txt

Benchmark Usage

In the following, we provide instructions for running the benchmark for each engine. The supported datasets are: COMMON_VOICE, LIBRI_SPEECH_TEST_CLEAN, LIBRI_SPEECH_TEST_OTHER, TED_LIUM, MLS, VOX_POPULI, FLEURS, ZEROTH_KOREAN, PANSORI and JSUT_BASIC. The supported languages are: EN, FR, DE, ES, IT, PT-BR, PT-PT, KO and JA.

To evaluate Punctuation Error Rate, use the --punctuation flag. Use --punctuation-set ${PUNCTUATION_SET} to select which punctuation marks to calculate PER against, where ${PUNCTUATION_SET} is one or more of ., ? and , (default .?).

Amazon Transcribe Instructions

Replace ${DATASET} with one of the supported datasets, ${DATASET_FOLDER} with the path to the dataset, ${LANGUAGE} with the target language, ${AWS_LOCATION} with the name of the AWS server and ${AWS_PROFILE} with the name of the AWS profile you wish to use.

python3 benchmark.py \
--dataset ${DATASET} \
--dataset-folder ${DATASET_FOLDER} \
--language ${LANGUAGE} \
--engine AMAZON_TRANSCRIBE \
--aws-profile ${AWS_PROFILE} \
--aws-location ${AWS_LOCATION}

Set --engine to AMAZON_TRANSCRIBE_STREAMING to use Amazon Transcribe in streaming mode.

Azure Speech-to-Text Instructions

Replace ${DATASET} with one of the supported datasets, ${DATASET_FOLDER} with the path to the dataset, ${LANGUAGE} with the target language, ${AZURE_SPEECH_KEY} and ${AZURE_SPEECH_LOCATION} information from your Azure account.

python3 benchmark.py \
--dataset ${DATASET} \
--dataset-folder ${DATASET_FOLDER} \
--language ${LANGUAGE} \
--engine AZURE_SPEECH_TO_TEXT \
--azure-speech-key ${AZURE_SPEECH_KEY} \
--azure-speech-location ${AZURE_SPEECH_LOCATION}

Set --engine to AZURE_SPEECH_TO_TEXT_REAL_TIME to use Azure Speech-to-text in streaming mode.

Google Speech-to-Text Instructions

Replace ${DATASET} with one of the supported datasets, ${DATASET_FOLDER} with the path to the dataset, ${LANGUAGE} with the target language and ${GOOGLE_APPLICATION_CREDENTIALS} with credentials download from Google Cloud Platform.

python3 benchmark.py \
--dataset ${DATASET} \
--dataset-folder ${DATASET_FOLDER} \
--language ${LANGUAGE} \
--engine GOOGLE_SPEECH_TO_TEXT \
--google-application-credentials ${GOOGLE_APPLICATION_CREDENTIALS}

Set --engine to GOOGLE_SPEECH_TO_TEXT_STREAMING to use Google Speech-to-text in streaming mode.

IBM Watson Speech-to-Text Instructions

Replace ${DATASET} with one of the supported datasets, ${DATASET_FOLDER} with the path to the dataset and ${WATSON_SPEECH_TO_TEXT_API_KEY}/${${WATSON_SPEECH_TO_TEXT_URL}} with credentials from your IBM account. This engine only supports English.

python3 benchmark.py \
--dataset ${DATASET} \
--dataset-folder ${DATASET_FOLDER} \
--language EN \
--engine IBM_WATSON_SPEECH_TO_TEXT \
--watson-speech-to-text-api-key ${WATSON_SPEECH_TO_TEXT_API_KEY} \
--watson-speech-to-text-url ${WATSON_SPEECH_TO_TEXT_URL}

OpenAI Whisper Instructions

Replace ${DATASET} with one of the supported datasets, ${DATASET_FOLDER} with the path to the dataset, ${LANGUAGE} with the target language and ${WHISPER_MODEL} with the whisper model type (WHISPER_TINY, WHISPER_BASE, WHISPER_SMALL, WHISPER_MEDIUM, WHISPER_LARGE, WHISPER_LARGE_V2, WHISPER_LARGE_V3 or WHISPER_LARGE_TURBO)

python3 benchmark.py \
--engine ${WHISPER_MODEL} \
--dataset ${DATASET} \
--language ${LANGUAGE} \
--dataset-folder ${DATASET_FOLDER} \

Whisper.cpp Streaming Instructions

Replace ${DATASET} with one of the supported datasets, ${DATASET_FOLDER} with the path to the dataset, ${LANGUAGE} with the target language, and ${WHISPER_CPP_MODEL} with the Whisper.cpp streaming model type (WHISPER_CPP_STREAMING_TINY, WHISPER_CPP_STREAMING_BASE, WHISPER_CPP_STREAMING_SMALL, WHISPER_CPP_STREAMING_MEDIUM, WHISPER_CPP_STREAMING_LARGE_V3 or WHISPER_CPP_STREAMING_LARGE_TURBO).

python3 benchmark.py \
--engine ${WHISPER_CPP_MODEL} \
--dataset ${DATASET} \
--dataset-folder ${DATASET_FOLDER} \
--language ${LANGUAGE} \

Moonshine Streaming Instructions

Replace ${DATASET} with one of the supported datasets, ${DATASET_FOLDER} with the path to the dataset and ${MOONSHINE_MODEL} with the Moonshine streaming model type (MOONSHINE_STREAMING_TINY, MOONSHINE_STREAMING_SMALL or MOONSHINE_STREAMING_MEDIUM). This engine only supports English.

python3 benchmark.py \
--engine ${MOONSHINE_MODEL} \
--dataset ${DATASET} \
--dataset-folder ${DATASET_FOLDER} \
--language EN \

Vosk Streaming Instructions

Replace ${DATASET} with one of the supported datasets, ${DATASET_FOLDER} with the path to the dataset and ${VOSK_MODEL} with the Vosk streaming model type (VOSK_STREAMING_SMALL or VOSK_STREAMING_LARGE). This engine only supports English.

python3 benchmark.py \
--engine ${VOSK_MODEL} \
--dataset ${DATASET} \
--dataset-folder ${DATASET_FOLDER} \
--language EN

Nemotron 3.5 ASR Streaming Instructions

Replace ${DATASET} with one of the supported datasets, ${DATASET_FOLDER} with the path to the dataset, ${LANGUAGE} with the target language, and ${CHUNK_SIZE_MS} with a supported chunk size: 80, 160, 320, 560 or 1120 (default 560).

python3 benchmark.py \
--engine NEMOTRON_3_5_ASR_STREAMING \
--dataset ${DATASET} \
--dataset-folder ${DATASET_FOLDER} \
--language ${LANGUAGE} \
--streaming-chunk-size-ms ${CHUNK_SIZE_MS}

Picovoice Cheetah Instructions

Replace ${DATASET} with one of the supported datasets, ${DATASET_FOLDER} with the path to the dataset, ${LANGUAGE} with the target language, and ${PICOVOICE_ACCESS_KEY} with AccessKey obtained from Picovoice Console. By default, the Cheetah English model is used. For non-English languages models replace ${PICOVOICE_MODEL_PATH} with the path to a model file acquired from the Cheetah Github Repo.

python3 benchmark.py \
--engine PICOVOICE_CHEETAH \
--dataset ${DATASET} \
--dataset-folder ${DATASET_FOLDER} \
--language ${LANGUAGE} \
--picovoice-access-key ${PICOVOICE_ACCESS_KEY} \
--picovoice-model-path ${PICOVOICE_MODEL_PATH}

Picovoice Leopard Instructions

Replace ${DATASET} with one of the supported datasets, ${DATASET_FOLDER} with the path to the dataset, ${LANGUAGE} with the target language, and ${PICOVOICE_ACCESS_KEY} with AccessKey obtained from Picovoice Console. If benchmarking a non-English language, include --picovoice-model-path and replace ${PICOVOICE_MODEL_PATH} with the path to a model file acquired from the Leopard Github Repo.

python3 benchmark.py \
--engine PICOVOICE_LEOPARD \
--dataset ${DATASET} \
--dataset-folder ${DATASET_FOLDER} \
--language ${LANGUAGE} \
--picovoice-access-key ${PICOVOICE_ACCESS_KEY} \
--picovoice-model-path ${PICOVOICE_MODEL_PATH}

Latency Benchmark Usage

In the following, we provide instructions for running the latency benchmark for each streaming engine. To run the benchmark, generate word timing alignment information for one of the supported datasets by following the instructions found here. Replace ${DATASET_PATH} with the path to the outputted folder from this process in the following commands.

Amazon Transcribe Instructions

Replace ${DATASET_FOLDER} with the path to an aligned dataset, ${LANGUAGE} with the target language, ${AWS_LOCATION} with the name of the AWS server and ${AWS_PROFILE} with the name of the AWS profile you wish to use.

python3 benchmark_latency.py \
--engine AMAZON_TRANSCRIBE_STREAMING \
--dataset-folder ${DATASET_FOLDER} \
--language ${LANGUAGE} \
--aws-profile ${AWS_PROFILE} \
--aws-location ${AWS_LOCATION}

Azure Speech-to-Text Instructions

Replace ${DATASET_FOLDER} with the path to an aligned dataset, ${LANGUAGE} with the target language, ${AZURE_SPEECH_KEY} and ${AZURE_SPEECH_LOCATION} information from your Azure account.

python3 benchmark_latency.py \
--engine AZURE_SPEECH_TO_TEXT_REAL_TIME \
--dataset-folder ${DATASET_FOLDER} \
--language ${LANGUAGE} \
--azure-speech-key ${AZURE_SPEECH_KEY} \
--azure-speech-location ${AZURE_SPEECH_LOCATION}

Google Speech-to-Text Instructions

Replace ${DATASET_FOLDER} with the path to an aligned dataset, ${LANGUAGE} with the target language, and ${GOOGLE_APPLICATION_CREDENTIALS} with credentials download from Google Cloud Platform.

python3 benchmark_latency.py \
--engine GOOGLE_SPEECH_TO_TEXT_STREAMING \
--dataset-folder ${DATASET_FOLDER} \
--language ${LANGUAGE} \
--google-application-credentials ${GOOGLE_APPLICATION_CREDENTIALS}

Whisper.cpp Streaming Instructions

Replace ${DATASET_FOLDER} with the path to an aligned dataset, ${LANGUAGE} with the target language, and ${WHISPER_CPP_MODEL} with the Whisper.cpp streaming model type (WHISPER_CPP_STREAMING_TINY, WHISPER_CPP_STREAMING_BASE, WHISPER_CPP_STREAMING_SMALL, WHISPER_CPP_STREAMING_MEDIUM, WHISPER_CPP_STREAMING_LARGE_V3 or WHISPER_CPP_STREAMING_LARGE_TURBO).

python3 benchmark_latency.py \
--engine ${WHISPER_CPP_MODEL} \
--dataset-folder ${DATASET_FOLDER} \
--language ${LANGUAGE}

Moonshine Streaming Instructions

Replace ${DATASET_FOLDER} with the path to an aligned dataset and ${MOONSHINE_MODEL} with the Moonshine streaming model type (MOONSHINE_STREAMING_TINY, MOONSHINE_STREAMING_SMALL or MOONSHINE_STREAMING_MEDIUM). This engine only supports English.

python3 benchmark_latency.py \
--engine ${MOONSHINE_MODEL} \
--dataset-folder ${DATASET_FOLDER} \
--language EN

Vosk Streaming Instructions

Replace ${DATASET_FOLDER} with the path to an aligned dataset and ${VOSK_MODEL} with the Vosk streaming model type (VOSK_STREAMING_SMALL or VOSK_STREAMING_LARGE).

python3 benchmark_latency.py \
--engine ${VOSK_MODEL} \
--dataset-folder ${DATASET_FOLDER} \
--language ${LANGUAGE}

Nemotron 3.5 ASR Streaming Instructions

Replace ${DATASET_FOLDER} with the path to an aligned dataset, ${LANGUAGE} with the target language, and ${CHUNK_SIZE_MS} with a supported chunk size: 80, 160, 320, 560 or 1120 (default 560).

python3 benchmark_latency.py \
--engine NEMOTRON_3_5_ASR_STREAMING \
--dataset-folder ${DATASET_FOLDER} \
--language ${LANGUAGE} \
--chunk-size-ms ${CHUNK_SIZE_MS}

Picovoice Cheetah Instructions

Replace ${DATASET_FOLDER} with the path to an aligned dataset, ${LANGUAGE} with the target language, and ${PICOVOICE_ACCESS_KEY} with an AccessKey obtained from Picovoice Console. By default, the Cheetah English model is used. For non-English languages models replace ${PICOVOICE_MODEL_PATH} with the path to a model file obtained from the Cheetah Github Repo.

python3 benchmark_latency.py \
--engine PICOVOICE_CHEETAH \
--dataset-folder ${DATASET_FOLDER} \
--language ${LANGUAGE} \
--picovoice-access-key ${PICOVOICE_ACCESS_KEY} \
--picovoice-model-path ${PICOVOICE_MODEL_PATH}

Results

English

Batch Engines Word Error Rate

Streaming Engines Word Error Rate

Streaming Engines Punctuation Error Rate

Core-Hour & Model Size

To obtain these results, we ran the benchmark across the entire LibriSpeech test-clean dataset and recorded the processing time. The measurement is carried out on an Ubuntu 22.04 machine with AMD CPU (AMD Ryzen 9 5900X (12) @ 3.70GHz), 64 GB of RAM, and NVMe storage, using 10 cores simultaneously. We omit Whisper Large from this benchmark.

Word Emission Latency

To obtain these results, we used 100 randomly selected files from the LibriSpeech test-clean dataset.

French

Batch Engines Word Error Rate

Streaming Engines Word Error Rate

Streaming Engines Punctuation Error Rate

German

Batch Engines Word Error Rate

Streaming Engines Word Error Rate

Streaming Engines Punctuation Error Rate

Italian

Batch Engines Word Error Rate

Streaming Engines Word Error Rate

Streaming Engines Punctuation Error Rate

Spanish

Batch Engines Word Error Rate

Streaming Engines Word Error Rate

Streaming Engines Punctuation Error Rate

Portuguese

For Amazon Transcribe, Azure Speech-to-Text, and Google Speech-to-Text, we report results with the language set to PT-BR, as this achieves better results compared to PT-PT across all engines.

Batch Engines Word Error Rate

Streaming Engines Word Error Rate

Streaming Engines Punctuation Error Rate

Korean

Streaming Engines Character Error Rate

Streaming Engines Punctuation Error Rate

Japanese

Streaming Engines Character Error Rate

Streaming Engines Punctuation Error Rate