Nyra Disfluency Speech English

nyrahealth/disfluency_speech_english is an English speech dataset for evaluating verbatim ASR: models that should transcribe not only the intended words, but also fillers, cutoffs, repetitions, and sound events.

This dataset is based on the AMAAI Lab DisfluencySpeech dataset and reformatted for verbatim-transcription benchmarking with paired:

• verbatim_transcript: what the speaker actually said
• intended_transcript: a cleaned version of what the speaker meant to say

It is used by the Nyra Verbatim Speech Benchmark, which evaluates verbatim ASR in detail and breaks errors down into fillers, sounds, cutoffs, repetitions, and intended-transcript failures.

For the exact convention definitions used by the benchmark, see:

• Nyra Verbatim Speech Benchmark
• Verbatim transcript conventions

Source

This release is derived from:

• Dataset: amaai-lab/DisfluencySpeech
• Paper: DisfluencySpeech -- Single-Speaker Conversational Speech Dataset with Paralanguage

The original dataset provides annotated transcripts and several progressively cleaned transcript variants. This Nyra release converts the data into a format that is directly usable for verbatim-ASR evaluation with paired verbatim and intended references.

Dataset Structure

The dataset contains 4,957 utterances and about 9.4 hours of audio.

Splits:

• train: 4,458
• validation: 250
• test: 249

Features:

• id
• audio
• duration_in_s
• split
• speaker
• verbatim_transcript
• intended_transcript

Transcription Conventions

Verbatim Transcript

The verbatim_transcript follows a small set of explicit conventions so disfluencies and non-speech events can be evaluated consistently:

• Cutoffs use *, for example th* or w*
• Fillers are bracketed tags, primarily [UH] and [UM]
• Sound events are also bracketed tags, for example [laughter], [breath], or [cough]
• Repetitions are written as repeated words, not as separate tags
• Spoken words are otherwise written as they were said

Example:

I mean we we [UH] should go on th* Thursday [laughter]

Intended Transcript

The intended_transcript is the cleaned target for intended ASR. It removes disfluent material while preserving the speaker's meaning, including fillers, sound tags, repeated restarts, and cutoff fragments.

Example:

verbatim: I mean we we [UH] should go on th* Thursday [laughter]
intended: we should go on Thursday

This makes the dataset suitable for evaluating both:

• verbatim transcription quality
• intended transcription quality

Tag Analysis

The counts below were computed over the full dataset from verbatim_transcript.

Summary:

• utterances: 4,957
• utterances with at least one bracketed tag or cutoff: 2,779
• total bracketed tags: 4,039
• total cutoff tokens: 582

Fillers

Sound Tags

Cutoffs

These statistics are useful when interpreting benchmark results: fillers are common, laughter is the most frequent sound event, and cutoffs occur often enough to matter as a separate evaluation category.

Benchmark Usage

This dataset is designed to be used with the Nyra Verbatim Speech Benchmark.

That benchmark:

• derives gold disfluency labels automatically from the verbatim and intended transcript pair
• computes transcript metrics such as vWER and iWER
• computes event metrics for fillers, sounds, cutoffs, and repetitions
• provides detailed error analysis for verbatim ASR models

Citation

If you use this dataset, please cite the original DisfluencySpeech paper:

@misc{wang2024disfluencyspeechsinglespeakerconversational,
  title={DisfluencySpeech -- Single-Speaker Conversational Speech Dataset with Paralanguage},
  author={Kyra Wang and Dorien Herremans},
  year={2024},
  eprint={2406.08820},
  archivePrefix={arXiv},
  primaryClass={eess.AS},
  url={https://arxiv.org/abs/2406.08820}
}