StepEval-Audio-Paralinguistic Dataset

Paper: Step-Audio 2 Technical Report
Code: https://github.com/stepfun-ai/Step-Audio2
Project Page: https://www.stepfun.com/docs/en/step-audio2

Overview

StepEval-Audio-Paralinguistic is a speech-to-speech benchmark designed to evaluate AI models' understanding of paralinguistic information in speech across 11 distinct dimensions. The dataset contains 550 carefully curated and annotated speech samples for assessing capabilities beyond semantic understanding.

Key Features

• Comprehensive coverage: 11 paralinguistic dimensions with 50 samples each
• Diverse sources: Combines podcast recordings with AudioSet, CochlScene, and VocalSound samples
• High-quality annotations: Professionally verified open-set natural language descriptions
• Challenging construction: Includes synthesized question mixing and audio augmentation
• Standardized evaluation: Comes with automatic evaluation protocols

Dataset Composition

Core Categories

1. Basic Attributes

   • Gender identification
   • Age classification
   • Timbre description

2. Speech Characteristics

   • Emotion recognition
   • Pitch classification
   • Rhythm patterns
   • Speaking speed
   • Speaking style

3. Environmental Sounds

   • Scenario detection
   • Sound event recognition
   • Vocal sound identification

Task Categories and Label Distributions

Dataset Notes:

• Total samples: 550 (50 per category × 11 categories)
• Underrepresented categories were augmented to ensure diversity
• Scene/event categories use synthetic audio mixing with controlled parameters
• All audio samples are ≤30 seconds in duration

Data Collection & Processing

Preprocessing Pipeline

• All audio resampled to 24,000 Hz
• Strict duration control (≤30 seconds)
• Demographic balancing for underrepresented groups
• Professional annotation verification

Special Enhancements

• Scenario: 6 environmental types mixed (from CochlScene)
• Event: AudioSet samples mixed
• Vocal: 7 paralinguistic types inserted (from VocalSound)

Dataset Construction

1. Collected raw speech samples from diverse sources
2. Generated text-based QA pairs aligned with annotations
3. Converted QAs to audio using TTS synthesis
4. Randomly inserted question clips before/after original utterances
5. For environmental sounds: additional audio mixing before question concatenation

Evaluation Protocol

The benchmark evaluation follows a standardized three-phase process:

1. Model Response Collection

Audio-in/audio-out models are queried through their APIs using the original audio files as input. Each 24kHz audio sample (≤30s duration) generates a corresponding response audio, saved with matching filenames for traceability.

2. Speech-to-Text Conversion

All model response audios are transcribed using a ASR system. Transcripts undergo automatic text normalization and are stored.

3. Automated Assessment

The evaluation script (LLM_judge.py) compares ASR transcripts against ground truth annotations using an LLM judge. Scoring considers semantic similarity rather than exact matches, with partial credit for partially correct responses. The final metrics include per-category accuracy scores.

Benchmark Results on StepEval-Audio-Paralinguistic