AudSem Dataset without Semantic Descriptors

• Accompanying paper: https://arxiv.org/abs/2505.14142.
• GitHub repo: https://github.com/gljs/audsemthinker

Dataset Description

Overview

The AudSem-Simple dataset (audsem-simple) is a streamlined version of the AudSem dataset, designed to enhance the reasoning capabilities of Audio-Language Models (ALMs) through structured thinking over sound, but without explicit semantic element breakdowns.

The dataset includes four task-specific configurations:

• aac (Audio Captioning): Audio captioning tasks with detailed descriptions
• creative_qa (Creative Question Answering): Creative writing and story generation based on audio
• mc_qa (Multiple Choice QA): Multiple-choice questions about audio content
• qa (Open-ended QA): Open-ended questions requiring free-form answers

The full version with semantic elements is available at: https://huggingface.co/datasets/gijs/audsem

All configurations are derived from the same rigorously filtered audio-visual data and are designed to minimize overlap with existing benchmarks like AudioSet, AudioCaps, and WavCaps, addressing a critical challenge of data contamination in zero-shot evaluations.

Motivation

Traditional audio-language models often struggle with complex reasoning over sound events, primarily due to:

1. Limited diverse training data: Most existing datasets are sourced from overlapping collections (e.g., AudioSet, Freesound), leading to homogenized data landscapes and hindering generalization.
2. Lack of structured reasoning mechanisms: Unlike general foundation models, audio understanding models rarely implement explicit "thinking" or reasoning phases before generating responses.

AudSem directly addresses these issues by:

• Providing a novel dataset built from YouTube closed captions, reducing redundancy with established benchmarks.
• Enabling the training of ALMs to perform structured reasoning by explicitly incorporating a "thinking" phase in the data, inspired by human auditory cognition.

Data Structure

Each configuration in the AudSem-Simple dataset has the following structure:

Common fields across all configurations:

• key: Unique identifier for the example
• file_name: Audio file name/identifier
• thinking: The model's reasoning process before answering (without semantic elements)
• answer: The final response to the prompt

Configuration-specific fields:

• aac: Contains question (the audio captioning prompt)
• creative_qa: Contains question (creative writing prompt)
• mc_qa: Contains question (multiple choice question) and choices (answer options as a string)
• qa: Contains question (open-ended question)

When loaded with the Hugging Face datasets library, you must specify a configuration:

{
  '__key__': 'f00b3c0c-5852-49fc-9962-866e38bcdebb',
  '__url__': './datasets--gijs--audsem/snapshots/6ec5552b4b16fcb94080a8a94972b33daaeeaf1b/creative_qa/train/0000.tar',
  'flac': {
    'path': 'f00b3c0c-5852-49fc-9962-866e38bcdebb.flac', 
    'array': array([0.0, 0.0, 0.0, ..., 0.01708984, 0.01660156, 0.01660156]), 
    'sampling_rate': 48000
  },
  'json': {
    '__key__': 'f00b3c0c-5852-49fc-9962-866e38bcdebb', 
    'answer': 'The first 30 seconds of the performance at the small, intimate venue were a testament...lasting impression on all who witnessed it.', 
    'file_name': 'f00b3c0c-5852-49fc-9962-866e38bcdebb', 
    'question': 'Imagine you are a seasoned music critic attending a live acoustic performance. Write ...e and the overall atmosphere of the venue.', 
    'thinking': 'To answer this question, I need to focus on the dynamic and passionate nature of the ...s reaction."
  }
}

The dataset includes four types of tasks, generated for the audsem-simple configuration:

• Audio Captioning: An instruction like "Describe the audio in detail."
• Multiple-choice Audio Question Answering: A question with four possible answers, requiring the model to select the correct one.
• Open-ended Audio Question Answering: A question requiring a free-form textual answer.
• Creative Writing/Story Generation: Prompts encouraging creative descriptions based on the audio.

Data Fields

The dataset examples have the following fields depending on configuration:

All configurations include:

• key: Unique identifier for the example
• file_name: Audio file name/identifier
• thinking: The model's reasoning process (detailed thought process about the audio, without semantic element breakdown)
• answer: The final response to the prompt

Configuration-specific fields:

• aac: Includes question (audio captioning prompt)
• creative_qa: Includes question (creative writing prompt)
• mc_qa: Includes question (multiple choice question) and choices (answer options as string)
• qa: Includes question (open-ended question)

Dataset Statistics

Data Sizes

• Initial YouTube Subtitles: Approximately 2.9 billion lines from over 18 million videos.
• Filtered SDH Captions: 75,794,563 lines.
• After BERT/Mixtral Filtering: 9,464,882 potential audio captions.
• After Video Download & Preprocessing: 5,332,211 raw audio-visual samples.
• After Quality Filtering: 213,908 high-quality audio-visual samples used for generation.
• Final Generated Examples:
  • audsem-simple: Approximately 873,000 examples.
• Audio Length: All audio samples are at least 3 seconds long.
• Average Caption Length (generated by Qwen2.5-72B): 852.63 characters (for the full generated text, including thinking and answer).

Overlap with other datasets

A rigorous filtering process was applied to minimize overlap:

• Only 12 examples were found to overlap with AudioSet.
• Only 1 example overlapped with AudioCaps.
• 0 examples overlapped with VGGSound.

Modality Distribution

The dataset was generated from data encompassing audio, video, text (closed captions), and image modalities, ensuring a rich contextual understanding for the synthetic generation process.

Data Collection and Annotation

The AudSem dataset was created through a robust, multi-stage, and fully automated pipeline, involving several advanced AI models.

Data Sources

The primary source for AudSem is a vast collection of manually annotated English closed caption subtitles from YouTube videos, provided by Filmot.com. These captions were filtered to specifically identify Subtitles for Deaf and Hard of Hearing (SDH) entries, which often contain sound descriptions enclosed in brackets.

Data Acquisition Process

1. Initial Filtering: YouTube subtitles are initially filtered using regular expressions to identify potential sound descriptions (e.g., lines starting and ending with brackets, parentheses, or curly braces). Captions outside a 1-10 second length range are removed.
2. Sound Description Verification: A BERT-based classifier and a Mixtral-8x7B-Instruct model (in a zero-shot setting) are used to confirm that filtered captions genuinely represent sound descriptions, excluding dialogue or unrelated text.
3. Video Segment Download: yt-dlp is used to download precise audio-visual segments corresponding to the verified captions based on their timestamps.
4. Standardization: ffmpeg converts videos to 360p (2 fps MP4) and extracts audio to WAV format (48kHz, 16-bit, mono) for consistent processing.
5. Packaging: Data is organized into WebDataset tar files for efficient loading.

Data Processing and Filtering

The acquired audio-visual segments undergo comprehensive analysis using an ensemble of specialized AI models across modalities:

• Audio Analysis:
  • Qwen2Audio-7B-Instruct: Generates detailed natural language audio descriptions.
  • BEATs: Performs multi-label audio tagging.
  • AST: Classifies audio events into AudioSet categories.
  • CoNeTTE: Provides contextual audio event descriptions.
  • LP-MusicCaps Model: Specializes in music captioning for musical segments.
• Visual Analysis (from sampled video frames):
  • BLIP: Generates natural language captions for frames.
  • CLIP: Performs zero-shot classification against object categories.
  • RT-DETR: Identifies objects with bounding boxes.
  • Places365-trained ResNet-50: Classifies scene/environment.
• Video Analysis (temporal context):
  • LLaVA-Video-7B-Qwen2: Processes uniformly sampled frames to capture temporal progression and spatial relationships.

Quality Filtering Steps:

• Outlier Removal: Samples with CLAP embedding cosine distance > 0.9 from the average embedding (for both audio and text) are removed.
• Length Filtering: Audio samples shorter than three seconds are excluded.
• Caption Alignment: A crucial step filters samples where the cosine similarity between the Qwen2Audio-7B generated caption and the original YouTube closed caption is below 0.5. This ensures the synthetic caption aligns with the human-provided ground truth.

Synthetic Annotation Process

The final captions and reasoning structures are synthetically generated using the Qwen2.5-72B-Instruct model, acting as a "teacher model."

1. Structured Generation: The model generates output in a structured JSON format, enforced by xgrammar and vLLM. This includes:
   • A <thinking> phase: Detailed reasoning about primary/background sounds, events, activities, and environment (minimum 50 words). This phase incorporates natural language thought expressions and avoids direct mention of model outputs or visual context.
   • An <answer> phase: A concise audio caption (under 50 words).
2. Multi-task Generation: For each filtered audio-visual sample, various task types (captioning, QA, creative writing) are generated, producing 2-3 examples per input for non-captioning tasks.
3. Validation and Regeneration: A separate "judging model" (also Qwen2.5-72B-Instruct) validates generated outputs against predefined guidelines (e.g., adherence to CoT, no direct data field mentions, no visual elements in answer). Failed outputs are regenerated up to five times.

This fully automated process ensures high quality, diversity, and scalability, with the human-created closed captions serving as an implicit ground truth for filtering and validation.

Usage

How to load

from datasets import load_dataset

# Load a specific configuration
dataset_aac = load_dataset("gijs/audsem-simple", "aac")  # Audio captioning
dataset_qa = load_dataset("gijs/audsem-simple", "qa")  # Open-ended QA
dataset_mc = load_dataset("gijs/audsem-simple", "mc_qa")  # Multiple choice QA
dataset_creative = load_dataset("gijs/audsem-simple", "creative_qa")  # Creative writing

# Access train and validation splits
train_data = dataset_aac["train"]
valid_data = dataset_aac["valid"]

# You can iterate through the dataset or access individual examples
print(train_data[0])

Example (Illustrative, actual content may vary)

Below are examples of the different task types in the dataset:

Open-ended Audio Question Answering (qa):

{
  "file_name": "FMbU3YwI0CY (99_12-105_22)",
  "question": "What type of drum patterns are being played in this segment?",
  "thinking": "To answer this, I need to focus on the specific drum sounds and their characteristics. The audio contains punchy snare hits and rolls, along with energetic cymbals and a steady kick drum beat. The drum patterns seem to follow a consistent rock groove, which suggests a rhythmic and energetic performance. The low recording quality adds a groovy feel.",
  "answer": "The drum patterns include punchy snare hits, rolls, and a steady kick drum beat, all following a consistent rock groove. The performance is rhythmic and energetic."
}

Multiple-choice Audio Question Answering (mc_qa):

{
  "file_name": "6QDMUkIUsu4 (468_841-472_008)",
  "question": "What is the primary mood of the music playing in the background of the audio clip?\nChoices:\nA. Light and upbeat\nB. Dark and ominous\nC. Sad and melancholic\nD. Romantic and dreamy",
  "choices": {
    "A": "Light and upbeat",
    "B": "Dark and ominous",
    "C": "Sad and melancholic",
    "D": "Romantic and dreamy"
  },
  "thinking": "The music is described as an electronic dance piece with a strong rhythmic beat, synths, bass, and drums. It has a touch of darkness, but the overall mood is light and upbeat, suitable for a soundtrack or video game. The key is E minor, which can sometimes indicate a slightly darker tone, but the tempo and rhythmic elements suggest a more positive and energetic feel. The model-generated audio information supports this by mentioning that it is light and upbeat.",
  "answer": "A: Light and upbeat"
}

Audio Captioning (aac):

{
  "file_name": "tar/INiaiPypaOU (57_132-60_632)",
  "thinking": "I start by listening to the audio and notice that there is a strong, consistent electronic beat throughout the segment. The music has a deep, spacey feel with a strong rhythmic foundation. There are also faint elements of synths and bass, which add to the experimental and techno genre. The music is instrumental, and there are no discernible vocals. The beat is steady and consistent, suggesting a danceable tempo. I detect no speech or other distinct sounds that could distract from the music. The overall atmosphere is upbeat and energetic, perfect for a scene where characters are engaged in a dynamic activity, such as preparing for a trip or moving items at night. The sound is clear and well-produced, with a professional quality that fits the genre.",
  "question": "Give a comprehensive breakdown of the audio content.",
  "answer": "Upbeat electronic music with a deep, spacey feel, featuring a strong rhythmic beat, synths, and bass. The music is instrumental and danceable, with a steady, consistent tempo."
}

Creative Writing/Story Generation (creative_qa):

{
  "file_name": "O93mKY3qlyA (433_666-437_937)",
  "question": "Imagine you are a journalist writing a live report of this event. Describe the atmosphere in the room as the audience claps, and capture the energy of the moment in a compelling report.",
  "thinking": "To answer this question, I will focus on the sound of the audience clapping, which suggests a positive and enthusiastic response to a speech. The key is to capture the energy and enthusiasm in the atmosphere, imagining the room's dynamics and the emotions of the people present. The clapping, which is described as rhythmic and strong, will be central to the narrative. I will also consider the possible reasons for the applause, such as the speaker's compelling delivery or the content of the speech.",
  "answer": "In the legislative chamber, a wave of enthusiastic applause washed over the room, punctuating the end of a powerful address. The audience, a sea of faces, erupted into rhythmic clapping, their hands merging into a symphony of approval. The energy was electric, a palpable surge of collective pride and agreement. Each clap seemed to amplify the impact of the words that had just been spoken, resonating through the air like a promise fulfilled. The room, usually a bastion of formal decorum, was momentarily transformed into a conduit of unbridled enthusiasm. It was clear that the speaker had struck a chord, his words resonating deeply with those present. The clapping, a testament to the power of oratory, continued, a rhythmic affirmation of the ideas that had been shared. The moment was one of unity and inspiration, a perfect encapsulation of the power of a well-delivered speech."
}

Limitations and Biases

Limitations

• Synthetic Nature: While rigorously filtered and validated, the captions are synthetically generated. This may introduce subtle biases or artifacts inherent to the large language model used for generation (Qwen2.5-72B-Instruct).

Potential Biases

• YouTube Data Bias: As the dataset is sourced from YouTube closed captions, it may inherit biases present in YouTube content, such as popular topics, common filming locations, or the demographic representation of video creators.
• Captioner Bias: While the initial closed captions are often human-generated, their original creators might have their own biases in what they choose to describe or how.
• LLM Generation Bias: The Qwen2.5-72B-Instruct model, like all large language models, may exhibit biases (e.g., societal, cultural) present in its vast training data. These biases could be reflected in the generated descriptions and reasoning, even with structured prompts.

Ethical Considerations

The AudSem dataset development adhered to ethical guidelines, including:

• Responsible Data Science: The scraping of YouTube data involved systematic checks for harmful content (e.g., child abuse, hate speech, sexual content, harassment). No such terms were found, and YouTube's community guidelines offer an initial safeguard.
• Privacy: The dataset focuses on environmental and musical sounds and does not contain personally identifiable information (PII) or direct speech content of individuals intended for identification.
• Broader Impact: The research primarily contributes to fundamental advancements in audio-language understanding. Potential positive societal impacts include:
  • Enhanced audio transcription and captioning for accessibility (e.g., for individuals who are deaf or hard of hearing).
  • Improved sophisticated monitoring systems for environmental sounds (e.g., wildlife, urban noise).
  • More accurate automated closed-caption generation for multimedia content.

No immediate negative real-world applications or misuse scenarios are proposed or directly facilitated by this fundamental research.

Citation

If you use the AudSem dataset in your research, please cite the accompanying paper:

@misc{wijngaard2025audsemthinkerenhancingaudiolanguagemodels,
  title={AudSemThinker: Enhancing Audio-Language Models through Reasoning over Semantics of Sound}, 
  author={Gijs Wijngaard and Elia Formisano and Michele Esposito and Michel Dumontier},
  year={2025},
  eprint={2505.14142},
  archivePrefix={arXiv},
  primaryClass={cs.SD},
  url={https://arxiv.org/abs/2505.14142}, 
}