📑 Paper | 👩‍💻 Github | 🤗 Model | 🤗 Dataset

DeSTA-AQA5M comprises 50 speech, environmental sound, and music datasets, totaling over 7,000 hours of audio. Our training framework centers on self-generated response for efficient cross-modal alignment. (see our paper!). In DeSTA, each audio clip is first transformed into a textual description using its metadata. A Large Language Model (LLM) is then prompted with this description to self-generate a response. Ultimately, we construct a large-scale dataset of 5 million audio–instruction–response triplets, enabling effective finetuning of a Large Audio Language Model (LALM)—without the need for task-specific instruction data.

Check GitHub dataset document for more details!

Load Dataset

from datasets import load_dataset

dataset = load_dataset("DeSTA-ntu/DeSTA-AQA5M-FROM-Llama3.1-8B-Instruct")

# Load from chunked data files
# dataset = load_dataset("DeSTA-ntu/DeSTA-AQA5M-FROM-Llama3.1-8B-Instruct", data_files=["audio.0.jsonl", "audio.1.jsonl", "speech.0.jsonl", "speech.1.jsonl"])

DatasetDict({
    train: Dataset({
        features: ['id', 'dataset', 'seed_description', 'prompt', 'response', 'messages'],
        num_rows: 4963845
    })
})

• Core fields (used for dataset generation and training):

  • messages: The input messages used for data generation and model training.
  • response: The model-generated response (used as the training target).

• Auxiliary fields (for display or metadata purposes):

  • id: Audio file ID(relative audio filepath)
  • dataset: The source dataset.
  • seed_description: The textual description constructed from the audio metadata.
  • prompt: The sampled prompt from the instruction pool.