🍣SALMon🍣

We present here the official dataset for SALMon🍣 benchmark as described in the paper "A Suite for Acoustic Language Model Evaluation" (ICASSP 2025 - Oral). We encourage you to also see the following resources, for further information:

🍣 Project Page: https://pages.cs.huji.ac.il/adiyoss-lab/salmon/
🍣 Paper: https://arxiv.org/abs/2409.07437
🍣 Code: https://github.com/slp-rl/salmon
🍣 Data Wavs (Google Drive): https://drive.google.com/drive/folders/1pVv6iMmP_VXH6Goxwnmpy-5h3jPAoJ0t?usp=share_link
🍣 Papers with Code (Leaderboard): https://paperswithcode.com/sota/language-modelling-on-salmon

If you use our datasets, please cite the paper as follows:

@INPROCEEDINGS{maimon2025salmon,
              author={Maimon, Gallil and Roth, Amit and Adi, Yossi},
              booktitle={ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)}, 
              title={Salmon: A Suite for Acoustic Language Model Evaluation}, 
              year={2025},
              volume={},
              number={},
              pages={1-5},
              keywords={Measurement;Codes;Publishing;Computational modeling;Pipelines;Benchmark testing;Signal processing;Acoustics;Background noise;Speech processing;Speech Language Models;Acoustic Modelling},
              doi={10.1109/ICASSP49660.2025.10888561}}
  

Dataset Summary

A dataset used to evaluate the ability of spoken language models to address acoustic content beyond the spoken text, through a modelling based metric. Modelling based metrics like SALMon🍣 request the SLM to assign a likelihood score for two audio samples and check whther the correct sample gets higher likelihood than the incorrect one.

Specifically, in SALMon🍣, we split the suite into several benchmarks, each measuring a specific aspect, and broadly categorised into two: acoustic consistency and semantic-acoustic alignment. In acoustic consistency we measure a real recording gets higher likelihood compared to a sample with the same content and beginning where an acoustic element (such as speaker or background noise) changes mid recording. In semantic-acoustic alignment we measure whether samples where the spoken content matches the acoustic get higher likelihood than mismatched ones. For instance, the phrase "I had the best day ever!" should be more likely in a happy voice.

You can either look at the entire dataset under "all", only one kind of task under "all_consistency" and "all_alignment". You can also focus on one specific task, such as "speaker_consistency".

Download

Using 🤗 Datasets

from datasets import load_dataset
# entire dataset
salmon = load_dataset('slprl/salmon', 'all')
# acoustic consistency only
salmon_consistency = load_dataset('slprl/salmon', 'all_consistency')
# One specific task
salmon_bg_alignment = load_dataset('slprl/salmon', 'bg_alignment')

You can also load and mix:

from datasets import concatenate_datasets, load_dataset
speaker_consistency = load_dataset('slprl/salmon', 'speaker_consistency')
rir_consistency = load_dataset('slprl/salmon', 'rir_consistency')
# Concatenate both datasets
concatenated = concatenate_datasets([speaker_consistency, rir_consistency])

Using Google Drive

We also provide raw wav files for any cases where Datasets is inconvient (such as Audio loading or API issues). We give an unzipped data here, or a zipped version here. We provide explaination of how to use this dataset to compute the SALMon🍣 scores in the provided code.

Data Fields

The data has several fields:

• task: The name of the specific task
• ind: The index of the sample in the task, note that this is not necessarily contiuous and can go beyond 200.
• positive_audio: The true audio sample
  • array: array of audio samples
  • sample_rate: audio sampling rate
  • path: path to the audio file saved location
• negative_audio: The distractor audio sample
  • array: array of audio samples
  • sample_rate: audio sampling rate
  • path: path to the audio file saved location