audio-embeddings

This tool exists to generate large tables of embeddings of audio waveforms that have been augmented in many ways for the purpose of studying the properties of the embeddings generated by various pre-trained audio models.

Getting Started

Installing

Note: requires installing laion_clap>=1.1.6 which contains important bug fixes, but is unavailable through PYPI. This fork provides a version which accepts numpy>=2.0.0 and is therefore compatible with this repository.

Installation via uv:

git clone https://github.com/Aramist/audio-embeddings.git && cd audio-embeddings
uv sync

Installation via pip:

git clone https://github.com/Aramist/audio-embeddings.git && cd audio-embeddings
python -m pip install .

Accessing API:

All augmentation and embedding modules accept audio in the form of (waveform, sample_rate) tuples and will internally perform up/downsampling conversions wherever necessary for compatibility with pretrained models.

import soundfile as sf
import torch

from audiomanifolds.embeddings import PannEmbedder
from audiomanifolds.transformations import Gain

audio, sr = sf.read(AUDIO_PATH)  # (num_channels, num_samples)
audio = torch.from_numpy(audio)

gain_aug = Gain(
    gains=[-5, 5, 10],  # gain manipulations in dB
)
audio_with_gain = gain_aug((audio, sr))  # (num_augs=3, num_channels, num_samples)

batched_audio = audio[None, ...].repeat(64, 1, 1)  # (64, num_channels, num_samples)
batched_aug_audio = gain_aug((batched_audio, sr))  # (64, num_augs=3, num_channels, num_samples)

pretrained_embedder = PannEmbedder.from_pretrained(auto_convert_sample_rate=True)

embedded_audio = pretrained_embedder((batched_audio, sr))  # (64, num_channels, embedding_size)

Implemented features

Pretrained embeddings:

• PANN (audiomanifolds.embeddings.PannEmbedder)
• CLAP (audiomanifolds.embeddings.CLAPAudioEmbedder)

Audio augmentations:

• Pitch shifting (audiomanifolds.transformations.PitchShifting)
• Temporal dilation / compression (audiomanifolds.transformations.TimeStretching)
• Gain (audiomanifolds.transformations.Gain)

Authors

• Aramis Tanelus

Repository structure

scripts/                        Scripts demonstrating package functionality
├─ make_embedding_table.py          Generate a table of embeddings for multiple augmentations of multiple audio examples
tests/                          Tests verifying core package functionality
src/                            Main implementation code
├─ audiomanifolds/                  
│  ├─ transformations                   Submodule containing audio augmentation functions
│  ├─ embeddings                        Submodule containing wrappers around pretrained models

Version History

• 0.0.1
  • Initial pre-release

License

TODO

Acknowledgments

• README template