ProtoCLR

This repository contains a CvT-13 Convolutional Vision Transformer model trained from scratch on the Xeno-Canto dataset, specifically on 6-second audio segments sampled at 16 kHz. The model is trained on Mel spectrograms of bird sounds using ProtoCLR (Prototypical Contrastive Loss) for 300 epochs and can be used as a feature extractor for bird audio classification and related tasks.

Files

• cvt.py: Defines the CvT-13 model architecture.
• protoclr.pth: Pre-trained model weights for ProtoCLR.
• config/: Configuration files for CvT-13 setup.
• mel_spectrogram.py: Contains the MelSpectrogramProcessor class, which converts audio waveforms into Mel spectrograms, a format suitable for model input.

Setup

1. Clone this repository: Clone the repository and navigate into the project directory: git clone https://huggingface.co/ilyassmoummad/ProtoCLR cd ProtoCLR/

2. Install dependencies: Ensure you have the required Python packages, including torch and any other dependencies listed in requirements.txt.

   pip install -r requirements.txt
   

Usage

1. Prepare the Audio:
   To ensure compatibility with the model, follow these preprocessing steps for your audio files:

   • Mono Channel (Mandatory):
     If the audio has multiple channels, convert it to a single mono channel by averaging the channels.
   • Sample Rate (Mandatory):
     Resample the audio to a consistent sample rate of 16 kHz.
   • Padding (Recommended):
     For audio files shorter than 6 seconds, pad with zeros or repeat the audio until it reaches a length of 6 seconds.
   • Chunking (Recommended):
     For audio files longer than 6 seconds, split them into chunks of 6 seconds each for better processing.

2. Process the Audio:
   Use the MelSpectrogramProcessor (from melspectrogram.py) to transform the prepared audio into a Mel spectrogram, a format suitable for model input, as demonstrated in the following example.

Example Code

The following example demonstrates loading, processing, and running inference on an audio file:

import torch
from cvt import cvt13  # Import model architecture
from melspectrogram import MelSpectrogramProcessor  # Import Mel spectrogram processor

# Initialize the preprocessor and model
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
preprocessor = MelSpectrogramProcessor(device=device)
model = cvt13()

# Load weights trained using Cross-Entropy
model.load_state_dict(torch.load("ce.pth", map_location="cpu")['encoder'])

# Load weights trained using SimCLR (self-supervised contrastive learning)
model.load_state_dict(torch.load("simclr.pth", map_location="cpu"))

# Load weights trained using SupCon (supervised contrastive learning)
model.load_state_dict(torch.load("supcon.pth", map_location="cpu"))

# Load weights trained using ProtoCLR (supervised contrastive learning using prototypes)
model.load_state_dict(torch.load("protoclr.pth", map_location="cpu"))

# Optional: Move the model to GPU for faster processing if available using : model = model.to('cuda') , for instance.
model = model.to(device)
model.eval()

# Load and preprocess a sample audio waveform
def load_waveform(file_path):
    # Replace this with your specific audio loading function
    # For example, using torchaudio to load and resample
    pass

waveform = load_waveform("path/to/audio.wav").to(device)  # Load your audio file here and convert it to a PyTorch tensor.

# Ensure waveform is sampled at 16 kHz, then pad/chunk as needed for 6s length
input_tensor = preprocessor.process(waveform).unsqueeze(0)  # Add batch dimension

# Run the model on the preprocessed audio
with torch.no_grad():
    output = model(input_tensor)
    print("Model output shape:", output.shape)

Model Performance Comparison

The following table presents the classification accuracy of various models on one-shot and five-shot bird sound classification tasks, evaluated across different soundscape datasets.

For additional details, please see the pre-print on arXiv and the official GitHub repository.

Citation

If you use our model in your research, please cite the following paper:

@misc{moummad2024dirlbs,
      title={Domain-Invariant Representation Learning of Bird Sounds}, 
      author={Ilyass Moummad and Romain Serizel and Emmanouil Benetos and Nicolas Farrugia},
      year={2024},
      eprint={2409.08589},
      archivePrefix={arXiv},
      primaryClass={cs.SD},
      url={https://arxiv.org/abs/2409.08589}, 
}