AdapTIGER: Adaptive Spectral Feature Compression for Efficient Speech Separation

AdapTIGER extends TIGER (ICLR 2025) by replacing its fixed band-split encoder/decoder with a lightweight Spectral Feature Compression (SFC) cross-attention mechanism, enabling input-adaptive spectral grouping while reducing parameters by 15%.

Based on:

• TIGER: Time-frequency Interleaved Gain Extraction and Reconstruction (Xu et al., ICLR 2025)
• SFC: Spectral Feature Compression - input-adaptive frequency band learning

---

Overview

TIGER achieves competitive speech separation with only 822K parameters using hand-crafted frequency band splits (67 fixed sub-bands). However, these fixed boundaries cannot adapt to input characteristics - different speaker pitches, noise profiles, or acoustic conditions would benefit from different spectral groupings.

AdapTIGER replaces TIGER's 134 independent per-band modules (67 encoder + 67 decoder) with a single SFC encoder-decoder pair based on cross-attention. This:

• Reduces parameters from 822K to 696K (15% reduction)
• Enables input-adaptive spectral grouping - attention weights vary per input, allowing the model to dynamically adjust how frequency bins are grouped into bands
• Discovers harmonic structure - mid-frequency bands learn to adaptively attend to the fundamental frequency (F0) region based on speaker content

What's Changed vs Original TIGER

New files (our contribution)

Modified files

Unchanged from original TIGER

• look2hear/models/tiger.py - core separator architecture (FFI blocks, F3A, MSA modules) is untouched
• look2hear/losses/ - PIT wrapper and loss functions (neg SI-SDR, neg SNR)
• look2hear/datas/ - EchoSet and Libri2Mix data modules
• inference_speech.py, inference_dnr.py - inference scripts
• download_echoset.py - dataset download
• assets/ - original TIGER figures and logo

Key Contributions

1. SFC Encoder/Decoder: Replaces 134 fixed per-band modules with a single cross-attention mechanism. Learnable band queries (K=67) attend to frequency bins (F=321), producing soft, input-dependent spectral groupings.

2. G1 Gated Attention: Per-head sigmoid gate applied after scaled dot-product attention, breaking the low-rank bottleneck (rank <= d_head) inherent in standard multi-head attention.

3. SVD Query Initialization: Band queries initialized deterministically from the top singular vectors of the positional bias matrix - eliminates 3dB seed sensitivity observed with random initialization.

4. Four-Part Anti-Collapse Regularization (AdapTIGER-Reg variant):

   • Cosine-similarity log-barrier (prevents band collapse)
   • Smoothness penalty (prevents sharp jumps in similarity matrix)
   • SIGReg loss (prevents dimensionality collapse)
   • Adaptivity reward (encourages input-dependent attention patterns)

Architecture

AdapTIGER preserves TIGER's core separator (8 FFI blocks with F3A and MSA modules) and modifies only the encoder/decoder:

Input Mixture
    |
   STFT (win=640, hop=160) --> 321 freq bins x T frames
    |
  [SFC Encoder]  <-- NEW: cross-attention (67 band queries attend to 321 freq keys)
    |               with G1 gating, positional bias, SwiGLU FFN
    |
  Separator (8 FFI blocks, unchanged from TIGER)
    |
  [SFC Decoder]  <-- NEW: cross-attention (321 freq queries attend to 67 band keys)
    |
  Complex Mask x iSTFT
    |
  Separated Sources (x2)

Results

Performance on EchoSet (two-speaker separation):

• AdapTIGER-clean: No regularization. Learns input-adaptive attention but suffers band-feature collapse (all bands converge to similar representations).
• AdapTIGER-reg: Trades 0.66dB for structured band diversity via four-part regularization.

Installation

git clone <this-repo>
cd TIGER-plus
pip install -r requirements.txt

Usage

Training

On a SLURM cluster (recommended):

sbatch train.slurm

Or directly with PyTorch Lightning:

python audio_train.py --conf_dir configs/tiger-sfc-clean-v2.yml

Available configurations:

• tiger-baseline-3s.yml - Baseline TIGER (no SFC)
• tiger-sfc-clean-v2.yml - AdapTIGER-clean (SFC, no regularization)
• tiger-sfc-v11-3s.yml - AdapTIGER with SIGReg regularization

Evaluation

python audio_test.py --conf_dir configs/tiger-sfc-clean-v2.yml

Results are appended to results/table_test.csv and results/table_validation.csv.

Inference

python inference_speech.py --audio_path path/to/mixture.wav

Dataset

EchoSet (20,268 train / 4,604 val / 2,650 test samples at 16kHz):

sbatch download.slurm
# or
python download_echoset.py

Project Structure

audio_train.py                    # Training entry point (Lightning CLI)
audio_test.py                     # Evaluation entry point
inference_speech.py               # Single-file inference

configs/
  tiger-baseline-3s.yml           # Baseline TIGER config
  tiger-sfc-clean-v2.yml          # AdapTIGER-clean config
  tiger-sfc-v11-3s.yml            # AdapTIGER with regularization

look2hear/
  models/
    tiger.py                      # TIGER model (use_sfc flag toggles SFC)
    sfc.py                        # SFC encoder/decoder (cross-attention)
    sigreg.py                     # SIGReg anti-collapse regularizer
  system/
    audio_litmodule.py            # PyTorch Lightning training module
  losses/
    matrix.py                     # Loss functions (neg SI-SDR, neg SNR)
    pit_wrapper.py                # Permutation Invariant Training
  datas/
    echosetdatamodule.py          # EchoSet data module
  metrics/
    wrapper.py                    # Evaluation metrics

results/
  table_test.csv                  # Test set results across experiments
  table_validation.csv            # Validation results across experiments

graphs/
  plot_sfc_interpretability.py    # SFC attention visualization

analysis_outputs/
  sfc_interpretability/           # Attention heatmaps, adaptivity plots

paper_figures/                    # Publication-ready figures

train.slurm                       # SLURM training script (8 GPUs, DDP)
eval.slurm                        # SLURM evaluation script
download.slurm                    # SLURM dataset download script

Experiments/                      # Training outputs (checkpoints, logs)
DataPreProcess/EchoSet/           # Downloaded dataset

Citation

@article{xu2024tiger,
  title={TIGER: Time-frequency Interleaved Gain Extraction and Reconstruction for Efficient Speech Separation},
  author={Xu, Mohan and Li, Kai and Chen, Guo and Hu, Xiaolin},
  journal={arXiv preprint arXiv:2410.01469},
  year={2024}
}

Acknowledgments

This project builds on TIGER by Mohan Xu, Kai Li, Guo Chen, and Xiaolin Hu (Tsinghua University), and incorporates ideas from the Spectral Feature Compression framework.