Evaluating parameter-efficient transfer learning approaches on SURE benchmark for speech understanding

Paper

Motivation

Fine-tuning is widely used as the default algorithm for transfer learning from pre-trained models. Parameter inefficiency can however arise when, during transfer learning, all the parameters of a large pre-trained model need to be updated for individual downstream tasks. As the number of parameters grows, fine-tuning is prone to overfitting and catastrophic forgetting. In addition, full fine-tuning can become prohibitively expensive when the model is used for many tasks. To mitigate this issue, parameter-efficient transfer learning algorithms, such as adapters and prefix tuning, have been proposed as a way to introduce a few trainable parameters that can be plugged into large pre-trained language models such as BERT, and HuBERT. In this paper, we introduce the Speech UndeRstanding Evaluation (SURE) benchmark for parameter-efficient learning for various speech-processing tasks. Additionally, we introduce a new adapter, ConvAdapter, based on 1D convolution. We show that ConvAdapter outperforms the standard adapters while showing comparable performance against prefix tuning and LoRA with only 0.94% of trainable parameters on some of the tasks in SURE. We further explore the effectiveness of parameter efficient transfer learning for speech synthesis task such as Text-to-Speech (TTS).

Installation

• Set up environments

conda create --name speechprompt python==3.8.5
conda activate speechprompt
conda install pytorch==1.10.0 torchvision==0.11.0 torchaudio==0.10.0 -c pytorch

• Install other dependencies

pip install -r requirements.txt

Supported tasks and datasets

How to run

First, we need to specify datasets and arguments. let's use "esd" as the dataset, "finetune" as the tuning method in "speech emotion recognition" task as an example:

CUDA_VISIBLE_DEVICES=2,3 python train.py \
		--dataset "esd" \
		--data_dir "/data/path/ESD" \
		--output_dir '/data/path/output_earlystop_ser_esd_finetune_2e3' \
		--do_train True \
		--do_eval True \
		--do_predict False \
		--evaluation_strategy "steps" \
		--save_strategy "steps" \
		--save_steps 500 \
		--eval_steps 25 \
		--learning_rate 2e-3 \
		--feat_adapter_name "conv_adapter" \
		--trans_adapter_name "adapterblock" \
		--output_adapter False \
		--mh_adapter False \
		--prefix_tuning False \
		--lora_adapter False \
		--feat_enc_adapter False \
		--fine_tune True \
		--per_device_train_batch_size 64 \
		--gradient_accumulation_steps 4 \
		--per_device_eval_batch_size 64 \
		--num_train_epochs 100 \
		--warmup_ratio 0.1 \
		--logging_steps 20 \
		--logging_dir '/data/path/output_earlystop_ser_esd_finetune_2e3/log' \
		--load_best_model_at_end True \
		--metric_for_best_model "f1" 

Parameters

• dataset: specify the dataset, such as "esd", "fleurs", "fluent_commands", etc.
• data_dir: path to the dataset file, for instance, "../data/path/ESD"
• output_dir: path to the checkpoints and logs, for instance, '../data/path/output_earlystop_ser_esd_finetune_2e3'
• do_train: True if want to train
• do_eval: True if want to eval
• do_predict: True if want to inference
• evaluation_strategy: It can be set according to the official setting of huggingface
• save_strategy: It can be set according to the official setting of huggingface
• save_steps: It can be set according to the official setting of huggingface
• eval_steps: It can be set according to the official setting of huggingface
• learning_rate: It can be set according to the official setting of huggingface
• feat_adapter_name: The adapter type added in the features encoder, but not applied to this article, can be skipped
• trans_adapter_name: The adapter type added in transformer layer, such as "adapterblock" for ConvAdapter and "bottleneck" for Bottleneck Adapter
• output_adapter: True if added after feedforward of every transformer layer, only control ConvAdapter and Bottleneck Adapter
• mh_adapter: True if added after multihead attention of every transformer layer, only control ConvAdapter and Bottleneck Adapter
• prefix_tuning: True if prefix-tuning is added
• lora_adapter: True if Lora is added
• feat_enc_adapter: True if adapter is add in features encoder of wav2vec2
• fine_tune: True if only need fine tuning
• per_device_train_batch_size: It can be set according to the official setting of huggingface
• gradient_accumulation_steps: It can be set according to the official setting of huggingface
• per_device_eval_batch_size: It can be set according to the official setting of huggingface
• num_train_epochs: It can be set according to the official setting of huggingface
• warmup_ratio: It can be set according to the official setting of huggingface
• logging_steps: It can be set according to the official setting of huggingface
• logging_dir: It can be set according to the official setting of huggingface
• load_best_model_at_end: It can be set according to the official setting of huggingface
• metric_for_best_model: It can be set according to the official setting of huggingface

Emotion classification

Let's further explain the five training methods of the model. For example, start a new emotion classification task, we will set the corresponding parameter like below:

## finetune
--fine_tune True
## bottleneck
--trans_adapter_name "bottleneck"
--output_adapter True
## prefix-tuning
--prefix_tuning True
## lora
--lora_adapter True
## ConvAdapter
--trans_adapter_name "adapterblock"
--output_adapter True

We also placed examples according to each training method in "emotion_cls.sh", using the following command to start new emotion classification task:

bash emotion_cls.sh

Tensorboard

In order to further supervise the convergence of model training, we can view the log file through Tensorboard:

tensorboard --logdir=/data/path/output_earlystop_asr_fleurs_lora_2e3/log --bind_all

Citation

@inproceedings{li2023evaluating,
  title={Evaluating Parameter-Efficient Transfer Learning Approaches on SURE Benchmark for Speech Understanding},
  author={Li, Yingting and Mehrish, Ambuj and Zhao, Shuai and Bhardwaj, Rishabh and Zadeh, Amir and Majumder, Navonil and Mihalcea, Rada and Poria, Soujanya},
  booktitle={ICASSP},
  year={2023}
}

Note: Please cite our paper if you find this repository useful.