Speech-IFEval: Evaluating Instruction-Following and Quantifying Catastrophic Forgetting in Speech-Aware Language Models

• Ke-Han Lu, Chun-Yi Kuan and Hung-yi Lee
• National Taiwan University
• Accepted to Interspeech 2025

• ⁉️ Most speech-aware language models (SLMs) are built from an instruction-tuned LLM, but we found they cannot follow even simple output constraints!
• 🤔 The catastrophic forgetting problem is often observed in SLM development, but we don't have an evaluation metric to measure it!

🏆 Leaderboard

Note: IFrate is the average of Closed-ended, Creative Writing, and CoT following rates.
Forgetting Rate (Δ) is computed relative to each model’s original text-only LLM.

Qwen-audio series use Qwen-7B as their backbone, which is not instruction-tuned. Therefore, no reference system is available for Δ calculation.

📬 If you have evaluated your model using Speech-IFEval, feel free to send your results to us. Once verified, we will update the leaderboard to include your entry!

---

📊 Evaluate your model

🔧 Setup

git clone https://github.com/kehanlu/Speech-IFEval.git
cd Speech-IFEval
pip install -r requirements.txt

📥 Download Audio Files

cd data
wget https://huggingface.co/datasets/kehanlu/Speech-IFEval/resolve/main/audios.tar
tar -xvf audios.tar

Directory structure:

data/
│── eval_data/
│   │── closed_ended_questions.jsonl             # Closed-ended tasks
│   │── creative_writing.jsonl                   # Creative writing tasks
│   │── chain-of-thought.jsonl                   # CoT reasoning tasks
│   │── closed_ended_questions-woprompt.jsonl    # Baseline version of closed-ended tasks (optional)
│
│── audios/
│   │── Automatic_speech_recognition/
│   │── Gender_recognition/
│   │── Speech_emotion_recognition/
│   │── MMAU/

1. Evaluate Instruction-Following Rate (IFrate)

Run your Speech-aware Language Model (SLM) evaluation (e.g., DeSTA2):

python examples/eval_desta2.py --data /lab/Speech-IFEval/data --output_dir outputs

Then compute IFrate with:

# Closed-ended and Creative Writing evaluation
python -m instruction_following_eval.evaluation_main -i outputs/DeSTA-ntu--DeSTA2-8B-beta/closed_ended_questions.jsonl
python -m instruction_following_eval.evaluation_main -i outputs/DeSTA-ntu--DeSTA2-8B-beta/creative_writing.jsonl

# Chain-of-Thought (CoT) reasoning evaluation
python script/llm_evaluation.py -i outputs/DeSTA-ntu--DeSTA2-8B-beta/chain-of-thought.jsonl --stage 0

Example Results (DeSTA2):

---

2. Evaluate Forgetting Rate (Δ)

With a reference system, we can assess the forgetting rate by comparing the speech-aware model to its text-only counterpart, thereby quantifying the degradation introduced by speech-text training.

Run the reference system baseline (e.g., Llama3-8B-Instruct for DeSTA2):

python examples/eval_llama3_8B_instruct.py --data /lab/Speech-IFEval/data --output_dir outputs

Reference System Results:

Calculate Forgetting Rate (Δ)

$$ Δ = (IFrate_{SLM} - IFrate_{Ref}) / (IFrate_{Ref}) = (89.23 - 92.53) / (92.53) = -3.57 $$

---

📌 (Optional) Task-Level Evaluation

To replicate Table 4 from the paper (with and without output constraints):

# Without constraint prompt (baseline task-level performance)
python script/llm_evaluation.py -i outputs/DeSTA-ntu--DeSTA2-8B-beta/closed_ended_questions-woprompt.jsonl --stage 0

# With constraint prompt
python script/llm_evaluation.py -i outputs/DeSTA-ntu--DeSTA2-8B-beta/closed_ended_questions.jsonl --stage 0

Citation

@article{lu2025speechifeval,
      title={Speech-IFEval: Evaluating Instruction-Following and Quantifying Catastrophic Forgetting in Speech-Aware Language Models}, 
      author={Ke-Han Lu, Chun-Yi Kuan and Hung-yi Lee},
      year={2025},
      eprint={2505.19037},
      archivePrefix={arXiv},
      primaryClass={eess.AS},
      url={https://arxiv.org/abs/2505.19037}, 
}