FD-Bench: A Full-Duplex Benchmarking Pipeline Designed for Full Duplex Spoken Dialogue Systems

📖 Abstract

FD-Bench is a comprehensive benchmarking pipeline specifically designed for evaluating Full-Duplex Spoken Dialogue Systems (FD-SDS). This benchmark provides standardized metrics and evaluation protocols to assess the performance of conversational AI systems in real-time, bidirectional communication scenarios.

🚀 Quick Links

• 🌐 Demo Page - More results
• 📄 ArXiv Paper - Detailed methodology and findings
• 🤗 Hugging Face Dataset - Download the benchmark dataset

✅ Released

• Inference sample released on the Demo website
• All benchmarking metrics for all generated datasets and all FD-SDS models we have tested
• All delay distributions for all generated datasets and all FD-SDS models we have tested
• Release of code for generating simulated TTS data
  • OpenAI API inference
  • TTS models inference
• Llama3 PPL calculation
• Dataset release to Hugging Face

🏗️ Installation

Prerequisites

• Python >= 3.8
• PyTorch >= 2.5.0
• CUDA >= 12.4

Setup

git clone https://github.com/pengyizhou/FD-Bench.git
cd FD-Bench

📊 Dataset

FD-Bench includes comprehensive evaluation data for full-duplex spoken dialogue systems:

• Objective Metrics: WER, BLEU, response time, interruption handling
• Subjective Metrics: Naturalness, coherence, turn-taking appropriateness
• Test Scenarios: Various conversation types and interruption patterns

Download Dataset

# Download from Hugging Face
git lfs install
git clone https://huggingface.co/datasets/pengyizhou/FD-Bench-Audio-Input

cd FD-Bench-Audio-Input
# Enter each subdirectory and unarchive the files
for dir in */; do
    cd "$dir"
    tar -xvf *.tar.gz
    rm *.tar.gz
    cd ..
done

🔧 Usage

Generating TTS Data

# Using ChatTTS
cd tts-generation/ChatTTS
python generate_using_chattts.py

# Using CosyVoice2
cd tts-generation/CosyVoice2
python generate_using_cosyvoice2.py

# Using F5TTS
cd tts-generation/F5TTS
python generate_using_f5tts.py

Evaluating Models using their specific inference scripts

Please refer to examples/ directory for specific model implementations. We include examples for:

• Freeze-omni: Freeze-omni Inference
• Moshi: Moshi Inference
• VITA-1.5: VITA-1.5 Inference

Running the Benchmark

Please refer to the benchmark/ directory for the main benchmarking pipeline.

📈 Results

Our benchmark evaluates various state-of-the-art FD-SDS models:

• Freeze-omni: Performance metrics and analysis
• Moshi: Objective and subjective evaluation results
• VITA-1.5: Comprehensive benchmarking scores

Detailed results and comparisons are available on our Demo Page.

🤝 Contributing

We welcome contributions! Please see our contributing guidelines for details on how to submit improvements, bug reports, or new features.

📜 License

This project is licensed under the NTUitive License - see the LICENSE.txt file for details.

📚 Citation

If you use FD-Bench in your research, please cite our paper:

@article{peng2025fdbenchfullduplexbenchmarkingpipeline,
      title={FD-Bench: A Full-Duplex Benchmarking Pipeline Designed for Full Duplex Spoken Dialogue Systems}, 
      author={Yizhou Peng and Yi-Wen Chao and Dianwen Ng and Yukun Ma and Chongjia Ni and Bin Ma and Eng Siong Chng},
      year={2025},
      eprint={2507.19040},
      archivePrefix={arXiv},
      url={https://arxiv.org/abs/2507.19040}, 
}

🙏 Acknowledgments

• Thanks to all contributors and the open-source community
• Special acknowledgments to the teams behind the evaluated models
• Funding and institutional support acknowledgments

📞 Contact

For questions or collaboration opportunities, please reach out:

• Email: yizhou004@e.ntu.edu.sg
• GitHub Issues: Submit an issue

---