Full-Duplex Spoken Language Models (FD-SLMs)

This is an evolving Github repository for the paper: From Turn-Taking to Synchronous Dialogue: A Survey of Full-Duplex Spoken Language Models, which is under review at ICASSP 2026. In this paper, we survey the field of Full-Duplex Spoken Language Models (FD-SLMs), which enable synchronous human–AI dialogue via simultaneous speaking and listening, achieving a more realistic human-computer interaction experience.

---

This is a survey, but more than a survey —— due to ICASSP's page limitation, we have omitted and abbreviated many technical details in the paper, which are highly valuable for guiding the future implementation of a Full-Duplex Spoken Language Model for practical production. Therefore, we will continue to update relevant content at this link.

If you find any mistakes, please don’t hesitate to open an issue, or contact to yxchen5522@mails.jlu.edu.cn directly.

---

Introduction

---

Background

---

Taxonomy

Note: A modular implementation does not necessarily imply plug-and-play compatibility with other SLMs—for example, VITA-1.5 and Freeze-Omni. They are end-to-end models and can only be integrated as a whole.

---

Existing Works

In this section, we will list all existing papers on full-duplex SLMs, covering both models and benchmarks.

1. Models

1.1 Learned Synchronization ( End-to-End ) :

1.2 Engineered Synchronization ( Modular ) :

1.3 Pseudo Full-Duplex :

1.4 Non-independent Models :

We define non-independent models as either prior or subsequent works from the same author team of an existing model, or fine-tuned variants built upon existing full-duplex models.

2. Benchmarks

---

Model Structure (only for e2e) :

After this, let's assume we set aside the issue of Transformer, no matter how it might be implemented—such as with a dual-tower architecture (dGSLM) or token interleaving (NTPP). In an end-to-end implementation solution, we must answer another fundamental question: who serves as the system's clock for perceiving the external world?

Some may ask: traditional SLMs don't incorporate clocks, yet they still function properly. In fact, it is not that they lack this ability, but rather that they employ a more subtle method, which is the more familiar turn-taking in conversation.

---

Training Datasets

We have compiled as comprehensive a list as possible of all existing datasets available for full-duplex training and provided the methods for obtaining them.

---

Training Strategy

---

Our Benchmark

Based on FD-Bench and Full-Duplex-Bench (v1.5)—especially the latter, for which we extend special thanks to Professor Hung-yi Lee—we have developed an even more convenient benchmark built upon the engineering details of the ICASSP HumDial Challenge. Our goal is to enable as close to one-click evaluation of your model as possible and ultimately provide a quantifiable score. We name this benchmark Badcat. For details, please refer to Badcat-Benchmark/README.md.

---

Pre-LLM Era Related Research

We believe that in the age of AI, it is more important than ever to honor the foundational work of our predecessors—whose ideas can be revitalized and find new life in the era of large language models, much like how LSTM once revolutionized NLP. In the pre-LLM era, the Spoken Dialogue Systems (SDS) community had long been exploring full-duplex interaction. Therefore, we list a selection of representative works from this line of research and provide brief summaries. We encourage readers to consult the original papers to fully grasp the authors’ ideas.

---

Safety and Security

Ultra-low latency is a core goal of Full-Duplex Spoken Language Models (FD-SLMs), but it also shrinks the time budget for safety filtering. In sequential pipelines, moderation can be applied before output is finalized; in full-duplex, the system may already be speaking while the semantic intent is still unfolding. This creates distinct safety and security risks that must be addressed architecturally, not only via post-hoc policy text.

Why Safety Becomes Harder in Full-Duplex

Full-duplex systems aim for "cognitive parallelism"—listening, reasoning, and speaking overlap. The same property that improves responsiveness also introduces:

• Reduced lookahead: harmful intent may appear late in an utterance, after generation has started.
• Streaming commitments: audio emission is irreversible once played (especially for real-time voice agents).
• Cross-channel prompt injection: adversarial speech can be injected via audio (including instructions hidden in longer contexts).
• Latency–quality–safety tri-lemma: stronger moderation typically costs compute and time; weak moderation risks unsafe content.

This repository encourages treating safety as a control loop that runs alongside generation, rather than a single gate at the end.

We focus on issues most relevant to FD-SLMs:

1. Toxic / hateful / harassing speech emitted with low delay.
2. Self-harm facilitation or encouragement in real-time conversational settings.
3. Illicit instructions (e.g., wrongdoing enablement) arising during incremental generation.
4. PII leakage in open-ended dialogue (names, addresses, contact info).
5. Audio prompt injection / instruction hijacking from untrusted speakers or media.

---

Architectural Mitigations

Below are implementation patterns that preserve low latency while enabling safety intervention. They are presented in a way that applies to both Engineered Synchronization (modular) and Learned Synchronization (end-to-end) designs.

1) Micro-Buffer + Streaming Moderatio

Introduce a small, fixed output buffer (e.g., 200–500 ms) and run moderation on a sliding window of partial text/audio.

This is a simple and efficient implementation solution. Its advantage is minimal added latency and allows pre-emission filtering. However, at the same time, it cannot catch everything if the buffer is too small and requires a extra streaming classifier.

How it works

• Generator emits partial text (or intermediate semantic units).
• A lightweight safety model scores the window.
• If unsafe risk exceeds a threshold, the system:
  • blocks/halts the next audio chunk,
  • replaces with a neutral fallback,
  • optionally asks for clarification.

This is a good default for both synchronization paradigms because it does not require internal access to model weights.

2) Dual-Track Generation: Fast Speaker + Slow Verifier

Run two concurrent tracks:

• Fast track produces a candidate response quickly.
• Verifier track performs deeper checks (policy compliance, PII, sensitive topics) with slightly higher latency.

Only release audio that is either:

• verified safe, or
• safe-by-construction (templates, constrained replies) while verifier catches up.

This is particularly effective for learned synchronization models where internal states are opaque: you can enforce safety at the release stage without modifying the core model.

3) Interruptible Synthesis

Design the speech output stack to be chunked and cancellable. Synthesize in short frames (e.g., 100–200 ms), and maintain an interrupt path that can immediately:

• stop emission,
• duck volume,
• crossfade to a neutral acknowledgement (“Sorry, I can’t help with that.”),
• request rephrase.

This is essential in FD settings because unsafe content may be detected after generation begins.

Mapping to Synchronization Strategies

• Engineered Synchronization (Modular):

  • Place moderation at multiple boundaries: ASR text, dialogue state, planned response, and TTS chunks.
  • Use explicit control signals (pause/interrupt/backchannel) as part of the turn-taking controller.

• Learned Synchronization (End-to-End):

  • Treat the end-to-end model as a fast proposal generator.
  • Enforce safety at release time via micro-buffer moderation + interruptible synthesis.
  • Prefer dual-track verification because internal decoding states may be inaccessible.

---

Citation

If you find our survey useful for your research, please 📚cite📚 the following paper:

@article{chen2025FD-SLMs,
  title={From Turn-Taking to Synchronous Dialogue: A Survey of Full-Duplex Spoken Language Models},
  author={Yuxuan Chen, Haoyuan Yu}
  journal={arXiv preprint arXiv:2509.14515},
  year={2025}
}

---

Change log

Update (October 31)

This repository will now receive regular updates to maintain the most current survey content.

I recently participated in the ICASSP HumDial Challenge, which temporarily delayed updates to this repository. However, this experience provided valuable insights into full-duplex implementation, particularly regarding modular approaches.

Additionally, several recent full-duplex papers have been published, such as FLM-Audio[2509.02521], and will be incorporated into upcoming updates.