DualTurn: Learning Turn-Taking from Dual-Channel Generative Speech Pretraining

Submitted to Interspeech 2026 | Paper (arXiv) | Checkpoints (HuggingFace)

Official implementation of DualTurn, a turn-taking prediction model for voice agent pipelines. DualTurn uses generative pretraining on dual-channel conversational audio to learn turn-taking dynamics, then predicts interpretable signals that map directly to agent actions.

Highlights

• Generative pretraining on dual-channel audio teaches the model conversational dynamics without any labels
• 12 classification heads predict six per-channel signals: EOT, HOLD, BOT, BC, VAD, FVAD
• 5 agent actions derived from signals: Start-Talking, Continue-Listening, Start-Listening, Continue-Talking, Backchannel
• 0.5B parameters (9M trainable via LoRA), runs on a single CPU at 78ms latency
• Outperforms VAP and 3.1B audio+text models on standard benchmarks

Results

Table 1: Agent Action Prediction (Switchboard 138 sessions)

Table 2: VAP Protocol Frame-Level (Switchboard)

Table 3: Word-Level Turn Prediction (Switchboard)

Installation

git clone https://github.com/anyreach-ai/dualturn.git
cd dualturn
pip install -e .

Data Preparation

DualTurn uses two datasets:

• otoSpeech (289h, 1,125 conversations) -- gated dataset on HuggingFace, requires access approval
• Switchboard (220h, 2,459 conversations) -- requires LDC license

otoSpeech Access

otoSpeech is a gated dataset. To download it:

1. Request access at otoearth/otoSpeech-full-duplex-280h on HuggingFace
2. Once approved, create a HuggingFace access token at huggingface.co/settings/tokens
3. Set your token:

   # Option A: Copy .env.example and fill in your token
   cp .env.example .env
   # Edit .env and set HF_TOKEN=hf_your_token_here
   
   # Option B: Set environment variable directly
   export HF_TOKEN=hf_your_token_here
   
   # Option C: Login via CLI
   huggingface-cli login
   

Quick Start

bash scripts/prepare_data.sh

Already have processed data elsewhere? The configs assume data lives at data/ inside the repo root. If your data is in a different location you have two options:

Option A — symlink (simplest, zero config changes):

ln -s /path/to/your/data data

Option B — --data-root flag (no filesystem changes):

python train.py --config configs/stage1/pretrain_audio.yaml --data-root /path/to/your/data

This remaps all data/... paths in the config to /path/to/your/data/... without touching any files.

Step-by-Step

# Step 1: Download otoSpeech (requires HF access, see above)
python -m dualturn.data.download_otospeech --output-dir data/otospeech/audio

# Step 2: Extract metadata
python -m dualturn.data.enumerate_otospeech

# Step 3: Process audio (Mimi encoding + Silero VAD)
python -m dualturn.data.process_otospeech
python -m dualturn.data.process_switchboard

# Step 4: Extract continuous Mimi encoder features
python -m dualturn.data.extract_mimi_features --dataset otospeech
python -m dualturn.data.extract_mimi_features --dataset switchboard

# Step 5: Generate labels
python -m dualturn.data.add_start_end_labels data/otospeech_processed_npy
python -m dualturn.data.add_start_end_labels data/switchboard_processed_npy
python -m dualturn.data.relabel_context_aware --dataset both

# Step 6: Create train/val/test splits (otoSpeech + Switchboard)
python -m dualturn.data.create_splits --dataset both

Switchboard Setup

Place Switchboard audio and transcripts as follows:

data/switchboard/
  audio_wav/           # Mono WAVs: sw02001A.wav, sw02001B.wav, ...
  transcripts/         # Word-level alignments (for Table 3 evaluation)

Training

Stage-1: Generative Speech Pretraining

Trains the backbone to predict next audio tokens for both speakers simultaneously.

# Audio-only pretraining (for models A, B, C, E, F)
bash scripts/train_stage1.sh

# Text-aware pretraining (for model G only)
bash scripts/train_stage1.sh --text

Or run individual phases:

python train.py --config configs/stage1/pretrain_audio.yaml      # LoRA phase
python train.py --config configs/stage1/pretrain_audio_ft.yaml   # Full fine-tune phase

Stage-2: Turn-Taking Signal Prediction

Fine-tunes the pretrained backbone with 12 classification heads.

# Train all model variants
bash scripts/train_stage2.sh

# Train specific models
bash scripts/train_stage2.sh A E

# Train a single model
python train.py --config configs/stage2/model_A.yaml

Model Variants

Evaluation

Run all eval scripts from the repo root with python -m (this puts the repo root on sys.path so the evaluation package is importable):

cd dualturn   # repo root

Reproduce All Paper Results

bash scripts/eval_all.sh --device cuda

Individual Tables

# Table 1: Agent action prediction (5-class wF1, BC F1, Anticipation)
python -m evaluation.eval_table1 --models A B C D E F G --device cuda

# Table 2: VAP protocol frame-level (S/H, S/L, S-P, BC-P)
python -m evaluation.eval_table2 --models A E --device cuda

# Table 3: Word-level turn prediction (AUC, EER)
python -m evaluation.eval_table3 --models A --device cuda

# VAP baseline comparison
python -m evaluation.eval_vap_baseline --device cuda

Results are saved to results/table1/, results/table2/, results/table3/.

Common Options

All eval scripts share these flags:

Example — evaluate a specific checkpoint with data in a custom location:

python -m evaluation.eval_table1 --models A --device cuda \
  --data-dir /path/to/data \
  --checkpoint /path/to/checkpoints/model_A/best.pt

# Table 3 also needs transcripts (word-level alignments from LDC Switchboard release)
python -m evaluation.eval_table3 --models A --device cuda \
  --data-dir /path/to/data \
  --checkpoint /path/to/checkpoints/model_A/best.pt \
  --transcript-dir /path/to/switchboard/transcripts

Architecture

Audio encoding: Frozen Mimi codec encodes 24kHz dual-channel audio into continuous 512-dim features at 12.5 Hz. Channel-specific MLPs project to 896-dim, concatenated and fed to the backbone.

Backbone: Qwen2.5-0.5B with LoRA adapters (rank 16, targeting attention + MLP layers). 9M trainable parameters.

Classification heads: 12 lightweight heads (6 per channel):

• Sparse event heads (2-layer MLP): EOT, HOLD, BOT, BC
• Dense heads (linear): VAD, FVAD (4 future horizons: 240ms, 480ms, 960ms, 2s)

Agent action inference: Six per-channel signals mapped to five agent actions via logistic regression probe or heuristic thresholds.

Signal Definitions

Agent Actions

Pretrained Checkpoints

All checkpoints are available on HuggingFace: anyreach-ai/dualturn-qwen2.5-mimi-0.5B

Project Structure

dualturn/
  model/          # TurnTakingModel, backbones, embeddings, losses
  data/           # Data download, processing, labeling, dataset classes
  config/         # ExperimentConfig dataclasses
  utils/          # Trainer, checkpoint management
configs/
  stage1/         # Stage-1 pretraining configs
  stage2/         # Stage-2 model variant configs (A-G)
evaluation/       # Paper evaluation scripts (Tables 1-3, Figure 2)
baselines/vap/    # VAP baseline (vendored)
scripts/          # Shell scripts for data prep, training, evaluation
train.py          # Main training entry point

Citation

@misc{rajaa2026dualturnlearningturntakingdualchannel,
      title={DualTurn: Learning Turn-Taking from Dual-Channel Generative Speech Pretraining},
      author={Shangeth Rajaa},
      year={2026},
      eprint={2603.08216},
      archivePrefix={arXiv},
      primaryClass={eess.AS},
      url={https://arxiv.org/abs/2603.08216},
}

License

This project is licensed under the MIT License. See LICENSE for details.