Synchronization and Turn-Taking in Full-Duplex Speech Dialogue Models

2026-05-19

AI Summary

This study investigates the synchronization mechanisms and turn-taking coordination required for human-like interaction in full-duplex spoken dialogue systems. It introduces the concept of neural coupling to this domain for the first time, simulating conversations between two pretrained Moshi models and measuring cross-lagged representational synchrony using Centered Kernel Alignment (CKA). A causal LSTM is employed to extract turn-prediction signals from delayed activations. Experimental results demonstrate that, under noise-free conditions, the internal states of the models exhibit strong near-zero-lag synchrony and encode turn-taking cues in advance, enabling predictive anticipation of speaker transitions. The prediction performance degrades with increasing acoustic noise, underscoring the critical role of neural synchrony in robust interactive dialogue.

Abstract

Full-duplex spoken dialogue models (SDMs) can listen and speak simultaneously, enabling interaction dynamics closer to human conversation than turn-based systems. Inspired by neural coupling in human communication, we study how such models coordinate their internal representations during interaction. We simulate full-duplex dialogues between two instances of the pretrained \textit{Moshi} model under controlled conditions, manipulating channel noise and decoding bias. Synchronization is measured using Centered Kernel Alignment (CKA) across temporal lags, while anticipatory turn-taking cues are probed from delayed internal activations using causal LSTM models, from both speaker and listener perspectives. We find strong representational synchronization under no noise conditions, peaking near zero lag and degrading with noise, and we show that internal states encode anticipatory information that supports turn-taking prediction ahead of time.

Problem

• full-duplex
• synchronization
• turn-taking
• spoken dialogue models
• neural coupling

Innovation

• full-duplex speech dialogue
• neural coupling
• representational synchronization
• turn-taking prediction
• Centered Kernel Alignment