RelayS2S: A Dual-Path Speculative Generation for Real-Time Dialogue

2026-03-24

AI Summary

This work addresses the fundamental trade-off in real-time spoken dialogue systems between low latency and high semantic quality: end-to-end speech-to-speech (S2S) models offer rapid responses but limited linguistic depth, whereas cascaded ASR+LLM pipelines yield semantically rich outputs at the cost of higher latency. To reconcile this tension, the authors propose a novel dual-path parallel architecture that introduces speculative generation to real-time spoken interaction. Upon detecting a speaker turn, the system concurrently executes a fast path (S2S) to generate an immediately synthesizable response prefix and a slow path (ASR+LLM) to produce high-quality continuation content, dynamically fusing both streams via a lightweight learned verifier. Experiments demonstrate that this approach achieves P90 audio onset latency comparable to pure S2S models while preserving 99% of the average response quality of the cascaded system, with performance gains amplifying as the slow-path model scales.

Technology Category

• Natural Language Processing: Conversational AI/Dialog Systems
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

Real-time spoken dialogue systems face a fundamental tension between latency and response quality. End-to-end speech-to-speech (S2S) models respond immediately and naturally handle turn-taking, backchanneling, and interruption, but produce semantically weaker outputs. Cascaded pipelines (ASR -> LLM) deliver stronger responses at the cost of latency that grows with model size. We present RelayS2S, a hybrid architecture that runs two paths in parallel upon turn detection. The fast path -- a duplex S2S model -- speculatively drafts a short response prefix that is streamed immediately to TTS for low-latency audio onset, while continuing to monitor live audio events. The slow path -- a cascaded ASR -> LLM pipeline -- generates a higher-quality continuation conditioned on the committed prefix, producing a seamless utterance. A lightweight learned verifier gates the handoff, committing the prefix when appropriate or falling back gracefully to the slow path alone. Experiments show that RelayS2S achieves P90 onset latency comparable to the S2S model while retaining 99% cascaded response quality in average score, with benefits growing as the slow-path model scales. Because the prefix handoff requires no architectural modification to either component, RelayS2S serves as a lightweight, drop-in addition to existing cascaded pipelines. Our code and data are publicly available at: https://github.com/mailong25/relays2s

Problem

• latency
• response quality
• speech-to-speech
• real-time dialogue
• turn-taking

Innovation

• speculative generation
• dual-path architecture
• real-time dialogue
• speech-to-speech translation
• latency-quality tradeoff