🤖 AI Summary
This study identifies three structural dialogue fractures—temporal misalignment, expressive flattening, and rigid repair—in modular Speech-to-Speech Retrieval-Augmented Generation (S2S-RAG) systems, arising from excessive component-level controllability at the expense of conversational fluidity. Using multimodal interaction experiments on production-grade systems, we integrate conversation analysis, latency-aware behavioral modeling, and architectural decoupling assessment. We establish, for the first time, that dialogue friction stems not from isolated technical flaws but from interface design mismatches across modules. Accordingly, we propose “interface orchestration” as a new paradigm—replacing conventional “component optimization”—to reframe infrastructure challenges in natural-speech AI. Our work formalizes reproducible fracture patterns and articulates architectural design principles that jointly ensure controllability and interactional fluency, providing theoretical foundations and practical guidance for next-generation spoken-language AI systems.
📝 Abstract
While voice-based AI systems have achieved remarkable generative capabilities, their interactions often feel conversationally broken. This paper examines the interactional friction that emerges in modular Speech-to-Speech Retrieval-Augmented Generation (S2S-RAG) pipelines. By analyzing a representative production system, we move beyond simple latency metrics to identify three recurring patterns of conversational breakdown: (1) Temporal Misalignment, where system delays violate user expectations of conversational rhythm; (2) Expressive Flattening, where the loss of paralinguistic cues leads to literal, inappropriate responses; and (3) Repair Rigidity, where architectural gating prevents users from correcting errors in real-time. Through system-level analysis, we demonstrate that these friction points should not be understood as defects or failures, but as structural consequences of a modular design that prioritizes control over fluidity. We conclude that building natural spoken AI is an infrastructure design challenge, requiring a shift from optimizing isolated components to carefully choreographing the seams between them.