The Model in the Middle: Toward AI-Native Real-Time Communication

2026-07-28

AI Summary

This work addresses the limitations of traditional real-time communication systems, which optimize network transmission, model inference, and user playback in isolation, thereby failing to support AI-native, full-duplex, multimodal continuous interaction. To overcome this, the paper introduces a “model-centric” paradigm that treats the AI model as a stateful computational node within the human–machine feedback loop, unifying communication, computation, and playback into a single joint control problem for the first time. The authors present Conflux, a real-time communication stack for AI-native applications, which integrates network-aware inference scheduling, execution-aware transmission prioritization, and playback control that dynamically adapts to variations in both network conditions and model behavior. Experimental results demonstrate that under degraded network conditions, Conflux significantly reduces response latency and improves deadline satisfaction rates for playback, validating the efficacy of cross-stage co-optimization.

Abstract

Full-duplex omni models are transforming human--AI interaction from turn-based exchanges into continuous multimodal conversations in which speaking, listening, and reasoning unfold concurrently. Rather than viewing the model as a replacement for a human endpoint, we argue for a new perspective: the model is a stateful computational middlebox inside a human-centered feedback loop, with network transport, model serving, and user playback jointly shaping how the interaction evolves. This perspective breaks the traditional boundaries among stages designed around local objectives. Rather than optimizing them in isolation, an AI-native real-time stack should allow the state of each stage to shape the actions of the others. We explore three cross-stage coordination opportunities: network-aware inference scheduling, execution-aware transport prioritization, and playback control that accounts for both network and model variability. We are building Conflux to explore these ideas, and preliminary results show substantial improvements in response latency and playback deadline adherence under network degradation. More broadly, we call for an AI-native real-time communication stack that resolve the joint control problem spanning communication, computation, and playback.

Problem

• AI-native
• real-time communication
• full-duplex
• joint control
• multimodal interaction

Innovation

• AI-native communication
• full-duplex omni models
• cross-stage coordination
• real-time inference
• computational middlebox