Dedelayed: Deleting remote inference delay via on-device correction

2025-10-15

AI Summary

Remote inference suffers from outdated predictions due to network latency, hindering real-time applications such as autonomous driving. To address this, we propose a latency-aware correction method: a lightweight model runs locally to process the current frame, while concurrently fusing spatiotemporal features extracted by a heavy cloud model from historical frames—enabling real-time semantic segmentation without waiting for cloud responses. Crucially, our approach corrects for arbitrary round-trip latencies (e.g., 100 ms) and delivers high-accuracy outputs under delay. On BDD100K, it achieves +6.4 and +9.8 mIoU over pure edge and pure cloud baselines, respectively; performance gains scale with latency and are especially pronounced in high-motion scenarios. Notably, the method incurs zero additional latency overhead, simultaneously enhancing both real-time responsiveness and segmentation accuracy.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Learning on the Edge & Model Compression
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Remote inference allows lightweight devices to leverage powerful cloud models. However, communication network latency makes predictions stale and unsuitable for real-time tasks. To address this, we introduce Dedelayed, a delay-corrective method that mitigates arbitrary remote inference delays, allowing the local device to produce low-latency outputs in real time. Our method employs a lightweight local model that processes the current frame and fuses in features that a heavyweight remote model computes from past frames. On video from the BDD100K driving dataset, Dedelayed improves semantic segmentation accuracy over the stronger of the local-only and remote-only baselines across all realistic communication network delays beyond 33 ms. Without incurring additional delay, it improves accuracy by 6.4 mIoU compared to fully local inference and 9.8 mIoU compared to remote inference, for a round-trip delay of 100 ms. The advantage grows under longer delays and higher-motion scenes, as delay-mitigated split inference sustains accuracy more effectively, providing clear advantages for real-time tasks that must remain aligned with the current world state.

Problem

• Mitigate remote inference delays for real-time applications
• Fuse lightweight local and heavyweight remote model features
• Improve semantic segmentation accuracy under network latency

Innovation

• Lightweight local model processes current video frame
• Fuses features from remote model's past frame analysis
• Corrects arbitrary network delays for real-time inference