OCTOPINF: Workload-Aware Inference Serving for Edge Video Analytics

2025-02-03

AI Summary

To address SLO violations in Edge Video Analytics (EVA) caused by resource constraints, network instability, and highly volatile workloads on edge devices, this paper proposes a co-optimization framework for real-time DNN inference. Our approach integrates: (1) fine-grained GPU resource allocation with spatio-temporal joint task scheduling to maximize resource colocation efficiency under strict SLO guarantees; and (2) adaptive dynamic batching coupled with multi-level, load-aware edge-cloud collaborative load balancing. Evaluated on a real-world edge platform, the framework achieves up to 10× higher throughput compared to baselines, significantly improves robustness against workload surges and weak-network conditions, and enables plug-and-play adaptation across diverse EVA tasks—without requiring model retraining or infrastructure modification.

Technology Category

• Machine Learning: Learning on the Edge & Model Compression
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems
• Computer Vision: Learning & Optimization for CV

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Cloud, edge and content delivery systems for the Web
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

Edge Video Analytics (EVA) has gained significant attention as a major application of pervasive computing, enabling real-time visual processing. EVA pipelines, composed of deep neural networks (DNNs), typically demand efficient inference serving under stringent latency requirements, which is challenging due to the dynamic Edge environments (e.g., workload variability and network instability). Moreover, EVA pipelines also face significant resource contention caused by resource (e.g., GPU) constraints at the Edge. In this paper, we introduce OCTOPINF, a novel resource-efficient and workload-aware inference serving system designed for real-time EVA. OCTOPINF tackles the unique challenges of dynamic edge environments through fine-grained resource allocation, adaptive batching, and workload balancing between edge devices and servers. Furthermore, we propose a spatiotemporal scheduling algorithm that optimizes the co-location of inference tasks on GPUs, improving performance and ensuring service-level objectives (SLOs) compliance. Extensive evaluations on a real-world testbed demonstrate the effectiveness of our approach. It achieves an effective throughput increase of up to 10x compared to the baselines and shows better robustness in challenging scenarios. OCTOPINF can be used for any DNN-based EVA inference task with minimal adaptation and is available at https://github.com/tungngreen/PipelineScheduler.

Problem

• Real-time Video Analysis
• Resource Allocation
• Deep Neural Networks

Innovation

• OCTOPINF
• GPU resource optimization
• Edge Video Analysis (EVA)