OmniSense: Towards Edge-Assisted Online Analytics for 360-Degree Videos

2025-08-19

AI Summary

To address the high computational and network overheads and poor real-time performance in online 360° video analytics, this paper proposes an edge-cooperative lightweight immersive analytics framework. Our method introduces three key innovations: (1) a spherical-geometry-based lightweight region-of-interest (ROI) prediction algorithm that accurately localizes user attention areas; (2) a content- and network-aware adaptive visual model scaling mechanism that dynamically optimizes computational load; and (3) an end-to-end joint optimization strategy for coordinated edge–cloud resource scheduling. Evaluated on real-world datasets, our framework achieves, compared to state-of-the-art baselines, 19.8%–114.6% higher average detection/identification accuracy, 2.0×–2.4× faster inference speed, and significantly reduced end-to-end latency—while maintaining high fidelity—thereby enabling low-latency, high-accuracy real-time understanding of 360° video.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Machine Learning: Learning on the Edge & Model Compression
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

With the reduced hardware costs of omnidirectional cameras and the proliferation of various extended reality applications, more and more $360^circ$ videos are being captured. To fully unleash their potential, advanced video analytics is expected to extract actionable insights and situational knowledge without blind spots from the videos. In this paper, we present OmniSense, a novel edge-assisted framework for online immersive video analytics. OmniSense achieves both low latency and high accuracy, combating the significant computation and network resource challenges of analyzing $360^circ$ videos. Motivated by our measurement insights into $360^circ$ videos, OmniSense introduces a lightweight spherical region of interest (SRoI) prediction algorithm to prune redundant information in $360^circ$ frames. Incorporating the video content and network dynamics, it then smartly scales vision models to analyze the predicted SRoIs with optimized resource utilization. We implement a prototype of OmniSense with commodity devices and evaluate it on diverse real-world collected $360^circ$ videos. Extensive evaluation results show that compared to resource-agnostic baselines, it improves the accuracy by $19.8%$ -- $114.6%$ with similar end-to-end latencies. Meanwhile, it hits $2.0 imes$ -- $2.4 imes$ speedups while keeping the accuracy on par with the highest accuracy of baselines.

Problem

• Online analytics for 360-degree videos with low latency
• Reducing computation and network resource challenges
• Pruning redundant information in 360-degree video frames

Innovation

• Edge-assisted framework for online immersive video analytics
• Lightweight spherical region of interest prediction algorithm
• Smart vision model scaling with optimized resource utilization