StreamSTGS: Streaming Spatial and Temporal Gaussian Grids for Real-Time Free-Viewpoint Video

2025-11-08

AI Summary

To address the high storage overhead (~10 MB per frame) and lack of adaptive bitrate support in real-time free-viewpoint video (FVV) streaming using 3D Gaussian Splatting (3DGS), this paper proposes StreamSTGS—a spatiotemporally compact FVV representation tailored for streaming. Our method encodes 3D Gaussian parameters as 2D texture maps and models temporal deformation fields as video sequences; it further introduces a sliding-window local motion aggregation scheme and a spatiotemporal Transformer to capture global dynamics. Crucially, bitrate adaptation is achieved without retraining. Evaluated on multiple FVV datasets, StreamSTGS reduces average frame size to 170 KB—achieving over 58× compression—while improving mean PSNR by 1.0 dB over state-of-the-art methods.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Time-Series/Data Streams
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

Streaming free-viewpoint video~(FVV) in real-time still faces significant challenges, particularly in training, rendering, and transmission efficiency. Harnessing superior performance of 3D Gaussian Splatting~(3DGS), recent 3DGS-based FVV methods have achieved notable breakthroughs in both training and rendering. However, the storage requirements of these methods can reach up to $10$MB per frame, making stream FVV in real-time impossible. To address this problem, we propose a novel FVV representation, dubbed StreamSTGS, designed for real-time streaming. StreamSTGS represents a dynamic scene using canonical 3D Gaussians, temporal features, and a deformation field. For high compression efficiency, we encode canonical Gaussian attributes as 2D images and temporal features as a video. This design not only enables real-time streaming, but also inherently supports adaptive bitrate control based on network condition without any extra training. Moreover, we propose a sliding window scheme to aggregate adjacent temporal features to learn local motions, and then introduce a transformer-guided auxiliary training module to learn global motions. On diverse FVV benchmarks, StreamSTGS demonstrates competitive performance on all metrics compared to state-of-the-art methods. Notably, StreamSTGS increases the PSNR by an average of $1$dB while reducing the average frame size to just $170$KB. The code is publicly available on https://github.com/kkkzh/StreamSTGS.

Problem

• Reducing storage requirements for real-time free-viewpoint video streaming
• Compressing 3D Gaussian representations for efficient transmission
• Enabling adaptive bitrate control without additional training overhead

Innovation

• Representing dynamic scenes with canonical Gaussians and deformation fields
• Encoding Gaussian attributes as compressed 2D images and video
• Using sliding window and transformer modules for motion learning