Struct-GStream: Towards Efficient Free-Viewpoint Video Streaming at Low-Bitrates with Structured 3D Gaussians

2026-08-02

AI Summary

This work addresses the challenge of achieving fast training, low storage overhead, and high-quality real-time rendering simultaneously in neural rendering for free-viewpoint video streams. To this end, the authors propose Struct-GStream, which models static scenes using a structured 3D Gaussian representation and efficiently handles dynamic content through a dynamic anchor mechanism. The method integrates a global generate-prune-optimize strategy for free 3D Gaussians with motion modeling based on local rigidity assumptions, effectively inpainting dynamic objects and missing regions. Experimental results demonstrate that Struct-GStream significantly outperforms existing online methods in terms of training speed, storage efficiency, and rendering quality, while maintaining real-time rendering capabilities.

Abstract

Constructing photorealistic Free-Viewpoint Videos (FVVs) of dynamic scenes from a set of posed 2D images has been an intriguing yet challenging task in computer vision. Methods based on neural rendering achieve high-fidelity image quality in FVV construction. However, most of these methods are unable to achieve real-time rendering and often require complete video sequences to train. Despite the existence of some online training methods capable of rendering FVVs in real time, they struggle to meet the requirements for storage and training time for downstream applications. To overcome this problem, we propose Struct-GStream, which can achieve efficient FVV streaming using structured 3D Gaussians (3DGs). Specifically, we introduce dynamic anchor points to generate structured 3DGs to construct basic scenes and model approximate scene movements based on the assumption of local rigidity in object motion. Besides, we introduce a global free 3DGs patching strategy involving free 3DGs' generation, pruning, and optimization to patch and model deficient areas and emerging objects. Our method achieves fast training at low bitrates while maintaining high rendering quality. Extensive experiments demonstrate that Struct-GStream significantly outperforms existing online training methods for FVV construction in terms of training time, storage, and rendering quality while maintaining competitive rendering speed.

Problem

• Free-Viewpoint Video
• low-bitrate streaming
• real-time rendering
• efficient training
• dynamic scenes

Innovation

• structured 3D Gaussians
• free-viewpoint video streaming
• dynamic anchor points
• online training
• low-bitrate rendering