Divide-and-Conquer: Dual-Hierarchical Optimization for Semantic 4D Gaussian Spatting

2025-03-24

AI Summary

Existing static 4D Gaussian reconstruction methods suffer from temporal feature loss, dynamic-region distortion, and severe noise in dynamic scenes. To address these issues, this paper proposes the Dual-Hierarchical Optimization (DHO) framework—the first semantic-aware 4D Gaussian dynamic modeling approach. DHO decouples static and dynamic Gaussian splats, introduces hierarchical Gaussian optical flow modeling, and incorporates semantic-guided constraints, coupled with temporal consistency regularization and joint semantic-geometric optimization. This significantly improves fidelity and texture detail in dynamic regions. Evaluated on both synthetic and real-world datasets, DHO achieves PSNR gains of 2.1–3.8 dB over state-of-the-art baselines. Moreover, it enables high-fidelity novel-view synthesis, semantic segmentation, and motion prediction—demonstrating strong generalizability to downstream tasks. By unifying semantics and dynamics within a differentiable 4D Gaussian representation, DHO establishes a new paradigm for semantic 4D reconstruction of dynamic scenes.

Technology Category

• Computer Vision: 3D Computer Vision
• Search and Optimization: Sampling/Simulation-based Search
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Semantic 4D Gaussians can be used for reconstructing and understanding dynamic scenes, with temporal variations than static scenes. Directly applying static methods to understand dynamic scenes will fail to capture the temporal features. Few works focus on dynamic scene understanding based on Gaussian Splatting, since once the same update strategy is employed for both dynamic and static parts, regardless of the distinction and interaction between Gaussians, significant artifacts and noise appear. We propose Dual-Hierarchical Optimization (DHO), which consists of Hierarchical Gaussian Flow and Hierarchical Gaussian Guidance in a divide-and-conquer manner. The former implements effective division of static and dynamic rendering and features. The latter helps to mitigate the issue of dynamic foreground rendering distortion in textured complex scenes. Extensive experiments show that our method consistently outperforms the baselines on both synthetic and real-world datasets, and supports various downstream tasks. Project Page: https://sweety-yan.github.io/DHO.

Problem

• Dynamic scene reconstruction with semantic 4D Gaussians
• Mitigating artifacts in dynamic and static Gaussian interactions
• Improving rendering distortion in complex textured scenes

Innovation

• Dual-Hierarchical Optimization for dynamic scenes
• Hierarchical Gaussian Flow divides static and dynamic features
• Hierarchical Gaussian Guidance reduces rendering distortion