FillGS: Filling Observation Gaps in 4D Gaussian Splatting via Viewpoint-Time Selection and Generative Refinement

2026-07-31

AI Summary

This work addresses the challenge of sparse spatio-temporal observations in 4D Gaussian splatting, which arises from limited viewpoint coverage and often leads to rendering artifacts—particularly in scenes with large motion. To mitigate this issue, the authors propose a motion-aware active virtual viewpoint selection mechanism that leverages both rendering sensitivity analysis and observation density estimation to generate critical virtual views that enrich under-sampled regions. Additionally, a reliable region filtering strategy is introduced to fine-tune the 4D Gaussian model using only trustworthy pixels. Evaluated on multi-view video benchmarks, the proposed method significantly outperforms existing approaches, effectively suppressing artifacts and achieving consistent improvements in both qualitative and quantitative metrics.

Abstract

4D Gaussian Splatting (4DGS) can render dynamic scenes photorealistically. However, with limited viewpoint coverage, some spatiotemporal regions remain sparsely observed, leading to artifacts, particularly in scenes with large motion. Existing approaches leveraging generative models rely on heuristic virtual-viewpoint selection before refining rendered views. As a result, they cannot actively explore such sparsely observed regions. To address this issue, we propose a pipeline that actively selects spatiotemporal virtual viewpoints to improve 4DGS reconstruction. Our method selects virtual viewpoints for generative enhancement based on the rendering sensitivity and motion-aware observation density of 4D Gaussians, prioritizing views that alleviate observation sparsity. In the refined images, we filter out regions that conflict with captured observations or are likely to contain generative artifacts and then fine-tune 4DGS using only the reliable regions. We evaluate our method on multi-view video benchmarks using new train/test splits designed to induce observation gaps. Results show consistent improvements over prior viewpoint selection strategies and fine-tuning methods in both qualitative and quantitative evaluations, while reducing artifacts.

Problem

• 4D Gaussian Splatting
• observation gaps
• spatiotemporal sparsity
• dynamic scene rendering
• viewpoint coverage

Innovation

• 4D Gaussian Splatting
• viewpoint-time selection
• generative refinement
• observation gap filling
• motion-aware density