🤖 AI Summary
Traditional novel view synthesis relies on external Structure-from-Motion (SfM) tools (e.g., COLMAP), which introduce pose estimation errors and computational bottlenecks. To address this, we propose the first end-to-end framework that jointly optimizes 3D Gaussian splatting radiance fields and camera poses. Our method decouples optimization into two alternating stages—Gaussian parameter refinement and pose fine-tuning—guided by differentiable rendering and a geometric-photometric consistency constraint enforced via 3D optical flow. This design significantly suppresses projection errors, especially under large parallax and sparse-texture conditions. Evaluated on multiple standard benchmarks, our approach not only substantially outperforms existing COLMAP-free methods but also surpasses the COLMAP+3DGS baseline across both reconstruction quality and camera pose accuracy.
📝 Abstract
Traditional novel view synthesis methods heavily rely on external camera pose estimation tools such as COLMAP, which often introduce computational bottlenecks and propagate errors. To address these challenges, we propose a unified framework that jointly optimizes 3D Gaussian points and camera poses without requiring pre-calibrated inputs. Our approach iteratively refines 3D Gaussian parameters and updates camera poses through a novel co-optimization strategy, ensuring simultaneous improvements in scene reconstruction fidelity and pose accuracy. The key innovation lies in decoupling the joint optimization into two interleaved phases: first, updating 3D Gaussian parameters via differentiable rendering with fixed poses, and second, refining camera poses using a customized 3D optical flow algorithm that incorporates geometric and photometric constraints. This formulation progressively reduces projection errors, particularly in challenging scenarios with large viewpoint variations and sparse feature distributions, where traditional methods struggle. Extensive evaluations on multiple datasets demonstrate that our approach significantly outperforms existing COLMAP-free techniques in reconstruction quality, and also surpasses the standard COLMAP-based baseline in general.