Recollection from Pensieve: Novel View Synthesis via Learning from Uncalibrated Videos

2025-05-19

AI Summary

Existing novel view synthesis methods heavily rely on camera calibration or geometric priors, limiting scalability to large-scale uncalibrated video data. To address this, we propose the first fully uncalibrated, purely self-supervised two-stage framework. In the first stage, scene and camera representations are jointly optimized in an implicit latent space under multi-view consistency constraints. In the second stage, 3D Gaussian primitives are introduced, integrated with differentiable Gaussian splatting rendering and a depth-projection loss to explicitly align the implicit representation with real-world 3D geometry. Our method trains end-to-end solely from raw video frames or multi-view images—without any camera intrinsics, extrinsics, or depth supervision. It achieves high-fidelity novel view synthesis and accurate camera pose estimation, matching state-of-the-art performance of strongly supervised approaches.

Technology Category

• Computer Vision: Computational Photography, Image & Video Synthesis
• Machine Learning: Unsupervised & Self-Supervised Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Security and Privacy: Data transparency and provenance
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Currently almost all state-of-the-art novel view synthesis and reconstruction models rely on calibrated cameras or additional geometric priors for training. These prerequisites significantly limit their applicability to massive uncalibrated data. To alleviate this requirement and unlock the potential for self-supervised training on large-scale uncalibrated videos, we propose a novel two-stage strategy to train a view synthesis model from only raw video frames or multi-view images, without providing camera parameters or other priors. In the first stage, we learn to reconstruct the scene implicitly in a latent space without relying on any explicit 3D representation. Specifically, we predict per-frame latent camera and scene context features, and employ a view synthesis model as a proxy for explicit rendering. This pretraining stage substantially reduces the optimization complexity and encourages the network to learn the underlying 3D consistency in a self-supervised manner. The learned latent camera and implicit scene representation have a large gap compared with the real 3D world. To reduce this gap, we introduce the second stage training by explicitly predicting 3D Gaussian primitives. We additionally apply explicit Gaussian Splatting rendering loss and depth projection loss to align the learned latent representations with physically grounded 3D geometry. In this way, Stage 1 provides a strong initialization and Stage 2 enforces 3D consistency - the two stages are complementary and mutually beneficial. Extensive experiments demonstrate the effectiveness of our approach, achieving high-quality novel view synthesis and accurate camera pose estimation, compared to methods that employ supervision with calibration, pose, or depth information. The code is available at https://github.com/Dwawayu/Pensieve.

Problem

• Enables novel view synthesis from uncalibrated videos without camera parameters
• Learns implicit 3D scene representation via self-supervised two-stage training
• Bridges latent representations to explicit 3D geometry for improved consistency

Innovation

• Two-stage training from uncalibrated videos
• Implicit scene reconstruction in latent space
• Explicit 3D Gaussian primitives refinement