Latent Riemannian Flow Matching for Geometry-Grounded 3D Foundation Models

2026-07-21

AI Summary

This work addresses the limitation of existing geometric foundation models in generating plausible 3D structures beyond the observed input views, a challenge exacerbated by the weak geometric priors inherent in conventional approaches. The authors propose the first method to introduce Riemannian flow matching on the product hyperspherical manifold—the latent space of the geometric foundation model VGGT—leveraging its learned 3D priors for coherent scene generation without relying on explicit representations such as meshes or Gaussians. By freezing the decoder-compatible latent space, aligning multi-scale encoder features, and applying Riemannian flow matching, the approach significantly outperforms recent baselines on RealEstate10K, ScanNet++, and ETH3D benchmarks, achieving state-of-the-art performance in both single-view appearance fidelity and overall 3D geometric quality.

Abstract

Geometric foundation models, such as the Visual Geometry Grounded Transformer (VGGT), provide strong 3D priors from unposed images. However, such models operate purely in a feed-forward, deterministic regime, \ie~they cannot generate plausible geometry beyond what the input views directly support. Generative models for 3D scenes, on the other hand, must rely on strong geometric priors to produce coherent outputs from sparse inputs. We bridge these two paradigms by performing flow matching directly in VGGT's latent space, leveraging its learned 3D priors without committing to any explicit downstream representation such as Gaussians, meshes, or video-VAE latents. This requires respecting the latent geometry: VGGT tokens occupy a product of high-dimensional hyperspheres on which standard Euclidean flow matching fails. We address this with a Riemannian Flow Matching framework defined on a product manifold of four hyperspheres, aligned with VGGT's multi-scale encoder, which keeps generated tokens on the valid data manifold required by the frozen decoding heads. On RealEstate10K, ScanNet++ and ETH3D, our method achieves strong performance against recent scene generation baselines in both per-view appearance and aggregated 3D geometry, establishing latent-space flow matching on geometric foundation models as a viable paradigm for 3D generation. The project page can be found $\href{https://lisaweijler.github.io/geometry-grounded-rfm/}{\text{here}}$.

Problem

• 3D generation
• geometric foundation models
• latent space
• flow matching
• Riemannian geometry

Innovation

• Riemannian Flow Matching
• Latent Space Generation
• Geometric Foundation Models
• Manifold-Constrained Learning
• 3D Scene Synthesis