Schrödinger's Cat: Probabilistic Representation and Prediction of Potential Scene Kinematics

2026-07-28

AI Summary

Existing approaches struggle to efficiently model the multimodal distribution of future scene dynamics from partial observations. To address this challenge, this work proposes GARFIELD, a novel framework that explicitly constructs a structured spatiotemporal latent variable distribution to represent all plausible future motions conditioned on an input image and sparse constraints. By integrating object-aware latent space modeling, a deterministic density decoder, and probabilistic trajectory generation, GARFIELD enables joint sampling, local uncertainty estimation, progressive constraint fusion, and interactive density queries—without requiring large-scale video generation or Monte Carlo sampling. Experiments demonstrate that GARFIELD achieves trajectory sampling 97× faster than large video generation models and density estimation two orders of magnitude faster, while delivering competitive performance in motion planning tasks and supporting real-time interaction.

Abstract

Predicting how a scene may evolve from partial observations requires reasoning about multiple possible futures rather than committing to a single trajectory. Existing approaches either generate appearance-dominated video predictions or sample a small number of trajectories without explicitly modeling the distribution of possible motion. We introduce Goal-Aware Representations of Future kInEmatic Latent Distributions (GARFIELD), a probabilistic model of scene kinematics that learns a structured spatio-temporal latent representation of the distribution over possible futures given an image and optional spatio-temporally sparse constraints. The same latent representation enables both joint sampling of all trajectories and direct access to the underlying motion distribution through an efficient deterministic density decoder. As a result, uncertainty about future motion can be localized to specific scene elements and timesteps and progressively refined through additional constraints. Experiments demonstrate strong motion planning performance competitive with large video generation models while sampling trajectories $97\times$ faster. Our method further estimates motion densities two orders of magnitude faster than Monte-Carlo sampling from motion generation models, enabling interactive exploration and uncertainty-aware planning.

Problem

• scene kinematics
• probabilistic prediction
• future motion
• uncertainty modeling
• trajectory distribution

Innovation

• probabilistic scene prediction
• structured latent representation
• kinematic distribution modeling
• uncertainty-aware planning
• efficient trajectory sampling