Flow and Depth Assisted Video Prediction with Latent Transformer

2025-11-20

AI Summary

This work addresses two key challenges in video prediction: occlusion handling and background motion modeling. Methodologically, it proposes a latent-variable Transformer framework integrated with geometric priors—specifically, point flow and depth maps—to explicitly encode motion structure and scene geometry as conditional inputs; additionally, it introduces a Wasserstein distance loss grounded in object masks to enforce consistency between predicted and ground-truth motion distributions. The core contribution lies in the principled fusion of dense geometric cues with probabilistic latent modeling, substantially improving joint representation of dynamic occlusions, complex motions, and background variations. Extensive experiments on both synthetic and real-world benchmarks demonstrate that the method achieves superior motion prediction accuracy and background consistency under occlusion compared to state-of-the-art approaches.

Technology Category

• Computer Vision: Motion & Tracking
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Video prediction is a fundamental task for various downstream applications, including robotics and world modeling. Although general video prediction models have achieved remarkable performance in standard scenarios, occlusion is still an inherent challenge in video prediction. We hypothesize that providing explicit information about motion (via point-flow) and geometric structure (via depth-maps) will enable video prediction models to perform better in situations with occlusion and the background motion. To investigate this, we present the first systematic study dedicated to occluded video prediction. We use a standard multi-object latent transformer architecture to predict future frames, but modify this to incorporate information from depth and point-flow. We evaluate this model in a controlled setting on both synthetic and real-world datasets with not only appearance-based metrics but also Wasserstein distances on object masks, which can effectively measure the motion distribution of the prediction. We find that when the prediction model is assisted with point flow and depth, it performs better in occluded scenarios and predicts more accurate background motion compared to models without the help of these modalities.

Problem

• Video prediction struggles with occlusion and background motion
• Explicit motion and geometric information improves occluded prediction
• Systematic study of depth and flow assisted video prediction

Innovation

• Incorporates point-flow for motion information
• Integrates depth-maps for geometric structure
• Uses latent transformer for occluded video prediction