Predict, Then Retrieve: Cross-Instance Future-State Retrieval from Video Prefixes

2026-08-05

AI Summary

This work proposes the Predictive State Retrieval (PSR) task, which aims to retrieve visual instances from external data that depict a specified future state of an object, given a video prefix and a temporally grounded query about that state. The authors formally define PSR and introduce a benchmark featuring difficulty tiers and human-verified ground truth. They identify the primary performance bottleneck as future state prediction rather than perceptual understanding and address it with LFTR, a lightweight retriever that leverages a frozen encoder to fuse semantic and visual-spatial matching, conditions future latent state prediction, and incorporates hard negative training. LFTR achieves superior performance over existing methods while substantially reducing inference cost. Ablation studies confirm the effectiveness of cross-space fusion and hard negative sampling. Code, benchmark, and evaluation scripts are publicly released.

Abstract

We introduce Predictive State Retrieval (PSR), a task in which a model observes a short video prefix and a temporal question about an object's future state, then retrieves instances from other videos or images that depict that state. Unlike action anticipation, which predicts a label, moment retrieval, which localizes an observed event within a video, or video generation, which synthesizes pixels, PSR combines anticipation with cross-instance retrieval across multiple temporal horizons. We construct a benchmark from four datasets with graded, human-validated ground truth, difficulty tiers, and an oracle ceiling. We also propose LFTR, a lightweight retriever with frozen encoders that predicts a question- and horizon-conditioned future latent and matches it in complementary semantic and visual spaces. A ceiling decomposition reveals a clear bottleneck: the true future state is highly retrievable once specified, whereas every predictor we evaluate, including a large multimodal language model with access to the prefix frames, remains far below the oracle. Thus, forecasting rather than perception is the central learnable challenge. LFTR narrows this gap at substantially lower inference cost, and ablations attribute its gains to cross-space fusion and hard-negative training rather than latent rollout. We release the benchmark, code, and evaluation scripts.

Problem

• Predictive State Retrieval
• future-state prediction
• cross-instance retrieval
• video understanding
• temporal reasoning

Innovation

• Predictive State Retrieval
• Cross-Instance Retrieval
• Future-State Anticipation
• Latent Space Matching
• Video Understanding