CAPE: Contrastive Action-conditioned Parallel Encoding for Embodied Planning

2026-06-05

AI Summary

This work addresses the challenge that existing visual dynamics models are often distracted by irrelevant visual content and struggle to accurately predict action-induced changes. To overcome this, the authors propose CAPE, a framework that leverages action-conditioned parallel encoding and latent trajectory decoding to efficiently generate a complete sequence of future states in a single forward pass. CAPE further incorporates a target-convergent contrastive learning mechanism that emphasizes outcome differences resulting from distinct actions, enabling precise long-horizon visual prediction. Evaluated on the real-world DROID dataset and zero-shot transfer tasks in RoboCasa, CAPE significantly outperforms current methods in state retrieval, offline action matching, and closed-loop planning, while substantially reducing inference overhead for long-horizon planning.

Abstract

Embodied agents need to predict the future consequences of candidate actions in order to plan effectively before execution. Existing visual dynamics models learn by reconstructing future visual states or rolling out dense latent representations, which spreads learning capacity across visually salient but planning-irrelevant content rather than the action-conditioned changes that drive manipulation outcomes. We propose CAPE, a Contrastive Action-conditioned Parallel Encoding framework that learns visual dynamics by distinguishing the future outcomes induced by different action sequences. Given an initial observation and a candidate action sequence, CAPE decodes the full future latent trajectory in a single forward pass and is trained with a Goal-Convergent Contrastive Objective that aligns predictions corresponding to the same future outcome while separating those corresponding to different outcomes. On real-world DROID and zero-shot transfer to RoboCasa, CAPE substantially outperforms prior baselines on future-state retrieval, offline action matching, and closed-loop planning, while notably reducing planning-time inference cost at long prediction horizons.

Problem

• visual dynamics
• embodied planning
• action-conditioned prediction
• future outcome prediction
• planning-irrelevant content

Innovation

• Contrastive Learning
• Action-conditioned Dynamics
• Parallel Encoding
• Embodied Planning
• Goal-Convergent Objective