TERRA: Learning Transportable Latent Actions through Temporal Effect Representation and Relational Alignment

2026-10-07

AI Summary

This study addresses the limitations of robotic latent actions in preserving temporal dynamics and maintaining cross-state semantic consistency by proposing the TERRA framework. The method models continuous latent actions through a compact temporal effect representation and introduces an Effect-Anchored Transfer (EAT) mechanism, enabling action representations to be jointly shaped by multi-context behaviors rather than relying solely on source-state transitions, thereby achieving cross-scenario transferability. Furthermore, relation alignment techniques are integrated with a frozen linear readout for action prediction. Experimental results demonstrate that TERRA achieves an average success rate of 93.4% on the LIBERO benchmark, outperforming UniVLA while exhibiting superior robustness and generalization under visual perturbations.

Abstract

Latent actions supervise robot policies with action-like codes inferred from visual transitions, and their usefulness hinges on two questions: what a code keeps from a transition, and whether it still means the same thing when reused in a different initial state. The first is a tension in time: an endpoint difference discards how motion unfolds, while the full sequence admits nuisance variation. The second is left open by reconstruction, which only ever observes a latent together with the state it came from. We argue that both questions can be answered in the same place. TERRA (Temporal Effect Representation and Relational Alignment) describes a transition by a compact temporal effect, its net feature change together with a low-order within-window dynamics component, and learns a continuous latent from this effect. The same effect space then serves as the reference for reuse: Effect-Anchored Transport (EAT) decodes a latent in other initial states and anchors the resulting effect to the one observed at its source, so that the latent is shaped by what it does across contexts rather than only by the transition it came from. With frozen linear readers, TERRA predicts actions more accurately than UniVLA and a LAPA-style baseline, degrades more slowly under visual distractors, and keeps transported transitions faithful to the donor action as the recipient context moves farther away; a same-budget control shows that these gains come largely from EAT. At matched pretraining scale, the complete system reaches 93.4% average success on LIBERO, compared with 91.8% for UniVLA.

Problem

• latent actions
• robot policy learning
• temporal representation
• action transportability
• visual transitions

Innovation

• Latent Actions
• Temporal Effect Representation
• Effect-Anchored Transport
• Relational Alignment
• Robot Policies