DLAM: Distributional Latent Actions with Temporal Constraints

2026-07-29

AI Summary

This work addresses the limitations of existing vision-language-action models, which suffer from scarce action-labeled data and temporal inconsistency due to error accumulation in recursive latent action composition. The authors propose a distributed latent action modeling approach that represents state transitions as diagonal Gaussian distributions, constraining their means through reference frame reconstruction. They introduce normalized triplet composition and inversion operations to jointly regularize both mean and variance, and model dependencies between adjacent transitions via shared correlation coefficients. Combined with a mean-inversion and variance-preservation mechanism, this design enhances temporal consistency. The method achieves superior direct and cumulative reconstruction on unseen videos and significantly outperforms baselines on MetaWorld MT50, LIBERO, and real-world robotic tasks. Ablation studies reveal that mean constraints primarily drive reconstruction gains, while variance and correlation modeling further improve control performance.

Abstract

Vision-language-action (VLA) models remain constrained by scarce action-labeled robot data, whereas action-free videos offer abundant observations of physical change. Latent action models can extract such priors, but reconstruction-trained codes may predict future observations without the structure required for joint generation with robot actions. Existing structured methods add temporal constraints but retain deterministic transition points, so residual errors in locally inferred transitions may propagate and compound under recursive composition. We introduce DLAM, a distributional latent-action model that represents each transition as a diagonal Gaussian. Reconstruction conditioned on the reference frame grounds the mean in observed visual change, while normalized composition and reversal over equal-gap triplets constrain both the mean and dimension-wise variance. Variance composition uses a lightweight shared-correlation coefficient to account for dependence between adjacent transitions that share an intermediate frame, whereas reversal negates the mean and preserves the variance. For downstream policy learning, we freeze the encoder and train a flow-matching policy to jointly generate mean transition sequences and robot actions. On held-out transitions, DLAM learns more temporally consistent latent dynamics than existing latent-action baselines and achieves stronger direct and cumulative reconstruction on held-out videos. Under the same controlled $π_0$ transfer protocol, it also improves policy performance on MetaWorld MT50, LIBERO, and real-world manipulation tasks. Controlled ablations show that normalized mean constraints account for most of the reconstruction gain, while learned variance and correlation-aware composition provide complementary improvements in downstream control.

Problem

• latent actions
• temporal constraints
• distributional representation
• action-free videos
• structured dynamics

Innovation

• distributional latent actions
• temporal constraints
• flow-matching policy
• variance composition
• action-free video priors