LARA: Latent Action Representation Alignment for Vision-Language-Action Models

2026-06-05

AI Summary

This work addresses the challenge that vision-language-action (VLA) models struggle to leverage the vast amount of unlabeled human videos due to the scarcity of real robot action data. To overcome this limitation, the authors propose LARA, a framework that enables the first end-to-end joint training of a latent action model (LAM) and a VLA model. Through a representation alignment mechanism, the two components mutually enhance each other: the LAM leverages action trajectories to filter out irrelevant visual variations, while its learned forward dynamics constrain the VLA to suppress implausible predictions. The approach supports multiple training paradigms—including pre-training, post-training enhancement, and LAM refinement—and achieves consistent performance gains, improving average results by approximately 10%, 5%, and 15% across three simulated and one real-world robotic manipulation benchmark, respectively.

Abstract

Visual-language action (VLA) models enable robots to predict actions directly from observations and language instructions, but their performance depends on large-scale, high-quality data and is limited by the scarcity of real-world robot action datasets. To facilitate VLA model learning with abundant unlabeled human videos, Latent Action Models (LAM) learn latent action representations from visual dynamics to provide additional supervision for VLA learning. However, LAM and VLA are typically trained separately, leaving LAM ungrounded during VLA training and VLA models constrained by frozen LAM representations. To address these issues, we propose Latent Action Representation Alignment (LARA), a plug-and-play framework that jointly optimizes LAM and VLA via representation alignment. This enables reciprocal benefits where LAMs learn with action trajectories to avoid spurious visual changes, while VLAs are regularized by forward dynamics learned within LAMs to reduce hallucinations of functionally ineffective trajectories. We demonstrate LARA versatility and effectiveness for pre-training, post-training enhancement of pre-trained VLA models, and LAM refinement, achieving an average of ~10%, ~5%, and ~15% improvement over 3 simulation and 1 meticulously designed real-world robotic manipulation benchmarks.

Problem

• Vision-Language-Action Models
• Latent Action Representation
• Representation Alignment
• Robot Learning
• Unlabeled Human Videos

Innovation

• Latent Action Representation Alignment
• Vision-Language-Action Models
• Representation Alignment
• Robot Learning
• Action Prediction