latent action modeling

This paper addresses the fundamental question of whether latent action models (LAMs) genuinely learn action-driven inter-frame dynamics or merely capture exogenous noise. Method: We develop an analytically tractable linear system model to theoretically characterize the learning mechanism of LAMs, uncovering their intrinsic relationship with principal component analysis (PCA) and rigorously analyzing how structural coupling among observations, actions, and noise governs model performance. Leveraging controllability theory, we derive principled guidelines for designing data generation strategies. Contribution/Results: These guidelines inform video data augmentation, noise denoising, and auxiliary action prediction. Numerical simulations demonstrate that our strategy significantly enhances learning of action-relevant features, thereby advancing the interpretability and reliability of unsupervised action representation learning.

This work addresses the challenge that existing latent action models struggle to capture long-term temporal structures and high-level skills in videos lacking explicit action labels. To overcome this limitation, the paper proposes a Hierarchical Latent Action Model that introduces a novel hierarchical architecture: it leverages a pretrained low-level latent action model to extract fundamental dynamic patterns and employs a sequential aggregation mechanism to automatically discover high-level latent skills. This design enables the model to effectively capture long-range temporal dependencies. Experimental results demonstrate that the proposed approach significantly outperforms current baselines on dynamic skill discovery tasks, exhibiting superior robustness and enhanced capability in modeling long-horizon temporal dynamics.

This work investigates how to learn action-agnostic latent representations from large-scale unlabeled human motion videos and leverage them for vision-to-action generalization in robotic control. To this end, we establish a unified evaluation framework that integrates over one million videos, images, and robot trajectory data, enabling the first systematic assessment of general-purpose vision foundation models on physical control tasks under zero action supervision. Our experiments demonstrate that such models significantly outperform specialized embodied models. Crucially, we find that semantically abstracted latent action spaces align more closely with the true distribution of physical actions than pixel-level representations, thereby enabling more effective cross-task and cross-domain vision-to-action mapping.

This work addresses the challenge of lacking explicit action labels in in-the-wild videos by proposing an unsupervised learning framework to construct generalizable world models for agent reasoning and planning. The approach introduces spatially localized, continuously constrained latent action representations, enabling self-supervised learning of action-state dynamics from diverse real-world videos without requiring a unified embodied structure. By integrating continuous latent action modeling, a controller mapping mechanism, and tailored architectural design, this framework is the first to successfully extend latent-action world models to complex in-the-wild scenarios. Experiments demonstrate that the model achieves performance on par with baselines using ground-truth action labels in cross-video action transfer and planning tasks, validating the effectiveness and scalability of latent actions as a universal interface.

This work addresses the limitations of existing vision-language-action models, which suffer from scarce action-labeled data and temporal inconsistency due to error accumulation in recursive latent action composition. The authors propose a distributed latent action modeling approach that represents state transitions as diagonal Gaussian distributions, constraining their means through reference frame reconstruction. They introduce normalized triplet composition and inversion operations to jointly regularize both mean and variance, and model dependencies between adjacent transitions via shared correlation coefficients. Combined with a mean-inversion and variance-preservation mechanism, this design enhances temporal consistency. The method achieves superior direct and cumulative reconstruction on unseen videos and significantly outperforms baselines on MetaWorld MT50, LIBERO, and real-world robotic tasks. Ablation studies reveal that mean constraints primarily drive reconstruction gains, while variance and correlation modeling further improve control performance.