Act, Sense, Act: Learning Non-Markovian Active Perception Strategies from Large-Scale Egocentric Human Data

2026-02-04

AI Summary

This work addresses the challenges of robotic manipulation in unstructured environments, where perception is limited and information is inherently uncertain. The authors propose CoMe-VLA, a novel vision-language-action framework that, for the first time, formulates active perception as a non-Markovian process driven by information gain and decision branching. By integrating a cognitive auxiliary head with a dual-track memory system, the architecture enables continuous self- and environment-state awareness. Built upon large-scale human egocentric hand-eye coordination data, CoMe-VLA undergoes progressive three-stage training within a unified action space. Experimental results demonstrate that the approach significantly improves task success rates on wheeled humanoid robots in complex, long-horizon manipulation scenarios, exhibiting strong robustness and generalization capabilities.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Language and Vision
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd work
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Achieving generalizable manipulation in unconstrained environments requires the robot to proactively resolve information uncertainty, i.e., the capability of active perception. However, existing methods are often confined in limited types of sensing behaviors, restricting their applicability to complex environments. In this work, we formalize active perception as a non-Markovian process driven by information gain and decision branching, providing a structured categorization of visual active perception paradigms. Building on this perspective, we introduce CoMe-VLA, a cognitive and memory-aware vision-language-action (VLA) framework that leverages large-scale human egocentric data to learn versatile exploration and manipulation priors. Our framework integrates a cognitive auxiliary head for autonomous sub-task transitions and a dual-track memory system to maintain consistent self and environmental awareness by fusing proprioceptive and visual temporal contexts. By aligning human and robot hand-eye coordination behaviors in a unified egocentric action space, we train the model progressively in three stages. Extensive experiments on a wheel-based humanoid have demonstrated strong robustness and adaptability of our proposed method across diverse long-horizon tasks spanning multiple active perception scenarios.

Problem

• active perception
• non-Markovian
• information uncertainty
• generalizable manipulation
• egocentric data

Innovation

• non-Markovian active perception
• vision-language-action (VLA)
• egocentric human data
• dual-track memory system
• cognitive auxiliary head