AxisGuide: Grounding Robot Action Coordinate System in RGB Observations for Robust Visuomotor Manipulation

2026-06-04

AI Summary

Existing visuomotor policies exhibit limited generalization under object position distribution shifts, primarily due to the absence of an explicit understanding of how actions expressed in the robot’s base coordinate frame manifest in image space. To address this, this work proposes AxisGuide, a method that leverages camera intrinsics and end-effector pose to render the base-frame coordinate axes directly onto RGB images as lightweight visual guidance channels. This approach seamlessly integrates semantic scene understanding with action coordinate interpretation without requiring any architectural modifications to the policy network. Evaluated in both simulated and real-world settings using the LIBERO benchmark, AxisGuide significantly enhances policy robustness and generalization to unseen object positions. Notably, it achieves the first explicit visualization of action coordinate frame orientations directly within image space.

Abstract

Visuomotor manipulation policies trained via large-scale behavior cloning have achieved strong semantic scene understanding, yet often fail to reliably execute correct low-level actions under distribution shifts. For example, even in a simple pickup task with identical scene layouts, camera viewpoints, and illumination, performance can degrade substantially when the object is placed at unseen locations. We argue that this gap arises from insufficient action understanding, namely the inability to interpret the robot's base-frame action coordinate system in image space. To address this issue, we introduce AxisGuide, a lightweight guidance method that bridges semantic scene understanding and action-coordinate interpretation. Using camera parameters and end-effector poses, AxisGuide renders the robot base-frame axes in each camera view and augments RGB observations with a small set of cue channels that explicitly visualize the meaning of the +x, +y, and +z motions in image space. Extensive evaluations in both the LIBERO simulation and real-world environments demonstrate that AxisGuide yields substantial performance gains and improved generalization, highlighting the effectiveness of explicit action-coordinate cues for learning reliable and transferable generalist visuomotor policies.

Problem

• visuomotor manipulation
• action coordinate system
• distribution shift
• RGB observations
• robotic generalization

Innovation

• AxisGuide
• visuomotor manipulation
• action coordinate grounding
• RGB observation augmentation
• generalization