Learning Terrain-Aware Whole-Body Control for Perceptive Legged Loco-Manipulation

2026-05-29

AI Summary

This work addresses the limitations of existing whole-body controllers, which predominantly rely on proprioception and struggle to adapt posture and foothold placement in complex terrains. The authors propose the Terrain-Aware Whole-Body Control (TA-WBC) framework, an end-to-end reinforcement learning approach that integrates legged locomotion with arm manipulation through terrain perception. The method introduces a hybrid exteroceptive encoder, a foot-contact-plane-based end-effector sampling mechanism, and a dual-policy distillation module to jointly enhance terrain adaptability and large-scale motion capabilities. Simulations and real-world experiments demonstrate that TA-WBC significantly expands the reachable manipulation workspace, reduces trajectory tracking error, minimizes unintended falls, and thereby improves the robustness of loco-manipulation across diverse terrains.

Abstract

Legged manipulators integrate exceptional terrain adaptability along with mobile manipulation capabilities, which make them highly promising for deployment in human-centric environments. By coordinating the control of both legs and arms, a whole-body controller can significantly expand the operational workspace of legged manipulators. However, many existing whole-body controllers primarily depend on proprioception and do not incorporate the critical exteroception required for effective terrain topology perception. This limitation can hinder their ability to adapt to varying environmental conditions and navigate complex terrains effectively. In this paper, we introduce TA-WBC, a terrain-aware whole-body control framework for legged manipulators, which features a novel RL-based unified policy tailored to whole-body loco-manipulation tasks in various terrains. Specifically, we employ a hybrid exteroception encoder to extract terrain features, providing an essential basis for the robot to proactively adapt posture and footholds. Furthermore, to facilitate stable cross-terrain loco-manipulation, we propose a novel end-effector sampling method based on the foot contact plane, decoupling manipulation target from base fluctuations. Moreover, a dual-policy distillation module is introduced to integrate expansive whole-body motion with terrain adaptability without catastrophic forgetting. The simulation and real-world experiments validate the robustness of our proposed controller, which leads to a larger reachable space, less tracking error, and reduced unexpected stumbles. This unified policy highlights the promising capabilities of legged manipulators in performing loco-manipulation tasks across complex terrains.

Problem

• legged manipulators
• whole-body control
• terrain perception
• loco-manipulation
• exteroception

Innovation

• terrain-aware control
• whole-body locomotion-manipulation
• exteroception encoder
• end-effector sampling
• policy distillation