NEXUS: Perceptive Whole-Body Control for Terrain-Adaptive Teleoperation

2026-09-30

AI Summary

This study addresses the challenge of terrain-adaptive control in whole-body teleoperation for humanoid robots, which is hindered by the scarcity of paired data. To this end, we propose a terrain-aware whole-body control framework. Methodologically, we design a fine-tuning-free terrain-aware adaptation algorithm that efficiently synthesizes nearly one thousand hours of high-quality paired motion data. Furthermore, a teacher-student learning strategy is employed to train the controller, deeply integrating multimodal perceptual feedback with human commands and whole-body dynamics modeling. Experimental results demonstrate that the proposed approach significantly outperforms existing baselines on standard benchmarks and achieves zero-shot deployment in real-world scenarios, exhibiting superior cross-terrain generalization capability alongside high-fidelity motion tracking accuracy.

Abstract

Whole-body teleoperation requires a humanoid robot to reproduce a human operator's behavior even when their terrains differ. This demands that the robot perceive local terrain and adapt its posture and contacts accordingly, rather than copy the operator's motion frame by frame. However, paired motion data linking the same behaviors across flat ground and different terrains remain scarce, limiting supervision for learning terrain-adaptive control. To enable whole-body teleoperation across mismatched terrains, we introduce NEXUS, a perceptive whole-body control framework that combines human motion commands with onboard sensory feedback. We first develop a scalable terrain-aware adaptation algorithm that efficiently generates high-quality motion pairs across motions and terrains without per-motion or per-terrain tuning. Using a paired motion corpus totaling nearly 1,000 hours, we train a perceptive whole-body controller through teacher-student learning to reproduce commanded behaviors across terrains. Experiments demonstrate efficient, scalable generation of high-quality motion data and show that NEXUS combines broad behavioral coverage with terrain adaptability and tracking fidelity, outperforming existing whole-body controllers on the evaluated benchmarks. Zero-shot real-world deployment enables real-time whole-body teleoperation on diverse unseen terrains, further validating the generalization of our method. Project website: https://nexus-humanoid.github.io/

Problem

• whole-body teleoperation
• humanoid robot
• terrain adaptation
• paired motion data
• perceptive control

Innovation

• Whole-Body Teleoperation
• Terrain-Adaptive Control
• Teacher-Student Learning
• Perceptive Control
• Zero-Shot Deployment