GenHOI: Contact-Aware Humanoid-Object Interaction by Imitating Generated Videos without Task-Specific Training

2026-06-11

AI Summary

This work addresses the challenge of enabling humanoid robots to perform diverse object interaction tasks while maintaining dynamic balance and stable contact, a capability often hindered by the reliance of existing methods on task-specific training or rigid trajectories that limit generalization. The authors propose a zero-shot approach that requires only a textual instruction and a single generated video to drive complex robotic interactions, eliminating the need for real-world demonstrations or task-specific fine-tuning. Central to their method is a contact-aware geometric constraint that translates visual cues from 2D videos into physically plausible priors for optimization, effectively resolving scale ambiguity and unknown object poses. Integrating video generation, scene reconstruction, contact estimation, geometric optimization, and closed-loop control, the system successfully executes tasks such as box lifting, two-arm chair carrying, under-table lifting, and cylindrical wrapping in both simulation and real-world environments, demonstrating strong generalization and practical utility.

Abstract

Humanoid-Object Interaction (HOI) is a fundamental capability for humanoid robots, yet it remains challenging due to the tight coupling between dynamic balance and stable interaction with diverse objects. Existing methods often require time-consuming task-specific policy training or rely on rigid trajectory replay, which limits their ability to accommodate novel interaction scenarios. In this work, we present \textit{GenHOI}, a simple yet effective framework that enables humanoid robots to perform diverse object-interaction tasks in a zero-shot manner by directly imitating a single generated video, without task-specific training or physical demonstration data. GenHOI first reconstructs the robot-object scene in simulation and renders a first-frame image, which, together with the language command, conditions the synthesis of a task-oriented interaction video. The generated video is then analyzed to identify interaction-relevant contact events and estimate hand-object contact regions, which are encoded as object-centric geometric constraints that convert visual interaction cues into physically grounded optimization priors. Guided by these priors, the reference motion recovered from the video is refined and smoothed to resolve the scale ambiguity inherent in 2D video generation, while adapting a single reference trajectory to unseen robot-object relative poses. The optimized trajectory is finally executed by a closed-loop tracking controller. We validate the proposed framework in extensive simulation and real-world experiments across diverse object-interaction tasks, including box grasping, asymmetric bimanual chair carrying, table lifting from below, and cylindrical-object enveloping.

Problem

• Humanoid-Object Interaction
• Dynamic Balance
• Stable Interaction
• Task-Specific Training
• Novel Interaction Scenarios

Innovation

• zero-shot imitation
• contact-aware interaction
• video-to-motion generation
• geometric constraints
• humanoid-object interaction