AgentHOI: Multi-Agent Reasoning for Human-Object-Interaction Video Generation via Implicit Representation Alignment

2026-07-24

AI Summary

Existing approaches rely on explicit motion control, limiting their generalization in scenarios involving diverse objects and complex interactions. This work proposes a “reason-then-generate” framework that incorporates a multi-agent collaboration mechanism to jointly reason about human-object perception, interaction intent, and motion planning. It further introduces an implicit text-motion alignment strategy to guide a video diffusion model in generating high-quality interactive videos. Notably, the method operates without explicit motion inputs, significantly enhancing its ability to understand and faithfully follow complex textual instructions. As a result, it achieves more natural interaction dynamics, higher object appearance fidelity, and improved controllability in tasks such as wearing garments or riding objects.

Abstract

Recent advances in video diffusion models have spurred interest in human-object interaction (HOI) video generation, which demands fine-grained control over interaction logic beyond single-subject animation. However, existing HOI methods rely heavily on explicit motion control, limiting scalability and generalization across diverse objects and interactions. In this study, we propose AgentHOI, a text-driven HOI video generation following a thinking-before-generation framework that bridges the gap between high-level textual intent and physical execution through multi-agent reasoning over perception, interaction, and motion planning. Building upon the generated interaction plans, we further strengthen text-driven motion understanding. We introduce an implicit text-motion alignment strategy that distills text-to-motion priors into the video diffusion model, enabling robust HOI synthesis without explicit motion inputs at inference. Experiments show that AgentHOI significantly improves interaction naturalness, object appearance preservation, and adherence to complex textual instructions across challenging object-centric scenarios such as wearing and riding. The code is available at https://github.com/bone-11/agenthoi.

Problem

• Human-Object Interaction
• Video Generation
• Motion Control
• Generalization
• Scalability

Innovation

• multi-agent reasoning
• implicit representation alignment
• text-driven video generation
• human-object interaction
• video diffusion model