Moving Out: Physically-grounded Human-AI Collaboration

2025-07-24

AI Summary

This work addresses the challenge of enabling embodied agents to adapt to human collaboration in continuous state-action spaces under complex physical constraints. We propose the Behavior-Augmented Simulation and Selection (BASS) framework, which—through behavior augmentation, physics-based simulation, and diversity-aware selection—first systematically models how physical attributes (e.g., mass, friction) and dynamic interactions influence collaborative behavior. BASS integrates physics-enabled simulation environment construction, real-world human interaction data collection, behavioral cloning, and reinforcement learning to achieve generalized modeling of diverse human collaboration strategies. Experiments demonstrate that BASS significantly outperforms state-of-the-art methods on both AI-AI and human-AI collaborative tasks—including joint object transport and corner navigation—exhibiting exceptional robustness and adaptability to unseen physical parameters and previously unobserved human behavioral patterns.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Multiagent Systems: Agent-Based Simulation and Emergent Behavior

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Research challenges in human and human-AI computation

Abstract

The ability to adapt to physical actions and constraints in an environment is crucial for embodied agents (e.g., robots) to effectively collaborate with humans. Such physically grounded human-AI collaboration must account for the increased complexity of the continuous state-action space and constrained dynamics caused by physical constraints. In this paper, we introduce extit{Moving Out}, a new human-AI collaboration benchmark that resembles a wide range of collaboration modes affected by physical attributes and constraints, such as moving heavy items together and maintaining consistent actions to move a big item around a corner. Using Moving Out, we designed two tasks and collected human-human interaction data to evaluate models' abilities to adapt to diverse human behaviors and unseen physical attributes. To address the challenges in physical environments, we propose a novel method, BASS (Behavior Augmentation, Simulation, and Selection), to enhance the diversity of agents and their understanding of the outcome of actions. Our experiments show that BASS outperforms state-of-the-art models in AI-AI and human-AI collaboration. The project page is available at href{https://live-robotics-uva.github.io/movingout_ai/}{https://live-robotics-uva.github.io/movingout_ai/}.

Problem

• Adapting to physical actions and constraints in human-AI collaboration
• Addressing continuous state-action space complexity in physical environments
• Enhancing agent diversity and action outcome understanding

Innovation

• BASS method enhances agent diversity and action understanding
• Moving Out benchmark tests diverse physical collaboration modes
• Behavior Augmentation improves human-AI collaboration performance