UniPhys: Unified Planner and Controller with Diffusion for Flexible Physics-Based Character Control

2025-04-17

AI Summary

Long-horizon, multimodal (text/trajectory/object)-guided physically realistic character motion generation remains challenged by the decoupling of planning and control, error accumulation in simulation, and poor generalization. This paper proposes the first end-to-end diffusion-driven framework that unifies motion generation and physics simulation within a single architecture. We introduce Diffusion Forcing—a novel training paradigm that explicitly models and corrects cumulative simulation errors. Furthermore, we design a multimodal conditional encoding scheme integrated with co-training alongside physics engines (MuJoCo and Isaac Gym). Our method achieves zero-shot generalization to unseen control signals without task-specific fine-tuning, delivering substantial improvements in physical plausibility, semantic consistency, motion naturalness, and robustness on hundred-frame motions—outperforming state-of-the-art two-stage approaches across all metrics.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Multimodal Learning
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Generating natural and physically plausible character motion remains challenging, particularly for long-horizon control with diverse guidance signals. While prior work combines high-level diffusion-based motion planners with low-level physics controllers, these systems suffer from domain gaps that degrade motion quality and require task-specific fine-tuning. To tackle this problem, we introduce UniPhys, a diffusion-based behavior cloning framework that unifies motion planning and control into a single model. UniPhys enables flexible, expressive character motion conditioned on multi-modal inputs such as text, trajectories, and goals. To address accumulated prediction errors over long sequences, UniPhys is trained with the Diffusion Forcing paradigm, learning to denoise noisy motion histories and handle discrepancies introduced by the physics simulator. This design allows UniPhys to robustly generate physically plausible, long-horizon motions. Through guided sampling, UniPhys generalizes to a wide range of control signals, including unseen ones, without requiring task-specific fine-tuning. Experiments show that UniPhys outperforms prior methods in motion naturalness, generalization, and robustness across diverse control tasks.

Problem

• Generating natural, physically plausible character motion
• Bridging domain gaps in motion planning and control
• Handling long-horizon control with diverse guidance signals

Innovation

• Unified diffusion-based motion planning and control
• Multi-modal input conditioning for flexible motion
• Diffusion Forcing to reduce prediction errors