PhysAgent: Automating Physics-Based 4D Synthesis via Trajectory-Grounded Multi-Agent Feedback

2026-06-07

AI Summary

Existing methods for automatically generating physically plausible 4D dynamic scenes often rely on manual intervention due to the difficulty of optimizing complex external force fields, leading to solutions that are prone to local optima and physically inaccurate. This work proposes PhysAgent—the first closed-loop, multi-agent framework for 4D content generation: a semantic agent initializes force fields, while a trajectory-driven optimization agent leverages vision foundation models to extract motion trajectories and employs large language models for zero-shot reasoning to achieve physically coherent synthesis. By uniquely integrating multi-agent feedback with trajectory guidance, the approach enables dynamic switching and macroscopic transitions in discrete force fields, effectively bridging the modality gap and escaping local optima. The method significantly outperforms existing baselines in both generation diversity and physical plausibility.

Abstract

Achieving fully automated, physically plausible 3D motion synthesis is a core objective in graphics and generative AI. However, configuring complex environmental force fields still relies entirely on manual expert intervention, creating a severe bottleneck for large-scale simulation data generation. Existing automated methods primarily focus on material optimization and exhibit severe modality gaps and technical flaws when applied to the vastly more complex force field optimization space: naive Large Language Models (LLMs) lack underlying simulation feedback, causing severe physical inaccuracies, while traditional Score Distillation Sampling (SDS) suffers from sluggish gradients, local optima entrapment, and a mathematical inability to dynamically switch discrete force fields. To address this, we propose PhysAgent, the first simulator-in-the-loop multi-agent framework that leverages multimodal inputs for automated, physically grounded 4D synthesis. By decoupling intrinsic materials from extrinsic dynamics, PhysAgent utilizes a Semantic Agent equipped with an externalized Force Field Skill module to master simulation rules and generate valid initializations. Subsequently, the Refine Agents, driven by Trajectory-Grounded Multi-Agent Feedback, leverage vision foundation models to extract dense point trajectories from rendered frames. By converting these explicit motion trajectories into structured textual descriptors, the agent harnesses LLM commonsense reasoning to execute zero-shot macroscopic leaps, effectively escaping local optima and dynamically switching discrete force fields. Extensive experiments demonstrate that PhysAgent rapidly generates stable, diverse physical scenes from arbitrary multimodal prompts, significantly outperforming existing baselines in both generation diversity and physical accuracy.

Problem

• physics-based synthesis
• force field optimization
• 4D generation
• physical plausibility
• automated simulation

Innovation

• Physics-Based 4D Synthesis
• Trajectory-Grounded Feedback
• Multi-Agent Framework
• Force Field Optimization
• Simulator-in-the-Loop