A Principled Approach for Creating High-fidelity Synthetic Demonstrations for Imitation Learning

2026-05-02

AI Summary

Existing 3D Gaussian Splatting (3DGS)-based synthesis methods often deviate from expert trajectories when generating novel motions, compromising the spatiotemporal structure essential for contact-intensive and shape-sensitive tasks, thereby degrading downstream policy learning. This work proposes a synthesis framework that explicitly preserves expert motion structure by integrating Dynamic Movement Primitives (DMPs) with the continuous density field of 3DGS, unifying photorealistic rendering and geometric reasoning without requiring additional geometric representations. Leveraging expert trajectories as strong priors, the method enables high-fidelity demonstration generation for novel goals, object configurations, and viewpoints, while exploiting the density field for obstacle-aware collision avoidance and safe trajectory redirection. Evaluated on three manipulation tasks with the Spot robot, the approach significantly reduces trajectory deviation and collision rates compared to planning- and optimization-based baselines, leading to improved task success for visuomotor policies.

Abstract

Recent advances in 3D Gaussian Splatting (3DGS) have enabled visually realistic demonstration generation from a single expert trajectory and a short multi-view scan. However, existing 3DGS-based synthesis pipelines typically generate new motions using sampling-based planners or trajectory optimization, which often deviate substantially from the expert's demonstrated path. While such deviations may be acceptable for tasks insensitive to motion shape, they discard subtle spatial and temporal structure that is critical for contact-rich and shape-sensitive manipulation, causing increased demonstration diversity to harm downstream policy learning. We argue that demonstration synthesis should treat the expert trajectory as a strong prior. Building on this principle, we propose a framework that synthesizes diverse task demonstrations while explicitly preserving expert motion structure. We model the expert trajectory using Dynamic Movement Primitives (DMPs) and retarget it to new goals, object configurations, and viewpoints within a reconstructed 3DGS scene, yielding phase-consistent, shape-preserving motion by construction. To safely realize this expert-preserving diversity in cluttered scenes, we introduce an analytic obstacle-aware DMP formulation that operates directly on the continuous density field induced by the 3DGS representation. This enables collision avoidance while minimally perturbing the nominal expert motion, unifying photorealistic rendering and geometric reasoning without additional scene representations. We evaluate our approach on a Spot mobile manipulator across three manipulation tasks with increasing sensitivity to trajectory fidelity. Compared to planner- and optimization-based synthesis, our method produces trajectories with lower deviation and collision rates and yields higher task success when training diffusion-based visuomotor policies.

Problem

• Imitation Learning
• 3D Gaussian Splatting
• Trajectory Fidelity
• Demonstration Synthesis
• Motion Structure Preservation

Innovation

• Dynamic Movement Primitives
• 3D Gaussian Splatting
• Imitation Learning
• Trajectory Retargeting
• Obstacle-aware Motion Generation