SM4RT: Learning Structured Motion Geometry for 4D Reconstruction

2026-07-24

AI Summary

This work addresses the limitation of existing methods that model motion as independent point displacements, thereby neglecting the rigid-body structure and geometric consistency inherent in real-world object motion. To overcome this, we propose SM4RT—an end-to-end 4D reconstruction Transformer that introduces structured motion modeling into monocular dynamic scene reconstruction for the first time. Our approach explicitly represents rigid-body motion trajectories through a sparse set of shared motion bases and encodes dynamics using 6D twist sequences in SE(3). A parallel motion-geometry encoder-decoder architecture enables joint inference of geometry, motion, and kinematic structure. Experiments demonstrate that SM4RT achieves high-fidelity 3D reconstruction while significantly outperforming current point-wise motion estimation methods, yielding marked improvements in structured motion reconstruction quality.

Abstract

Geometry Foundation Models (GFMs) have substantially advanced monocular 3D reconstruction, yet extending this capability to 4D dynamic understanding remains a fundamental challenge. Most existing motion perception methods (e.g., sparse tracking, dense point-wise flow) treat motion as independent point-wise displacements, ignoring the structured nature of physical motion. However, real-world objects usually obey rigid-body kinematics, and points thus usually move collectively, not in isolation. Motion itself possesses geometric structure: physical objects undergo a set of rigid-body transformations governed by SE(3), rather than unstructured point-wise displacements. Building on this insight, we propose SM4RT, a Structured Motion 4D Reconstruction Transformer for end-to-end 3D reconstruction and structured motion perception. SM4RT introduces Structure-of-Motion to represent scene dynamics, where scene motion is decomposed into a compact set of motion bases, each represented as a temporal sequence of 6D twists in SE(3). Dense scene motion is then recovered by sparse, time-shared per-pixel assignment weights over these bases, ensuring points on the same object share a common rigid-body motion trajectory. SM4RT introduces a parallel motion geometry encoder and decoder that jointly infer 3D geometry, world-coordinate motion, and scene kinematic structure in a single forward pass from monocular RGB video. SM4RT achieves strong motion reconstruction performance while preserving the geometric structure of scene motion.

Problem

• 4D reconstruction
• structured motion
• rigid-body motion
• motion geometry
• monocular video

Innovation

• Structured Motion
• 4D Reconstruction
• SE(3)
• Motion Bases
• Geometry Foundation Models