joint human-scene reconstruction

This work addresses the challenges of scale ambiguity, misalignment between humans and scenes, and occlusion interference in human reconstruction from dynamic scenes captured by a moving monocular camera. To tackle these issues, the authors propose SHOW—a framework that jointly infers human meshes and scene geometry in a unified metric space through a feed-forward pipeline. The method integrates the semantic structure and scale priors of parametric human models and employs a promptable mask mechanism to flexibly specify target individuals. Mutual guidance between human and scene representations enhances spatial alignment and metric scale consistency. Experiments demonstrate that SHOW significantly improves metric-scale reconstruction accuracy, human-scene alignment quality, and overall robustness in complex scenarios involving multiple people, severe occlusions, and cluttered backgrounds.

This work addresses the joint reconstruction of multi-person 3D meshes, scene point clouds, and camera poses from sparse, uncalibrated multi-view images, all within a unified metric world coordinate system that explicitly models spatial relationships among humans, environment, and cameras. Methodologically, it is the first to embed the SMPL human statistical model into a Structure-from-Motion (SfM) framework, leveraging human priors to impose absolute scale constraints. A multi-module joint optimization scheme is introduced to co-estimate human meshes, scene geometry, and camera parameters, synergistically integrating data-driven reconstruction with classical SfM principles. Evaluated on EgoHumans and EgoExo4D, the method reduces world-coordinate human localization error to 1.04 m and 0.56 m, respectively, and improves camera pose accuracy (RRA@15) by 20.3%, significantly enhancing overall geometric fidelity and cross-modal consistency.

This work addresses the domain shift, geometric distortions, and misalignment commonly encountered when jointly reconstructing real-world scenes and 3D human structures due to reliance on synthetic data. The authors propose an end-to-end feedforward framework that simultaneously recovers metric-scale scene geometry, human point clouds, camera parameters, and SMPL body models in a single forward pass. Leveraging unlabeled real-world videos, the method innovatively integrates scene reconstruction with HMR priors through a two-stage training strategy—coarse localization using synthetic data followed by geometric refinement on real data—and employs a high-frequency detail distillation scheme to effectively bridge the sim-to-real domain gap. Experiments demonstrate that the approach achieves state-of-the-art performance in human-centric scene reconstruction and outperforms both optimization-based and pure HMR methods in global human motion estimation.

This work addresses the lack of physical plausibility in single-image 3D reconstruction. We propose the first physics-compatible reconstruction framework that enforces static equilibrium as a hard constraint. Methodologically, we explicitly decouple and jointly optimize material stiffness, external loading forces, and the static equilibrium geometry; deformation responses are modeled via differentiable physics simulation, enabling gradient-based joint optimization of all variables. Our approach breaks from conventional simplifications—such as rigid-body assumptions or neglect of external forces—by embedding real-world physical constraints directly into the single-image reconstruction pipeline. Evaluated on Objaverse, our method yields reconstructions with significantly improved mechanical stability, suitable for downstream dynamic simulation and 3D printing. Physical validation via real-world force testing further confirms the structural robustness of the generated models.

Existing implicit-representation-based methods for 3D human reconstruction from a single RGB image suffer from limited generalization, robustness, and geometric controllability. This paper proposes the first fully explicit, point-cloud-driven end-to-end framework—eliminating implicit functions entirely and directly regressing, generating, and optimizing human point clouds in 3D space. Key contributions include: (1) a geometric center regression paradigm to enhance structural controllability; (2) an SMPL-guided explicit point cloud estimation and refinement network; and (3) a prior-constrained point cloud generation and registration mechanism. Our method achieves significant improvements over state-of-the-art approaches across multiple benchmarks, with 20–40% reductions in Chamfer Distance (CD) and Mean Per-Joint Position Error (MPJPE), yielding more detailed and complete reconstructions. Code and data are publicly released.

Existing multi-person 3D reconstruction methods often suffer from geometric incompleteness and implausible interpenetrations when handling occlusions and close interactions. This work proposes HUG3D, a novel framework that explicitly incorporates group-level contextual cues and physical interaction priors to jointly model individual and collective information in an orthonormal canonical space, enabling collaborative optimization of occlusion, contact, and spatial relationships. HUG3D comprises two core modules: Human Group-Instance Multi-View Diffusion (HUG-MVD) and Geometric Reconstruction (HUG-GR), which integrate multi-view normal generation with physics-aware geometric refinement to achieve high-fidelity texture fusion. Requiring only a single input image, HUG3D significantly outperforms current state-of-the-art single- and multi-person reconstruction approaches in both reconstruction fidelity and physical plausibility.

3D human mesh reconstruction from in-the-wild images suffers from inaccurate orientation estimation in the world coordinate system, primarily due to the absence of ground-truth camera rotation—especially pitch angle—leading to substantial errors under the common zero-rotation assumption. To address this, we propose a human-centered strategy that estimates camera pitch solely from RGB images and synthetic depth maps. We further introduce a plug-and-play Mesh-Plug module that jointly optimizes root joint orientation and full-body pose. Additionally, we design a camera rotation prediction network grounded in human spatial configuration. Our method achieves significant improvements over state-of-the-art approaches on the SPEC-SYN and SPEC-MTP benchmarks, enabling more accurate and robust world-coordinate human mesh reconstruction without requiring real camera calibration.

为解决几何感知与机器人动作对齐问题，提出Metric-Aware Geometry Perception框架，通过相机参数和深度观测重建度量几何，提高跨场景一致性。

Existing single-image human-scene interaction reconstruction methods struggle to balance speed and physical plausibility: optimization-based approaches are accurate but slow, while feedforward methods are fast yet lack explicit interaction modeling, often producing floating or interpenetration artifacts. This work proposes GRAFT, a learnable interaction prior that generates volume-anchored tokens via geometric probes and employs a lightweight recurrent Transformer to predict interaction gradients, thereby transforming geometric fitting into efficient feedforward inference. GRAFT achieves explicit and physically reasonable 3D interaction reconstruction at high speed. Experiments show that GRAFT improves interaction quality by up to 113% over state-of-the-art feedforward methods, operates approximately 50 times faster than optimization-based approaches, generalizes well to in-the-wild multi-person scenes, and achieves a user preference rate of 64.8%.

Existing methods for recovering 3D human geometry from monocular video face significant challenges: Vision Transformers (ViTs) tend to overfit to 2D viewpoints, while NeRF- and Gaussian Splatting–based avatars decouple pose and appearance, limiting generalization to novel poses. This work proposes HumanSplatHMR, the first approach to tightly couple human mesh recovery with Gaussian Splatting avatar modeling within a joint optimization framework. By leveraging differentiable rendering, photometric, segmentation, and depth losses are end-to-end backpropagated to pose parameters, enabling closed-loop optimization of both pose and appearance. Requiring neither motion capture nor offline refinement, the method substantially improves 3D pose accuracy in real-world scenes and enhances rendering fidelity under novel poses and viewpoints, outperforming existing decoupled baselines.