PhotoHOI: Synthesizing 3D Hand-Object Interactions from a Single RGB Photograph

2026-08-03

AI Summary

This work proposes a novel method for generating physically plausible and semantically coherent 3D hand–object interaction sequences from a single RGB image and open-vocabulary language instructions. The approach leverages a vision–language model to parse the input and construct a structured task specification, jointly reconstructing the task-relevant 3D scene and planning collision-free object trajectories. It introduces an innovative joint learning mechanism that integrates task-conditioned contact priors with contact-conditioned grasp priors, and optimizes hand poses in a latent space to ensure physical plausibility and generalization. Experiments demonstrate that the method significantly reduces interpenetration and improves contact quality on the GRAB and H2O datasets, achieves higher task success rates and scene consistency on real-world images, and generalizes effectively to unseen objects and arbitrary language instructions.

Abstract

Hand-object interaction (HOI) is a fundamental human behavior with broad applications in AR/VR, digital humans, and embodied interaction. Existing methods typically require predefined object geometry, object trajectories, or task-specific conditions, limiting their use with natural real-world inputs. To address this, we study a more practical problem of synthesizing 3D hand-object interaction sequences from a single RGB photograph and an open-vocabulary language instruction, and introduce PhotoHOI. PhotoHOI first uses a vision-language model to parse the input image and instruction into a structured task specification, including the interaction object, target region, and spatial relation. It then recovers a compact task-relevant 3D scene and plans a smooth collision-aware object trajectory based on the recovered object states, support relations, and surrounding scene geometry. To synthesize hand motion that generalizes to real-world photographs and unseen objects, it learns transferable task-conditioned contact and contact-conditioned grasp priors from large-scale affordance and HOI data. The grasp is further refined in a learned latent space, constraining the optimization to a plausible hand-pose manifold. Experiments on GRAB and H2O demonstrate improved contact quality and reduced penetration over representative baselines. Results on real-world photographs further demonstrate higher task success and scene consistency, together with generalization to unseen objects and open-vocabulary instructions.

Problem

• hand-object interaction
• 3D synthesis
• single RGB image
• open-vocabulary instruction
• real-world generalization

Innovation

• hand-object interaction
• 3D synthesis
• vision-language model
• contact prior
• open-vocabulary instruction