PartInteractor: Intent-Driven Part-Aware 3D Authoring for Continuous Co-Creation in XR

2026-08-02

AI Summary

This work addresses the limitations of existing generative XR systems, which often lack semantic structure and thus struggle to support fine-grained editing and control. To overcome this, the authors propose a semantics-aware, part-based generative framework for XR content creation that interprets user intent from speech, sketches, or multimodal inputs. By integrating a large language model interpreter, a retrieval-augmented generation strategy, and hierarchical part-based 3D generation, the system produces 3D assets with explicit part structures. The approach introduces part-aware representations and an intent scaffolding mechanism, transforming one-shot generation into an interactive, editable, and iterative co-creation process. This significantly enhances controllability and expressiveness of the outputs, reduces the need for regeneration, and improves alignment between user intent and generated results, thereby validating the efficacy of part-aware modeling and intent scaffolding in XR authoring.

Abstract

As Extended Reality (XR) evolves into an immersive computing medium, interactive 3D authoring becomes essential for creative and functional workflows. However, existing generative XR systems produce monolithic outputs lacking explicit semantic structure, limiting post-generation control. We introduce PartInteractor, a representation-to-interaction framework that investigates how semantic part hierarchies can be incorporated into generative XR authoring, and exposed as first-class, directly manipulable units, turning one-shot prompt-to-object generation into continuous component-level co-creation. PartInteractor supports speech, sketch, and image inputs, integrating an LLM interpreter with a retrieval-generation strategy to scaffold user intent prior to 3D generation. Instead of producing monolithic objects, our system generates semantically decomposed 3D assets with explicit part hierarchies, enabling rich component-level interaction over object structure and composition. Our evaluations suggest that part-aware representation increases post-generation control and reduces reliance on whole-object regeneration, while intent scaffolding mitigates ambiguity and improves intent-result alignment, together supporting more expressive and controllable human-AI co-creation workflows. These results highlight part-aware representation and intent scaffolding as promising design considerations for future generative XR authoring systems.

Problem

• generative XR
• semantic structure
• post-generation control
• 3D authoring
• part-aware representation

Innovation

• part-aware representation
• intent scaffolding
• generative XR authoring
• component-level co-creation
• semantic part hierarchies