CustomDance: Customized 3D Dance Generation with Coarse-to-Fine Human-Centered Interactive Control

2026-08-06

AI Summary

Existing 3D dance generation methods struggle to achieve fine-grained control over multimodal inputs such as music and text, resulting in limited expressiveness and misalignment with creative intent. This work proposes a coarse-to-fine interactive generation system inspired by professional choreographic workflows: it first leverages a multimodal large language model to interpret user prompts and retrieve high-quality motion clips, then employs a music-conditioned diffusion refiner to seamlessly connect and iteratively optimize the motion sequence. Introducing a human-centered interactive AI choreography paradigm, the approach incrementally integrates user intent throughout the generation pipeline, jointly enhancing controllability, expressiveness, and output quality. Experimental results demonstrate that the proposed system significantly outperforms existing methods in both quantitative and qualitative evaluations, effectively empowering users’ creative expression and practical choreographic utility.

Abstract

With the rise of AI-generated content (AIGC) and advanced techniques for 3D human representation, the task of generating 3D dance movements has become an exciting area of research. Despite significant advancements, current methods often fail to provide comprehensive and distinct control over various multimodal inputs from users, such as music or specific descriptions of desired movements. As a result, the generated motions may be statistically plausible and technically correct, but they often lack depth, expressiveness, and alignment with the user's creative vision. To address this issue, we present CustomDance, a coarse-to-fine interactive system designed for customized 3D dance generation. Inspired by the workflows of expert choreographers, CustomDance introduces a novel paradigm to AI-assisted choreography through three interconnected stages. First, a multimodal Large Language Model (MLLM) analyzes the music and a high-level text prompt to identify key temporal anchors and creative cues for the piece. Next, for each anchor, a multimodal retriever suggests high-quality motion clips from a dance library based on local music and text, empowering the user with concrete and predictable options. Finally, a custom music-conditioned diffusion in-painter seamlessly connects the selected phrases, allowing for iterative, user-guided refinement of the final composition, supported by visualizations of motion dynamics. Our evaluations demonstrate that CustomDance not only highlights the significant creative utility and empowering potential of our AI-assisted choreography paradigm, but also outperforms competitive baselines across quantitative and qualitative comparisons.

Problem

• 3D dance generation
• multimodal control
• user customization
• expressive motion
• AI-assisted choreography

Innovation

• coarse-to-fine control
• multimodal retrieval
• diffusion inpainting
• AI-assisted choreography
• interactive 3D dance generation