MatchDance: Collaborative Mamba-Transformer Architecture Matching for High-Quality 3D Dance Synthesis

2025-05-20

AI Summary

Music-driven 3D dance generation suffers from insufficient choreographic consistency. To address this, we propose a two-stage collaborative framework: (1) a kinematic-dynamic-constrained Finite Scalar Quantization (FSQ) scheme that constructs high-fidelity, disentangled dance latent representations; and (2) a Mamba-Transformer hybrid architecture enabling precise cross-modal mapping from music to the learned latent space. We introduce the first kinematic-dynamic quantization paradigm synergized with Mamba-Transformer modeling, and establish the inaugural music–dance cross-modal retrieval evaluation framework tailored for dance generation, including dedicated metrics. Evaluated on the FineDance dataset, our method achieves state-of-the-art performance, significantly improving motion coherence, beat alignment, and 3D motion naturalness. The source code is publicly available.

Technology Category

Machine Learning: Deep Generative Models & AutoencodersNatural Language Processing: Code Generation / Program Synthesis from Natural LanguageComputer Vision: Motion & Tracking

Application Category

Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAGUser Modeling, Personalization and Recommendation: Fairness-aware retrieval and rankingGraph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Music-to-dance generation represents a challenging yet pivotal task at the intersection of choreography, virtual reality, and creative content generation. Despite its significance, existing methods face substantial limitation in achieving choreographic consistency. To address the challenge, we propose MatchDance, a novel framework for music-to-dance generation that constructs a latent representation to enhance choreographic consistency. MatchDance employs a two-stage design: (1) a Kinematic-Dynamic-based Quantization Stage (KDQS), which encodes dance motions into a latent representation by Finite Scalar Quantization (FSQ) with kinematic-dynamic constraints and reconstructs them with high fidelity, and (2) a Hybrid Music-to-Dance Generation Stage(HMDGS), which uses a Mamba-Transformer hybrid architecture to map music into the latent representation, followed by the KDQS decoder to generate 3D dance motions. Additionally, a music-dance retrieval framework and comprehensive metrics are introduced for evaluation. Extensive experiments on the FineDance dataset demonstrate state-of-the-art performance. Code will be released upon acceptance.

Problem

• Enhancing choreographic consistency in music-to-dance generation
• Mapping music to latent dance representations accurately
• Overcoming limitations in 3D dance motion synthesis

Innovation

• Kinematic-Dynamic Quantization for latent representation
• Mamba-Transformer hybrid music-to-dance mapping
• Music-dance retrieval framework for evaluation