SoPo: Text-to-Motion Generation Using Semi-Online Preference Optimization

2024-12-06

arXiv.org

AI Summary

Text-to-motion generation suffers from insufficient motion coherence, realism, and misalignment with human preferences. To address these challenges, we propose Semi-online Preference Optimization (SoPo), the first method to identify inherent limitations of offline and online Direct Preference Optimization (DPO)—namely, overfitting and biased sampling. SoPo innovatively introduces a “semi-online” pairwise data paradigm, synergistically combining online-generated low-quality motions with offline-curated high-quality ones to enable bidirectional compensation. Built upon the DPO framework, SoPo integrates online policy sampling, offline static preference modeling, and gradient-coordinated parameter updates. Evaluated on MLD and MDM, SoPo achieves +3.25% and +2.91% improvements in MM-Dist, respectively—outperforming MoDiPO significantly. After fine-tuning MLD, SoPo attains state-of-the-art performance in both R-precision and MM-Dist.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Mixture of Experts (MoE)
• Humans and AI: Learning Human Values and Preferences

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Text-to-motion generation is essential for advancing the creative industry but often presents challenges in producing consistent, realistic motions. To address this, we focus on fine-tuning text-to-motion models to consistently favor high-quality, human-preferred motions, a critical yet largely unexplored problem. In this work, we theoretically investigate the DPO under both online and offline settings, and reveal their respective limitation: overfitting in offline DPO, and biased sampling in online DPO. Building on our theoretical insights, we introduce Semi-online Preference Optimization (SoPo), a DPO-based method for training text-to-motion models using"semi-online"data pair, consisting of unpreferred motion from online distribution and preferred motion in offline datasets. This method leverages both online and offline DPO, allowing each to compensate for the other's limitations. Extensive experiments demonstrate that SoPo outperforms other preference alignment methods, with an MM-Dist of 3.25% (vs e.g. 0.76% of MoDiPO) on the MLD model, 2.91% (vs e.g. 0.66% of MoDiPO) on MDM model, respectively. Additionally, the MLD model fine-tuned by our SoPo surpasses the SoTA model in terms of R-precision and MM Dist. Visualization results also show the efficacy of our SoPo in preference alignment. Our project page is https://sopo-motion.github.io.

Problem

• Improving text-to-motion generation for realistic motions
• Addressing limitations of online and offline DPO methods
• Enhancing preference alignment in motion generation models

Innovation

• Semi-online data pair optimization
• Combines online and offline DPO
• Improves motion quality and preference