MoRAE: Flow-Friendly Self-Supervised Latents for Text-to-Motion Generation

2026-07-31

AI Summary

This work addresses the instability and motion distortion commonly observed in text-to-motion generation when directly employing latent representations from self-supervised encoders such as Motion-JEPA, which often fail to simultaneously preserve semantic accuracy, temporal coherence, and physical plausibility. To overcome these limitations, the authors propose MoRAE, a novel framework that systematically identifies and mitigates two key bottlenecks: ill-conditioned spectral properties in the latent space and misalignment between latent geometry and decoder-sensitive directions. MoRAE introduces compact bottleneck distillation to refine the spectral structure of JEPA features and incorporates motion-coupled training to align the latent manifold with decoder sensitivities. Integrated with Flow-Matching and a DiT architecture, the method achieves state-of-the-art performance under standard non-autoregressive settings, significantly enhancing both the quality and stability of generated motions.

Abstract

Text-to-motion generation must produce motions that are semantically correct, temporally coherent, and physically plausible. A natural approach is to first project motion data into a structured semantic space and then train a generative model within that space. Such a paradigm has been highly successful in image generation through Representation Autoencoders (RAEs), where a frozen self-supervised encoder provides semantic features for diffusion or flow models to learn from. However, direct transfer of such a paradigm to motion space using Motion-JEPA as the frozen encoder fails dramatically. We diagnose this failure geometrically and identify two motion-specific bottlenecks: (1) the JEPA feature space is spectrally ill-conditioned, making the Gaussian-to-data transport unstable; and (2) even with a well-conditioned spectrum, flow residuals tend to align with decoder-sensitive directions, where small latent errors are amplified into large motion artifacts after decoding. Based on these insights, we propose MoRAE. MoRAE addresses the two bottlenecks separately. A compact bottleneck distills the structured JEPA representation while removing weak and redundant directions, bringing the latent spectrum into a transport-stable regime. Motion-coupled training then aligns the retained latent geometry with the decoder, making characteristic flow errors less costly after decoding. With this flow-friendly latent, a standard non-autoregressive Flow-Matching DiT achieves state-of-the-art performance.

Problem

• text-to-motion generation
• self-supervised representation
• flow matching
• latent space
• motion generation

Innovation

• MoRAE
• flow-friendly latent
• text-to-motion generation
• self-supervised representation
• motion generation