FineMoLA: Towards Fine-Grained Motion-Language Alignment from Clip-Level Supervision

2026-08-02

AI Summary

This work addresses the challenge of achieving fine-grained temporal alignment between motion frames and linguistic phrases in text-driven human motion generation, a task hindered by the common reliance on clip-level supervision. To overcome this limitation without requiring explicit frame-level annotations, the authors propose FineMoLA, a weakly supervised framework that establishes many-to-many alignments between frames and action phrases. The approach first segments input text into action phrases and then leverages optimal transport theory with entropy regularization to efficiently infer pseudo frame-level alignments via Sinkhorn iterations. FineMoLA is the first method to achieve fine-grained motion-language alignment under weak supervision, significantly outperforming existing baselines on the SnapMoGen dataset and demonstrating superior performance in motion-text grounding tasks.

Abstract

Text-conditioned human motion generation has made rapid progress with the emergence of large-scale motion--language datasets. However, even datasets with rich long-form descriptions typically provide supervision only at the clip level, without explicit temporal correspondence between motion frames and language. This limits fine-grained motion--text grounding and temporally precise generation. We propose FineMoLA, a weakly supervised framework that learns fine-grained frame--phrase correspondence directly from clip-level annotations. Our method first segments long-form descriptions into action-bearing phrases, and then formulates motion--language alignment as an optimal transport problem, which naturally models many-to-many relations between motion frames and text under global constraints. With entropic regularization and Sinkhorn iterations, FineMoLA efficiently infers pseudo frame-level alignments without human labeling. Experiments on SnapMoGen demonstrate that the learned alignments outperform baselines in motion--text grounding.

Problem

• fine-grained alignment
• motion-language grounding
• clip-level supervision
• temporal correspondence
• text-conditioned motion generation

Innovation

• fine-grained alignment
• optimal transport
• weakly supervised learning
• motion-language grounding
• text-conditioned motion generation