MIME: Multimodal Interactive Motion Encoder

2026-07-18

AI Summary

This work addresses the challenge of aligning the intricate dynamics and interdependencies between language and dyadic interaction motions. To this end, it proposes the first multimodal encoder specifically designed for dyadic interactions, introducing explicit interaction features through a shared structural representation and a streaming co-attention mechanism to capture both individual and interactive temporal dependencies. Coupled with a curriculum-based contrastive learning strategy, the model achieves precise language-motion alignment. Experiments demonstrate that the proposed approach improves text-to-motion retrieval performance by 12.8% in R@1 on the Inter-X dataset. Furthermore, when integrated as an auxiliary prior into TIMotion and InterMask, it significantly enhances semantic alignment without compromising generation quality (as evidenced by unchanged FID scores), confirming its effectiveness in cross-task transfer.

Abstract

Text-motion representation learning has advanced rapidly, with growing interest in multi person interactions for animation, AR/VR, and embodied AI. These settings require representations that align language with both individual actor dynamics and the relationships between actors. We introduce the Multimodal Interactive Motion Encoder (MIME), which, to our knowledge, represents the first dedicated multimodal encoder designed specifically for two person interactive motion. MIME captures individual and shared structure using stream based co-attention with explicit interaction features and curriculum based contrastive training. On Inter-X text-motion retrieval, MIME consistently outperforms early and late fusion baselines across gallery sizes, achieving a 12.8% relative improvement in text-to-motion R@1 at a 2,000-sample gallery. We further evaluate MIME as a frozen auxiliary prior within TIMotion and InterMask on the unseen InterHuman dataset. MIME improves semantic alignment metrics while maintaining comparable FID in TIMotion. These results show that interaction aware multimodal encoding improves multi person motion retrieval and transfers across datasets to support downstream motion generation.

Problem

• text-motion representation
• multimodal learning
• interactive motion
• two-person interaction
• motion retrieval

Innovation

• multimodal interaction
• interactive motion encoding
• co-attention
• contrastive learning with curriculum
• text-motion alignment