Learning Multi-Modal Trajectory Policies for Data-Efficient Robotic Manipulation

2026-05-31

AI Summary

This work addresses the challenges of modality interference and inefficient representation in multimodal trajectory policy learning under data scarcity, where shared parameters across visual, linguistic, and trajectory inputs often degrade performance. To mitigate these issues, the authors propose the MATE framework, which leverages sub-token-level feature disentanglement and a cross-modal cosine routing mechanism to enable efficient and robust policy learning. The method incorporates temperature-controlled expert assignment and stochastic noise injection to ensure scale-invariant routing and prevent expert collapse, thereby enhancing stability and generalization in low-data regimes. Experimental results demonstrate that MATE achieves an average 4.75% higher success rate than existing trajectory-conditioned methods on the LIBERO benchmark and effectively guides downstream execution in a real-world robotic table tennis task through accurate trajectory prediction.

Abstract

Robotic manipulation requires the effective integration of heterogeneous inputs, including visual observations, language instructions, and trajectory representations, to generate accurate actions. Existing transformer-based policies typically process these heterogeneous modalities within a shared parameter space, which often leads to modality interference and inefficient representation learning, especially in data-scarce scenarios. While Mixture-of-Experts (MoE) offers a scalable solution through expert specialization, conventional routing mechanisms are often sensitive to such cross-modal representation discrepancies, resulting in unstable expert assignment and expert collapse. In this work, we propose MATE (Multi-ModAl TrajEctory Policies), a novel trajectory prediction framework built upon MoE. Specifically, we introduce a Multi-Modal MoE architecture to achieve fine-grained sub-token feature decoupling, and design a cross-modal cosine router for stable and scale-invariant expert assignment across heterogeneous modalities. We further employ temperature-controlled routing and stochastic noise injection to improve expert balance and prevent premature routing collapse under scarce demonstrations. Experiments on the LIBERO benchmark show that our MATE consistently outperforms prior work under data scarcity. It achieves a 4.75% improvement in average success rate over the trajectory-guided counterpart. Real-world experiments on robotic ping-pong also suggest that the predicted trajectories can provide useful guidance for downstream robotic execution, further indicating the practical feasibility of our algorithm.

Problem

• multi-modal learning
• robotic manipulation
• data efficiency
• Mixture-of-Experts
• trajectory prediction

Innovation

• Multi-Modal MoE
• Cross-Modal Cosine Router
• Trajectory Prediction
• Data-Efficient Manipulation
• Expert Balance