3D-Mix for VLA: A Plug-and-Play Module for Integrating VGGT-based 3D Information into Vision-Language-Action Models

2026-03-25

AI Summary

Existing Vision-Language-Action (VLA) models suffer from limited three-dimensional spatial awareness due to their reliance on 2D training data, which constrains their performance in robotic manipulation tasks. This work presents the first systematic evaluation of nine strategies for integrating 3D information into VLA systems and introduces a novel fusion mechanism based on semantic conditional gating. The authors further design 3D-Mix, a lightweight, plug-and-play module that enhances spatial intelligence without requiring modifications to existing multimodal large language models (MLLMs) or action policies. Compatible with mainstream VLA architectures such as GR00T-style and π-style, 3D-Mix demonstrates consistent improvements across MLLM variants ranging from 2B to 8B parameters. Evaluated on the SIMPLER and LIBERO benchmarks, the method achieves an average 7.0% absolute increase in success rate on out-of-domain SIMPLER tasks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Vision-Language-Action (VLA) models leverage Multimodal Large Language Models (MLLMs) for robotic control, but recent studies reveal that MLLMs exhibit limited spatial intelligence due to training predominantly on 2D data, resulting in inadequate 3D perception for manipulation tasks. While recent approaches incorporate specialized 3D vision models such as VGGT to enhance spatial understanding, they employ diverse integration mechanisms without systematic investigation, leaving the optimal fusion strategy unclear. We conduct a comprehensive pilot study comparing nine VGGT integration schemes on standardized benchmarks and find that semantic-conditioned gated fusion, which adaptively balances 2D semantic and 3D geometric features based on task context, achieved the strongest performance among all nine evaluated fusion schemes in our pilot study. We present 3D-Mix, a plug-and-play module that integrates into diverse VLA architectures (GR00T-style and $π$-style) without modifying existing MLLM or action expert components. Experiments across six MLLM series (nine model variants, 2B--8B parameters) on SIMPLER and LIBERO show that 3D-Mix delivers consistent performance gains, averaging +7.0% on the out-of-domain (OOD) SIMPLER benchmark across all nine GR00T-style variants, establishing a principled approach for enhancing spatial intelligence in VLA systems.

Problem

• Vision-Language-Action
• 3D perception
• spatial intelligence
• VGGT
• multimodal fusion

Innovation

• 3D-Mix
• semantic-conditioned gated fusion
• Vision-Language-Action
• spatial intelligence
• plug-and-play module