Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model

2026-07-27

AI Summary

This work addresses the inefficiency of existing vision-language models in streaming perception tasks, which hinders real-time understanding. The authors propose a natively encoder-decoder–based, efficient streaming multimodal foundation model that innovatively integrates a Mage-ViT visual encoder with a biologically inspired dual-system architecture. It employs a motion-vector- and residual-energy–driven sparse visual tokenization strategy—featuring dynamic 16×16 region selection and I/P-frame anchoring—to process only high-entropy regions. A lightweight System 1 performs event gating, while a causal System 2 handles reasoning. The model matches Qwen3-VL-4B in static tasks, significantly outperforms Phi-4-reasoning-vision (15B) in video understanding and spatial reasoning, achieves up to 3.5× faster inference, and reduces visual token consumption by over 75%.

Abstract

Standard vision-language models (VLMs) suffer from Moravec's paradox: they excel at complex offline visual reasoning but struggle with simple streaming perception tasks and process them inefficiently. We present Mage-VL, an efficient codec-native streaming foundation model for real-time multimodal understanding and interaction. At its core, our custom tokenizer, Mage-ViT, replaces uniform frame sampling by selectively encoding dynamic, entropy-rich regions using motion vectors and residual energy across sparse anchor (I) and predicted (P) frames. Operating at a 16 x 16 patch level, this reduces visual token consumption by over 75% while preserving spatiotemporal context. Trained from scratch on approximately 560M unlabeled images and 100M unlabeled video frames, Mage-ViT matches or outperforms flagship encoders trained on billions of image-text pairs. We establish AI4AI data pipelines encompassing prompt-code joint optimization for multimodal captioning and AI-driven performance diagnosis to guide training recipes. Furthermore, through a bio-inspired dual-system architecture - a lightweight System 1 event gate and a causal System 2 decoder - Mage-VL enables proactive streaming perception. Extensive evaluations show that Mage-VL-4B matches Qwen3-VL-4B on static tasks while achieving strong gains in video understanding and 2D/3D spatial reasoning, with up to a 3.5x wall-clock inference speedup, and comprehensively surpasses the 15B Phi-4-reasoning-vision baseline. Beyond model artifacts, we deliver seven key empirical findings covering pre-training data efficiency, variable-resolution scaling, codec system acceleration, VideoQA SFT redundancy, motion-spatial synergy, AI4AI data pipelines, and Zero-Vision SFT for multimodal RL.

Problem

• vision-language models
• streaming perception
• Moravec's paradox
• real-time multimodal understanding
• efficient inference

Innovation

• codec-native
• streaming multimodal model
• selective tokenization
• dual-system architecture
• AI4AI data pipeline