UltraViT: Latency-Optimized On-device Vision Encoder for Large Vision-Language Models

2026-07-25

AI Summary

This work addresses the latency bottleneck in deploying large vision-language models (LVLMs) on edge devices, primarily caused by the high computational cost of visual encoders. To this end, we propose a pyramid-structured visual encoder explicitly optimized for low-latency edge inference. The architecture incorporates a macro block-level heterogeneous spatial mixing module and leverages a two-stage generative pretraining strategy guided by a frozen large language model, further enhanced with dense distillation to strengthen semantic representation and cross-modal alignment. As the first LVLM visual encoder designed directly for on-device latency reduction, our approach achieves nearly 1.7× faster inference than existing encoder-centric baselines while maintaining competitive performance.

Abstract

Large Vision-Language Models (LVLMs) remain bottlenecked by massive computational footprints, precluding their deployment on resource-constrained edge devices. While efforts to compress LVLMs focus heavily on vision token reduction or smaller language models, the vision encoder is largely overlooked, typically deployed as a monolithic, computationally heavy feature extractor. Moreover, there is no previous effort that designs a vision encoder for LVLMs directly optimized for on-device latency. In this paper, we present UltraViT, a vision encoder for LVLMs, explicitly designed and optimized for on-device performance. Specifically, by taking into account real on-device latencies, we systematically design a pyramidal architecture that strategically integrates and adapts heterogeneous spatial mixers at the macro-block level. Furthermore, to pre-train UltraViT, we propose a novel two-stage generative pre-training strategy: cultivating rich spatial features via dense distillation, followed by direct generative supervision from a capacity-mixed frozen LLM. Compared to standard contrastive and SSL, we show that our pre-training is much more effective for achieving high-level semantic grounding for UltraViT needed for the subsequent generative multimodal alignment of LVLM training. Extensive experiments demonstrate that our on-device latency-informed design combined with our tailored training strategy establishes a new state-of-the-art for efficient LVLM encoding, significantly outperforming existing encoder-centric baselines while operating on-device at nearly 1.7xthe speed.

Problem

• Large Vision-Language Models
• on-device latency
• vision encoder
• edge devices
• computational footprint

Innovation

• on-device latency optimization
• pyramidal architecture
• heterogeneous spatial mixers
• two-stage generative pre-training
• dense distillation