ParVL: Parallel Scaling and Expandable Compute Allocation for Multimodal LLMs

2026-08-04

AI Summary

This work addresses the inflexible computational allocation in existing large vision-language models, which struggle to dynamically adjust resource distribution between visual and language components according to task demands and often incur additional overhead. To overcome this limitation, we propose ParVL, a parallel scaling framework that, under a fixed backbone parameter budget, constructs multiple parallel branches sharing a common vision (ViT) and language (LLM) backbone while introducing branch-specific prefix parameters to enable task-adaptive dynamic computation allocation. ParVL represents the first parallel multimodal scaling mechanism built upon a shared backbone, supporting end-to-end full-parameter fine-tuning. Experiments demonstrate that, under identical training settings, ParVL significantly outperforms single-branch baselines, and the optimal vision–language computation ratio varies across tasks, validating the efficacy of flexible allocation strategies.

Abstract

Existing scaling strategies for Multimodal Large Language Models (MLLMs) typically expand either model parameters or sequential inference computation, incurring substantial memory or latency overhead. More importantly, most existing methods fail to alter the rigid, fixed computation allocation between the Vision Transformer and the Large Language Model components, limiting task-specific optimization. To address this, we introduce the Parallel Vision-Language (ParVL) scaling framework for MLLMs, which scales parallel computation by reusing the existing ViT and LLM backbone parameters across multiple vision and language branches. This framework raises a central question: given a fixed backbone parameter budget, how should additional shared-backbone computation be allocated between the vision and language modalities? We instantiate each parallel computational stream with branch-specific prefix parameters over a shared backbone, and train the entire model end-to-end via full-parameter supervised fine-tuning on roughly 13B tokens. We systematically study the computation-allocation trade-off between the ViT encoder and LLM decoder. ParVL improves overall multimodal performance over same-recipe single-branch baselines, and the best evaluated vision--language allocation varies across tasks. Code is available at https://github.com/YangYangGirl/ParVL.

Problem

• Multimodal Large Language Models
• computation allocation
• scaling strategies
• Vision Transformer
• LLM

Innovation

• Parallel Scaling
• Expandable Compute Allocation
• Multimodal LLMs
• Shared Backbone
• Prefix Parameters