Decoupled Visual Processing: Efficient Multimodal Adaptation via Modality-Specific Transformer Substitution

2026-07-29

AI Summary

This work addresses the high computational cost and representational mismatch between vision and language modalities in full-parameter fine-tuning of multimodal large language models for visual instruction tuning. To improve parameter efficiency, the authors propose a decoupled visual processing framework built upon LLaVA-1.5, which freezes the original decoder and explicitly separates visual and textual tokens at the top layer. A lightweight, trainable Transformer block is introduced exclusively for processing visual tokens, enabling distinct pathways for each modality. Despite updating only a minimal fraction of the model’s parameters, this approach achieves performance on par with full fine-tuning across multiple benchmarks—including MME, POPE, and ChartQA—demonstrating significantly enhanced parameter efficiency without compromising task performance.

Abstract

Multimodal large language models (MLLMs) have demonstrated remarkable capabilities by integrating visual and textual understanding within a unified transformer architecture. However, fine-tuning all parameters of these models for visual instruction tuning is computationally expensive and often unnecessary, as the representation requirements for visual and textual tokens diverge significantly in the deeper layers of the network. In this paper, we propose Decoupled Visual Processing (DVP), an efficient training framework that replaces the upper decoder layers of a pretrained LLM with a lightweight, independently trainable single transformer block dedicated exclusively to visual token processing. Specifically, after shared processing through the first half of the decoder layers, visual and textual tokens are split: visual tokens are routed through a newly initialized single transformer block while textual tokens continue through the original frozen decoder layers. The two streams are then concatenated before the language modeling head. During training, only the single transformer block is updated, dramatically reducing the number of trainable parameters. Experiments on the LLaVA-1.5 framework demonstrate that DVP achieves competitive performance on MME, POPE, and ChartQA benchmarks while training only a fraction of the total parameters, suggesting that visual representations in MLLMs can be effectively learned through a decoupled, parameter-efficient pathway.

Problem

• multimodal large language models
• visual instruction tuning
• parameter-efficient adaptation
• modality-specific processing
• visual-textual representation divergence

Innovation

• Decoupled Visual Processing
• Modality-Specific Transformer
• Parameter-Efficient Tuning
• Multimodal Large Language Models
• Visual Instruction Tuning