Visual Cue Enhancement and Dual Low-Rank Adaptation for Efficient Visual Instruction Fine-Tuning

2024-11-19

arXiv.org

AI Summary

To address weak fine-grained visual understanding, inter-task data conflicts, and low efficiency in visual instruction tuning of multimodal large language models (MLLMs), this paper proposes EVIT, an efficient visual instruction tuning framework. Methodologically, EVIT introduces (1) a novel Visual Cue Enhancement (VCE) mechanism that fuses multi-level visual features to strengthen fine-grained perception, and (2) a Dual-LoRA architecture—comprising two decoupled low-rank adaptation modules—that separately parameterizes skill space and task space, enabling parameter-efficient and precisely controllable cross-task transfer. By enhancing the visual projector and integrating hierarchical visual features, EVIT significantly improves visual understanding accuracy and cross-task generalization on both general benchmarks and downstream tasks, achieving state-of-the-art performance with lightweight implementation.

Technology Category

Computer Vision: Multi-modal VisionMachine Learning: Large Multimodal Models (LMMs)Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved informationSemantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMsEconomics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd work

Abstract

Parameter-efficient fine-tuning multimodal large language models (MLLMs) presents significant challenges, including reliance on high-level visual features that limit fine-grained detail comprehension, and data conflicts that arise from task complexity. To address these issues, we propose an efficient fine-tuning framework with two novel approaches: Vision Cue Enhancement (VCE) and Dual Low-Rank Adaptation (Dual-LoRA). VCE enhances the vision projector by integrating multi-level visual cues, improving the model's ability to capture fine-grained visual features. Dual-LoRA introduces a dual low-rank structure for instruction tuning, decoupling learning into skill and task spaces to enable precise control and efficient adaptation across diverse tasks. Our method simplifies implementation, enhances visual comprehension, and improves adaptability. Experiments on both downstream tasks and general benchmarks demonstrate the effectiveness of our proposed approach.

Problem

• Adapt MLLMs to tasks with minimal computational overhead
• Resolve data conflicts in diverse complex tasks
• Enhance vision-language projection with local details

Innovation

• Dual-LoRA framework for holistic-to-local adaptation
• Visual Cue Enhancement for local feature aggregation
• Memory- and time-efficient adapter optimization