MuRA: Multi-Rank Adaptation for Efficient and Effective Test-Time Vision-Language Generalization

2026-08-04

AI Summary

This work addresses the significant performance degradation of vision-language models under distribution shifts by proposing MuRA, a multi-rank adaptation framework that overcomes the limitations of existing test-time low-rank adaptation methods constrained by fixed-rank configurations, which often lead to underfitting or overfitting. MuRA dynamically selects and fuses low-rank adaptation modules of varying capacities based on token-level visual complexity of the input image. It introduces a novel combination of multi-rank orthogonal decomposition initialization, continuous routing updates, and a unified component fusion strategy, while optimizing gradient pathways at the deepest visual layer to enhance stability. Experiments demonstrate that MuRA achieves state-of-the-art accuracy across multiple domain generalization and cross-dataset benchmarks, while substantially reducing computational overhead and memory consumption.

Abstract

Vision-language models exhibit remarkable zero-shot capabilities but suffer significant performance degradation under distribution shifts. While test-time adaptation (TTA) via Low-Rank Adaptation offers a parameter-efficient solution, we identify a fundamental bottleneck in current methods: the reliance on static rank configurations. Because visual inputs inherently possess varying information densities, a fixed rank forces an inevitable optimization compromise, leading to underfitting on complex scenes and overfitting on simple ones. To bridge this gap, we propose Multi-Rank Adaptation (MuRA), a novel framework that dynamically selects and fuses adaptation modules of varying capacities based on token-level visual complexity. MuRA synergizes Multi-Rank Orthogonal Decomposition to provide a superior, knowledge-preserving initialization, and Unified Component Fusion with Continuous Router Updating to sustainably learn semantic-to-rank mappings. Furthermore, we provide rigorous theoretical justifications mathematically proving the necessity and gradient stability of this adaptive mechanism. Crucially, MuRA's dynamic design uniquely thrives at the deepest visual layer, capitalizing on the shortest gradient backpropagation path. Extensive experiments demonstrate that MuRA achieves state-of-the-art accuracy across extensive domain generalization and cross-dataset benchmarks while significantly reducing both computational and memory overhead.

Problem

• test-time adaptation
• low-rank adaptation
• distribution shift
• vision-language models
• rank configuration

Innovation

• Multi-Rank Adaptation
• Test-Time Adaptation
• Vision-Language Models
• Dynamic Rank Selection
• Low-Rank Adaptation