Mitigating Cache Noise in Test-Time Adaptation for Large Vision-Language Models

2025-03-24

AI Summary

Existing cache-based test-time adaptation (TTA) methods for vision-language models suffer from pseudo-label noise, leading to feature drift, retrieval sensitivity, and difficulty in modeling class-wise distributions. To address these issues, we propose CRG—a zero-shot TTA framework that jointly optimizes cache quality and noise robustness. CRG mitigates cache bias via learnable residual alignment between visual and textual prototypes; it introduces Gaussian Discriminant Analysis (GDA) into the TTA caching mechanism for the first time, enabling dynamic intra-class distribution modeling; and it establishes a unified Cache-Residual-Gaussian协同 framework for end-to-end optimization. Evaluated on 13 benchmarks, CRG significantly outperforms state-of-the-art methods, especially under severe distribution shifts and noisy pseudo-labeling scenarios, demonstrating substantial improvements in both robustness and generalization.

Technology Category

• Computer Vision: Language and Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Test-time adaptation (TTA) of visual language models has recently attracted significant attention as a solution to the performance degradation caused by distribution shifts in downstream tasks. However, existing cache-based TTA methods have certain limitations. They mainly rely on the accuracy of cached feature labels, and the presence of noisy pseudo-labels can cause these features to deviate from their true distribution. This makes cache retrieval methods based on similarity matching highly sensitive to outliers or extreme samples. Moreover, current methods lack effective mechanisms to model class distributions, which limits their ability to fully exploit the potential of cached information. To address these challenges, we introduce a comprehensive and reliable caching mechanism and propose a novel zero-shot TTA method called ``Cache, Residual, Gaussian"(CRG). This method not only employs learnable residual parameters to better align positive and negative visual prototypes with text prototypes, thereby optimizing the quality of cached features, but also incorporates Gaussian Discriminant Analysis (GDA) to dynamically model intra-class feature distributions, further mitigating the impact of noisy features. Experimental results on 13 benchmarks demonstrate that CRG outperforms state-of-the-art TTA methods, showcasing exceptional robustness and adaptability.

Problem

• Mitigating noisy pseudo-labels in cache-based TTA methods
• Improving class distribution modeling for cached features
• Enhancing robustness to outliers in similarity-based cache retrieval

Innovation

• Uses learnable residual parameters for prototype alignment
• Incorporates Gaussian Discriminant Analysis for feature modeling
• Introduces a reliable caching mechanism for noise mitigation