Adaptive Prompt Tuning: Vision Guided Prompt Tuning with Cross-Attention for Fine-Grained Few-Shot Learning

2024-12-19

AI Summary

To address the challenges of large intra-class variation and scarce labeled data in fine-grained few-shot image classification—which degrade model discriminability—this paper proposes a vision-guided adaptive prompt tuning method. Our approach integrates CLIP and ViT architectures to jointly optimize visual and textual representations. Key contributions include: (1) a novel vision-driven dynamic text prompt generation mechanism, where cross-attention dynamically aligns image patches with textual prompts; and (2) the first coupling of cross-attention with Monte Carlo Dropout to simultaneously enhance classification accuracy and enable well-calibrated uncertainty estimation. Evaluated on three fine-grained benchmarks—CUBirds, Oxford Flowers, and FGVC Aircraft—the method significantly outperforms mainstream baselines including CoOp and VPT, achieving an average accuracy improvement of 5.2%. It thus delivers both strong discriminative capability and reliable, uncertainty-aware predictions.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Prompt Engineering / Prompting

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Few-shot, fine-grained classification in computer vision poses significant challenges due to the need to differentiate subtle class distinctions with limited data. This paper presents a novel method that enhances the Contrastive Language-Image Pre-Training (CLIP) model through adaptive prompt tuning, guided by real-time visual inputs. Unlike existing techniques such as Context Optimization (CoOp) and Visual Prompt Tuning (VPT), which are constrained by static prompts or visual token reliance, the proposed approach leverages a cross-attention mechanism to dynamically refine text prompts for the image at hand. This enables an image-specific alignment of textual features with image patches extracted from the Vision Transformer, making the model more effective for datasets with high intra-class variance and low inter-class differences. The method is evaluated on several datasets, including CUBirds, Oxford Flowers, and FGVC Aircraft, showing significant performance gains over static prompt tuning approaches. To ensure these performance gains translate into trustworthy predictions, we integrate Monte-Carlo Dropout in our approach to improve the reliability of the model predictions and uncertainty estimates. This integration provides valuable insights into the model's predictive confidence, helping to identify when predictions can be trusted and when additional verification is necessary. This dynamic approach offers a robust solution, advancing the state-of-the-art for few-shot fine-grained classification.

Problem

• Computer Vision
• Image Classification
• Limited Training Data

Innovation

• Adaptive Prompt Tuning
• Dynamic Cross-Attention
• Monte Carlo Dropout