Attention Lattice Adapter: Visual Explanation Generation for Visual Foundation Model

2025-09-18

AI Summary

Weak interpretability of vision foundation models and the incompatibility of existing explanation methods with complex architectures pose significant challenges. To address these, this paper proposes the Attention Lattice Adapter (ALA) and the Alternating Epoch Architecture (AEA), enabling adaptive visual explanation generation without manual layer specification. ALA extends the receptive field via structured sparse attention, while AEA jointly optimizes explanation fidelity and model fine-tuning performance through parameter alternation across training epochs. The method integrates multi-dimensional evaluation metrics—including IoU, insertion, and deletion scores—to enhance both explanation faithfulness and model adaptability. Extensive experiments on CUB-200-2011 and ImageNet-S demonstrate consistent superiority over baselines: on CUB-200-2011, the average IoU improves by up to 53.2 percentage points. These results validate the effectiveness and generalizability of the proposed framework.

Technology Category

• Computer Vision: Interpretability, Explainability, and Transparency
• Machine Learning: Deep Neural Architectures and Foundation Models
• Humans and AI: Explainable AI (XAI) for Human Understanding

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization

Abstract

In this study, we consider the problem of generating visual explanations in visual foundation models. Numerous methods have been proposed for this purpose; however, they often cannot be applied to complex models due to their lack of adaptability. To overcome these limitations, we propose a novel explanation generation method in visual foundation models that is aimed at both generating explanations and partially updating model parameters to enhance interpretability. Our approach introduces two novel mechanisms: Attention Lattice Adapter (ALA) and Alternating Epoch Architect (AEA). ALA mechanism simplifies the process by eliminating the need for manual layer selection, thus enhancing the model's adaptability and interpretability. Moreover, the AEA mechanism, which updates ALA's parameters every other epoch, effectively addresses the common issue of overly small attention regions. We evaluated our method on two benchmark datasets, CUB-200-2011 and ImageNet-S. Our results showed that our method outperformed the baseline methods in terms of mean intersection over union (IoU), insertion score, deletion score, and insertion-deletion score on both the CUB-200-2011 and ImageNet-S datasets. Notably, our best model achieved a 53.2-point improvement in mean IoU on the CUB-200-2011 dataset compared with the baselines.

Problem

• Generating visual explanations in visual foundation models
• Overcoming lack of adaptability in complex models
• Enhancing interpretability through parameter updates

Innovation

• Attention Lattice Adapter simplifies layer selection automatically
• Alternating Epoch Architect updates parameters every other epoch
• Method enhances interpretability by partial parameter updates