Consistent Evidence, Robust Recognition: Faithful Attribution Regularization under Geometric Transformations

2026-07-26

AI Summary

This work addresses the inconsistency of existing attribution methods under geometric transformations and the limited fidelity of conventional gradient-based attributions in reflecting true model decision rationales. The authors propose an unsupervised attribution regularization framework that, for the first time, leverages submodular search to generate compact, class-discriminative, and faithful attribution supervision signals. They introduce path consistency and termination alignment losses to enable differentiable joint regularization of the discrete evidence selection process. Evaluated on ImageNet-100, the method substantially improves attribution stability and Insertion/Deletion metrics with only a 0.28% accuracy drop for ViT-B/16. On ImageNet-1K, it also enhances robustness to transformations while incurring no more than a 0.30% loss in clean accuracy.

Abstract

Attribution methods are widely used to characterize the evidence underlying model predictions, yet their potential to improve model behavior remains underexplored. Attribution inconsistency under label-preserving geometric transformations may indicate transformation-sensitive evidence reliance, motivating attribution regularization. However, such supervision is valid only when attribution faithfully reflects the evidence driving predictions. Existing self-supervised methods typically align gradient-based maps such as Grad-CAM, whose limited faithfulness means that attribution consistency need not imply consistency of the underlying decision process, leaving transformation robustness unresolved. We propose an annotation-free attribution regularization framework based on submodular search over image regions. By measuring how candidate subsets affect model outputs, the search extracts compact, class-discriminative evidence as search-derived supervision. We further introduce a submodular ranking loss with path-consistency and termination-alignment terms that respectively align spatially corresponding candidate rankings along paired search trajectories and encourage the transformed trajectory to satisfy the stopping criterion at the target terminal step. The loss provides a differentiable surrogate for regularizing both final attributions and the otherwise discrete evidence-selection process. Experiments on ImageNet-100 show that our method substantially improves attribution stability, Insertion, and Deletion on ViT-B/16 with only a 0.28-point accuracy drop, with similar gains on ViT-L/16. On ImageNet-1K, it improves transformed-input accuracy on ResNet-50 and ConvNeXt-B while limiting the clean-accuracy drop to 0.30 points, demonstrating more consistent evidence reliance with minimal performance loss. Code will be released soon.

Problem

• attribution consistency
• geometric transformations
• faithful attribution
• model robustness
• evidence reliance

Innovation

• attribution regularization
• submodular search
• geometric invariance
• faithful explanation
• differentiable surrogate