saliency mapping

This work addresses the limited interpretability of current vision-language models (VLMs) in understanding data visualizations, which hinders verification of whether their reasoning focuses on semantically relevant image regions. The authors propose a lightweight diagnostic saliency mapping method that, for the first time, aggregates attention weights across all layers and attention heads of a Transformer model with respect to visual tokens and back-projects them onto the image patch grid to establish direct correspondences between generated text and specific image regions. This approach requires no gradient computation and efficiently produces causally faithful explanations. Experimental results demonstrate that the resulting saliency maps accurately highlight the regions attended by the model, and deletion tests confirm their causal fidelity to the model’s behavior.

Existing saliency map methods often produce explanations with high generalizability but low discriminability, failing to precisely localize class-specific decision evidence. To address this, we propose the Cross-Class Attribution Fusion (CCAF) framework—the first to explicitly integrate inter-class attributions by decoupling shared features from discriminative ones. CCAF enhances attribution specificity in a model- and method-agnostic manner via plug-and-play aggregation of gradient- and perturbation-based attributions, guided by counterfactual contrastive masking. The framework comprises three stages: attribution aggregation, class-contrastive masking, and randomized robustness validation. Evaluated on grid-pointing localization and randomized sanity checks, CCAF significantly improves discriminative accuracy across mainstream attribution methods. It reliably identifies both class-discriminative and class-shared visual evidence on multiple benchmarks, advancing the fidelity and interpretability of post-hoc explanations.

Explainability of deep learning image classifiers suffers from a trade-off: pixel-level attribution maps (e.g., saliency maps) lack generalizability, while global explanations oversimplify model behavior. To bridge this gap, we propose Segment Attribution Tables (SATs), which align saliency maps with semantic segmentation masks to aggregate attributions into named, semantically meaningful image segments (e.g., “eye”, “background”). SATs operate post-hoc—requiring no model modification—and are compatible with any saliency method. They quantify model reliance on semantic regions, enabling stable detection of spurious correlations (e.g., watermarks, background biases) and redundant features. Experiments demonstrate that SATs effectively expose model biases even when out-of-distribution performance remains largely intact—uncovering flaws undetected by conventional metrics. By providing interpretable, mid-granularity insights, SATs serve as a practical diagnostic and debugging tool for model behavior analysis.

Large vision models suffer from poor interpretability, and conventional attribution methods merely localize attended regions without providing semantic explanations. Method: This paper proposes a dual-path framework—“aligning with human reasoning” and “concept-level explanation”—featuring the first formally guaranteed attribution method based on verification-aware perturbation analysis (EVA), and the CRAFT-MACO collaborative framework for automatic discovery, importance quantification, and visualization of internal model concepts. It integrates algorithmic stability metrics, Sobol indices computed via quasi-Monte Carlo sampling, 1-Lipschitz function-space optimization, and concept activation mapping. Results: The approach enables interactive, concept-level explanations across all 1,000 ImageNet classes on ResNet, significantly improving explanation fidelity and human consistency—particularly in complex scenes—while offering rigorous theoretical guarantees and actionable semantic insights.

Existing pixel-level feature attribution methods neglect the structural semantics of input data—such as texture and transients—and lack cross-modal consistency. To address this, we propose the Wavelet Attribution Method (WAM), the first interpretability framework grounded in wavelet theory, enabling unified attribution across images, audio, and 3D shapes. WAM simultaneously answers *where* features are important (spatial/temporal localization) and *why* they matter (multi-scale frequency-domain semantic interpretation), achieved via continuous wavelet transform, gradient-driven attribution, and a cross-modal adaptation architecture that explicitly models the semantic meaning of structural components. Evaluated on classification tasks across all three modalities, WAM matches or surpasses state-of-the-art methods, achieving an average 12.7% improvement in faithfulness metrics—demonstrating the effectiveness and generalizability of its dual-dimensional (localization + semantic) explanatory capability.

This work addresses the limitation of existing visual saliency models, which predominantly rely on the free-viewing assumption and thus struggle to capture task-driven attentional patterns. To overcome this, the authors propose a task-driven saliency prediction model that explicitly integrates natural language descriptions of task semantics into the visual attention mechanism for the first time, establishing a task-conditioned architecture for saliency prediction. Experimental results demonstrate that the proposed approach effectively captures the dynamic shifts in human attention across different tasks and significantly improves prediction accuracy in task-oriented viewing scenarios compared to conventional methods.

Existing attribution methods for image geolocation models struggle to reveal whether predictions rely on human-interpretable, object-level visual cues. This work proposes an object-centric analysis pipeline that first extracts salient regions from attribution maps such as Grad-CAM, then decomposes them into object-like elements using image segmentation. The predictive relevance of these elements is rigorously evaluated through crop-based deletion and insertion tests. This approach enables, for the first time, an object-level interpretation of attribution outcomes. Experiments across three benchmark datasets demonstrate that attribution-guided cropping preserves significantly more predictive information than random cropping, providing strong evidence that geolocation models indeed leverage localized, interpretable object-level cues in their decision-making process.

Existing layout-to-image generation methods suffer from fragmented representations under few-shot, atypical scenarios, leading to image distortion and loss of detail. This work proposes a representation-driven framework that explicitly decouples semantic identity from visual primitives for the first time. Semantic anchoring aggregates category-level semantics to stabilize object identity, while primitive injection models recombinable local primitives to enhance fine-grained details. Furthermore, a concept-guided mechanism incorporating saliency-aware optimization is introduced to improve foreground semantic consistency. Evaluated under a strict 5-shot setting, the proposed method consistently outperforms state-of-the-art approaches across multiple atypical domains, achieving notable improvements in both visual fidelity and layout alignment.

Existing path-based feature attribution methods define trajectories in the input space, rendering them susceptible to path artifacts and unable to discern the semantic significance of input perturbations, which leads to unstable explanations. This work proposes Reveal-IG, a novel framework that lifts path attribution from the input space into a structured probe distribution space centered around the target sample, computing integrated gradients along distributional paths with respect to the model’s expected output. By supporting multi-scale image probes and modeling feature uncertainty in tabular data, Reveal-IG preserves attribution completeness while avoiding input-space artifacts. Experiments demonstrate that Reveal-IG produces stable, signed attributions on ImageNet classification and tabular regression tasks, significantly outperforming existing methods on sign-dependent metrics and remaining competitive on others, with synthetic diagnostics further confirming its robustness against artifacts.