Technology Category
Computer Vision: Large Vision ModelsMachine Learning: Matrix & Tensor MethodsNatural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
VISTA: Visualization of Token Attribution via Efficient Analysis
🤖 AI Summary
Existing attention visualization methods often rely on specific model architectures and incur high computational costs, lacking lightweight and general-purpose tools for token importance analysis. This work proposes a model-agnostic attribution method that incurs no additional overhead by perturbing inputs and introducing a three-matrix analytical framework: the Angular Deviation Matrix, Magnitude Deviation Matrix, and Dimensional Importance Matrix. These matrices respectively capture semantic directional shifts, magnitude changes, and dimensional contributions, enabling fine-grained and mathematically rigorous assessment of token importance. The approach demonstrates strong efficiency and interpretability across multiple large language models, and the authors release their code to support reproducible research.
Technology Category
Application Category
📝 Abstract
Understanding how Large Language Models (LLMs) process information from prompts remains a significant challenge. To shed light on this "black box," attention visualization techniques have been developed to capture neuron-level perceptions and interpret how models focus on different parts of input data. However, many existing techniques are tailored to specific model architectures, particularly within the Transformer family, and often require backpropagation, resulting in nearly double the GPU memory usage and increased computational cost. A lightweight, model-agnostic approach for attention visualization remains lacking. In this paper, we introduce a model-agnostic token importance visualization technique to better understand how generative AI systems perceive and prioritize information from input text, without incurring additional computational cost. Our method leverages perturbation-based strategies combined with a three-matrix analytical framework to generate relevance maps that illustrate token-level contributions to model predictions. The framework comprises: (1) the Angular Deviation Matrix, which captures shifts in semantic direction; (2) the Magnitude Deviation Matrix, which measures changes in semantic intensity; and (3) the Dimensional Importance Matrix, which evaluates contributions across individual vector dimensions. By systematically removing each token and measuring the resulting impact across these three complementary dimensions, we derive a composite importance score that provides a nuanced and mathematically grounded measure of token significance. To support reproducibility and foster wider adoption, we provide open-source implementations of all proposed and utilized explainability techniques, with code and resources publicly available at https://github.com/Infosys/Infosys-Responsible-AI-Toolkit
Problem
Research questions and friction points this paper is trying to address.
attention visualization
model-agnostic
token attribution
large language models
interpretability
Innovation
Methods, ideas, or system contributions that make the work stand out.
model-agnostic
token attribution
perturbation-based analysis
attention visualization
explainable AI
🔎 Similar Papers
No similar papers found.
