analyze attention dynamics

The mechanisms underlying the emergence of semantic structure during diffusion model generation remain poorly understood, and existing approaches struggle to simultaneously capture the dynamic evolution of attention across both spatial and temporal dimensions. This work proposes a novel visual analytics framework that, for the first time, integrates timestep-indexed token-level cross-attention maps with data-driven phase identification. By combining time-series clustering, quantitative attention metrics, and interactive visualization, the framework enables structured analysis of attention dynamics in Stable Diffusion–like models. Evaluated on a benchmark of 60 structured prompts, it reveals interpretable patterns of attention evolution, effectively supporting human-in-the-loop understanding and control of the generative process.

Self-attention lacks an interpretable dynamical systems characterization, hindering theoretical understanding of its relationship with classical machine learning paradigms such as similarity computation and information propagation. Method: We establish, for the first time, that self-attention converges to a drift-diffusion process in the continuous limit, yielding an equivalent heat equation; based on this, we propose metric-attention—a novel attention mechanism grounded in learnable pseudo-metrics that unifies metric learning and attention modeling. Our approach integrates continuous-time modeling, manifold analysis, and diffusion PDE derivation. Contribution/Results: Experiments demonstrate that metric-attention significantly outperforms standard self-attention in training efficiency, accuracy, and adversarial robustness. This work provides a theoretically rigorous yet practically superior paradigm for attention, bridging foundational principles of dynamical systems, geometry, and statistical learning.

How do language models—lacking explicit visual pretraining—achieve image understanding? Method: We systematically analyze 16 multimodal large language models (MLLMs) spanning four architectural families and four parameter scales. Introducing the concept of “vision-preferring attention heads,” we identify such heads via attention behavior analysis, statistical modeling of attention weights, and cross-scale ablation experiments, empirically validating their strong, consistent focus on visual tokens. Contribution: We are the first to discover and formally define this generalizable, modular visual-perception substructure within LLMs. Our work reveals the pivotal role of attention mechanisms in cross-modal adaptation, demonstrating how vision-preferring heads mediate text–vision alignment. This provides an interpretable, spatially localizable mechanism underlying joint text–vision representation learning, thereby advancing research toward transparent, controllable, and analyzable multimodal foundation models.

This work addresses the challenge of enhancing neural networks’ ability to focus on salient information in long-sequence and multimodal tasks. By establishing a unified theoretical framework for attention mechanisms, the study systematically analyzes their mathematical foundations, computational properties, and cross-task generalizability. The framework is instantiated across diverse architectures—including autoregressive Transformers, bidirectional encoders, Vision Transformers, and cross-modal attention models—demonstrating consistent performance gains. The research further uncovers an intrinsic relationship between attention structure and model interpretability, validates empirical scaling laws governing training dynamics and performance, and achieves state-of-the-art results on multiple benchmark datasets. Attention visualization techniques are employed to enhance model transparency, offering insights into the decision-making process of these architectures.

This paper identifies a “visual attention sink” phenomenon in large multimodal models (LMMs): certain visual tokens consistently receive high attention weights despite being semantically irrelevant to the text, thereby impairing cross-modal alignment. To address this, we propose Visual Attention Redistribution (VAR), a training-free, plug-and-play method that recalibrates attention weights via decoder attention analysis, hidden-state activation diagnostics, and identification of centralized attention heads. VAR breaks from conventional optimization paradigms reliant on fine-tuning or architectural modifications—introducing zero trainable parameters and incurring no inference overhead. Extensive evaluation demonstrates significant improvements across general vision-language understanding, visual hallucination suppression, and vision-centric tasks. Our results validate an effective, interpretable pathway for optimizing attention mechanisms in LMMs through post-hoc, analysis-driven weight redistribution.

Traditional approaches to modeling research attention predominantly rely on static aggregate counts, which fail to capture the temporal evolution of attention structures across varying contexts, thereby creating a disconnect between representation and interpretation. This work proposes “attention flow”—a streaming representation that integrates contextual structure with temporal dynamics—to model research attention as an evolvable, structured signal for the first time. By constructing an analogy-based evaluation benchmark, the study systematically compares three representational forms: signals, sequences, and flows. Experimental results demonstrate that attention flow significantly outperforms conventional methods in structural comparison tasks, exhibiting superior robustness and structural transferability, particularly in scenarios influenced by temporal progression or shifts in contextual distributions.

Existing attention visualization methods often rely on specific model architectures and incur high computational costs, lacking lightweight and general-purpose tools for token importance analysis. This work proposes a model-agnostic attribution method that incurs no additional overhead by perturbing inputs and introducing a three-matrix analytical framework: the Angular Deviation Matrix, Magnitude Deviation Matrix, and Dimensional Importance Matrix. These matrices respectively capture semantic directional shifts, magnitude changes, and dimensional contributions, enabling fine-grained and mathematically rigorous assessment of token importance. The approach demonstrates strong efficiency and interpretability across multiple large language models, and the authors release their code to support reproducible research.

This study addresses a critical gap in visualization research by examining how divided attention in real-world multitasking scenarios affects users’ interpretation of visualizations—contrary to the prevailing single-task assumption in existing literature. Through two behavioral experiments integrated with the Linear Ballistic Accumulator (LBA) cognitive model, the work systematically compares user performance under single- and dual-task conditions when interpreting visual designs that either align with or violate viewer expectations (e.g., in color schemes or spatial-semantic mappings). The findings reveal, for the first time, that distraction significantly amplifies the impact of expectation consistency on response times, accuracy, and time-constrained judgment capabilities. These results underscore the crucial importance of expectation-aligned design in authentic multitasking contexts and demonstrate how process-oriented computational modeling can elucidate the underlying cognitive mechanisms.

This work uncovers the structural origin of attention sinks at the first token in large language models, demonstrating that variance disparities in representations—arising from value aggregation in self-attention—are dramatically amplified by hypersensitive neurons in feedforward networks, leading to dimensional imbalance. The study provides the first mechanistic explanation of attention sinks by establishing a complete causal chain from value aggregation and hypersensitive neuron activation to dimensional imbalance. Through targeted interventions such as attention mask modification and variance enhancement at specific tokens, the sink phenomenon is controllably reproduced at arbitrary positions. Furthermore, the authors propose a head-wise RMSNorm architecture that effectively restores statistical equilibrium across token representations, substantially accelerating pretraining convergence.