oversmoothing mitigation

This work investigates how attention mechanisms in Transformers mitigate oversmoothing through sink and diagonal patterns that enable attention switching. By integrating geometric analysis, theoretical proofs, and empirical validation, the study establishes—for the first time—an equivalence between sink tokens and hard attention switching, clarifies the precise conditions under which sinks effectively prevent oversmoothing, and quantitatively explains why pretrained models exhibit a preference for sink-based representations. The paper further introduces a diagonal pattern that permits self-communication as a more flexible mechanism for suppressing oversmoothing, thereby generalizing the applicability of attention switching. Additionally, it provides a quantitative comparison of the representational costs of sink versus diagonal patterns and elucidates the mechanistic conditions under which attention layers degenerate into MLP-like behavior.

This work reveals that Message Passing Neural Networks (MPNNs) suffer from oversquashing even on short-range graph tasks—challenging the prevailing view that oversquashing affects only long-range tasks. The authors decouple oversquashing into two distinct mechanisms: (i) short-range bottlenecks, arising from local neighborhood structure-induced information capacity limits, and (ii) long-range gradient vanishing. Crucially, existing mitigation strategies—such as virtual nodes—are shown to be ineffective against short-range bottlenecks. Through theoretical analysis and empirical evaluation, the study demonstrates that Graph Transformers, leveraging global attention, effectively alleviate this bottleneck and significantly outperform MPNNs on short-range tasks. This is the first systematic characterization of short-range oversquashing as a standalone phenomenon, with an independent causal origin. The findings provide novel theoretical insights and empirical evidence to guide principled architecture selection in graph representation learning.

This work identifies a critical mechanism by which random edge dropping (e.g., DropEdge) exacerbates over-squashing in message-passing neural networks (MPNNs): it attenuates node sensitivity to distant neighbors and contracts the effective receptive field, thereby degrading long-range modeling capacity. We provide the first theoretical characterization of this phenomenon, demonstrating that it improves short-range task performance while significantly impairing accuracy on long-range tasks—and inducing short-range overfitting and generalization failure. Through systematic empirical evaluation across multiple real-world graph datasets, we analyze structural dropout strategies—including DropEdge, DropNode, DropAgg, and DropGNN—validating our theoretical insights. Our core contributions are: (i) establishing a causal link between stochastic structural dropout and over-squashing; (ii) arguing that deep GNN training must explicitly accommodate long-range interactions; and (iii) providing both theoretical foundations and practical guidance for designing robust graph learning algorithms capable of capturing distant dependencies.

Deep graph neural networks (GNNs) suffer from three structural bottlenecks—over-smoothing, over-squashing, and underreaching—arising from synchronous message passing. To address these limitations, this paper proposes the first variational inference-based adaptive message-passing framework. Our method jointly optimizes both propagation depth and message-path selection via learnable skip connections, dynamic message gating, and asynchronous depth adaptation, enabling precise modeling of long-range dependencies. Theoretical analysis demonstrates that our framework significantly enhances long-distance information capture. Extensive experiments across five mainstream node- and graph-level prediction benchmarks achieve state-of-the-art performance, validating both effectiveness and generalizability.

Long-range information propagation in Graph Neural Networks (GNNs) is fundamentally hindered by the *oversquashing* problem, causing exponential decay of information flow with increasing hop count. To address this, we introduce SWAN—a novel GNN architecture grounded in dynamical systems theory—that jointly enforces global and local non-dissipativity via dual anti-symmetric parameterizations in both the spatial and weight domains. By ensuring energy conservation and nonlinear stability throughout message passing, SWAN theoretically guarantees constant-rate long-range information propagation. Empirically, SWAN significantly alleviates oversquashing on synthetic and real-world long-range interaction benchmarks. It consistently improves performance on node classification and graph-level prediction tasks, demonstrating enhanced modeling of long-range dependencies. Our work establishes non-dissipative dynamics as a fundamental mechanism for boosting the expressive power of GNNs.

This work addresses the over-smoothing problem in deep graph convolutional networks (GCNs), where node representations collapse as network depth increases. It is the first to model over-smoothing as a depth-dependent phenomenon in the spectral domain and proposes a lightweight, interpretable low-rank spectral adaptation mechanism. By introducing learnable low-rank correction terms in the spectral domain, the method adaptively adjusts the fixed Laplacian propagation operator, selectively mitigating representation contraction while preserving the low-pass inductive bias inherent to GCNs. Notably, this approach significantly delays over-smoothing without requiring any redesign of the message-passing architecture. Experiments on multiple benchmark datasets demonstrate that the effective depth of GCNs nearly doubles, and both embedding variance and spectral analysis confirm a substantial alleviation of representation collapse.

This work addresses the oversquashing problem that hinders message-passing neural networks in modeling long-range dependencies. The authors propose a lightweight global mixing module that effectively alleviates information bottlenecks by integrating local message passing with low-complexity spectral filtering, without requiring strong theoretical assumptions. The method incorporates standard stability constraints to ensure both robustness in feature transformation and computational efficiency. Experimental results demonstrate that the proposed approach reduces model error by up to an order of magnitude while cutting parameter count by as much as 50% across diverse tasks, including long-range graph learning, knowledge graph question answering, and mesh-based fluid dynamics simulations.

This work uncovers the structural origin of attention sinks at the first token in large language models, demonstrating that variance disparities in representations—arising from value aggregation in self-attention—are dramatically amplified by hypersensitive neurons in feedforward networks, leading to dimensional imbalance. The study provides the first mechanistic explanation of attention sinks by establishing a complete causal chain from value aggregation and hypersensitive neuron activation to dimensional imbalance. Through targeted interventions such as attention mask modification and variance enhancement at specific tokens, the sink phenomenon is controllably reproduced at arbitrary positions. Furthermore, the authors propose a head-wise RMSNorm architecture that effectively restores statistical equilibrium across token representations, substantially accelerating pretraining convergence.

This work addresses the commonly oversimplified notion of “attention sinks” in attention mechanisms, revealing that they actually encompass two distinct computational phenomena: adaptive no-ops (nop) and broadcast operations. The study formally distinguishes these mechanisms for the first time and introduces verifiable diagnostic criteria based on value vector norms and output rank. Through synthetic tasks and empirical analysis of pretrained vision Transformers, the authors demonstrate their coexistence and layer-wise distribution patterns. Furthermore, the research shows that existing intervention strategies—such as gating and register tokens—implicitly favor one mechanism over the other, and that combining both approaches yields complementary benefits, leading to improved model stability and performance.

Inspired by the human phenomenon of “insight,” this work proposes Enlightenment, a training-free post-tuning paradigm designed to elicit abrupt, emergent capability improvements in large-scale models. Moving beyond existing training-free approaches that solely adjust attention weights, Enlightenment introduces an architecture-aware shortcut rewiring mechanism—specifically, attention head mixing shortcuts for large language models (LLMs) and scalar modulation of residual connections for vision-language models (VLMs). Experimental results demonstrate that Enlightenment consistently achieves significant performance gains across diverse models and benchmarks, effectively unlocking latent capabilities embedded in pretrained architectures without additional training.