Learning Spatial Decay for Vision Transformers

2025-08-13

AI Summary

Vision Transformers (ViTs) lack explicit spatial inductive bias in self-attention, resulting in weak spatial structure modeling. Existing spatial decay methods rely on fixed distance metrics—ignoring image content—and thus suffer from limited generalizability. To address this, we propose a data-dependent spatial decay mechanism and introduce Context-Aware Gating (CAG), the first such gating module for 2D ViTs. CAG jointly encodes spatial priors (Manhattan distance) and learnable content representations to enable dynamic, adaptive inter-patch attention modulation. This establishes a novel paradigm unifying spatial and content-aware dynamic decay. Extensive experiments on ImageNet-1K classification and generative modeling demonstrate significant improvements over strong baselines, validating the method’s effectiveness, robustness, and cross-task generalization capability.

Technology Category

• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Vision Transformers (ViTs) have revolutionized computer vision, yet their self-attention mechanism lacks explicit spatial inductive biases, leading to suboptimal performance on spatially-structured tasks. Existing approaches introduce data-independent spatial decay based on fixed distance metrics, applying uniform attention weighting regardless of image content and limiting adaptability to diverse visual scenarios. Inspired by recent advances in large language models where content-aware gating mechanisms (e.g., GLA, HGRN2, FOX) significantly outperform static alternatives, we present the first successful adaptation of data-dependent spatial decay to 2D vision transformers. We introduce extbf{Spatial Decay Transformer (SDT)}, featuring a novel Context-Aware Gating (CAG) mechanism that generates dynamic, data-dependent decay for patch interactions. Our approach learns to modulate spatial attention based on both content relevance and spatial proximity. We address the fundamental challenge of 1D-to-2D adaptation through a unified spatial-content fusion framework that integrates manhattan distance-based spatial priors with learned content representations. Extensive experiments on ImageNet-1K classification and generation tasks demonstrate consistent improvements over strong baselines. Our work establishes data-dependent spatial decay as a new paradigm for enhancing spatial attention in vision transformers.

Problem

• ViTs lack spatial biases for structured tasks
• Fixed spatial decay limits adaptability in vision
• Need dynamic spatial-content fusion for 2D attention

Innovation

• Context-Aware Gating for dynamic decay
• Manhattan distance-based spatial priors
• Unified spatial-content fusion framework