🤖 AI Summary
This study investigates the influence of positional encoding on the internal spatial representation structure and robustness of Vision Transformers. To this end, the authors propose a Spatial Similarity Distance Correlation (SSDC) metric and systematically evaluate stability under content perturbations by combining multiple positional encoding schemes—learned absolute, sinusoidal, and rotary—with representational geometry analysis. The findings reveal that positional encoding establishes an indexing anchoring mechanism that shapes a stable internal spatial organization. Crucially, robustness primarily stems from this stable positional reference frame rather than the specific form of the encoding itself. Models without positional encoding exhibit fragile spatial structures highly susceptible to perturbations, whereas incorporating positional encoding significantly enhances robustness under distributional shifts.
📝 Abstract
Positional embeddings (PEs) in Vision Transformers (ViTs) are known to impact performance and robustness, but their role in shaping internal spatial representations is not well understood. In this work, we study how different forms of PEs influence the representational geometry of ViTs and how these changes relate to robustness under content-disrupting distribution shifts. We introduce a metric, the Spatial Similarity Distance Correlation (SSDC), to quantify spatial structure in token representations. Using this metric, we show that ViTs trained without PEs still develop non-trivial spatial structure, but this structure is driven by visual content and collapses under token permutation. In contrast, we find that all PEs considered (learned absolute, sinusoidal, and rotary) are associated with a consistent shift toward an index-anchored spatial organization. Representations in these models remain stable under perturbations that disrupt content, and exhibit substantially improved robustness to such distributional shifts. We further show that while different PEs produce distinct depth-wise trajectories of spatial structure, their robustness properties are largely similar (with secondary variation across encoding schemes), suggesting that robustness appears to depend on the presence of a stable positional reference frame more than it depends on the specific encoding mechanism. These results offer a geometric account of how positional encodings shape internal representations, with implications for the principled design of future encoding schemes.