🤖 AI Summary
This work investigates how to explicitly control and quantify the information flow within the attention mechanism of Vision Transformers to uncover the evolutionary process from independent local patch processing to the emergence of global representations. The authors introduce a variational information bottleneck along all pathways where attention writes into the residual stream, without altering the model architecture. This approach models the transmitted information as an explicit, tunable variable, enabling precise constraint and intervention on internal communication. Experiments on ImageNet-100 demonstrate that the method effectively modulates the degree of attention coordination and reveals a clear relationship between classification performance and information routing strategies, offering a novel perspective for understanding the internal mechanisms of Vision Transformers.
📝 Abstract
We make the information transmitted by attention an explicit, measurable quantity in vision transformers. By inserting variational information bottlenecks on all attention-mediated writes to the residual stream -- without other architectural changes -- we train models with an explicit information cost and obtain a controllable spectrum from independent patch processing to fully expressive global attention. On ImageNet-100, we characterize how classification behavior and information routing evolve across this spectrum, and provide initial insights into how global visual representations emerge from local patch processing by analyzing the first attention heads that transmit information. By biasing learning toward solutions with constrained internal communication, our approach yields models that are more tractable for mechanistic analysis and more amenable to control.