🤖 AI Summary
The computational principles underlying Transformer attention remain poorly understood.
Method: We establish, for the first time, a theoretical correspondence between attention and Pavlovian conditioning—mapping queries, keys, and values to test stimuli, conditioned stimuli, and unconditioned stimuli, respectively—and formalize attention as Hebbian-driven formation of transient associative memory. Leveraging a linear attention model, we integrate associative learning theory, matrix analysis, and error propagation analysis to rigorously characterize this memory process.
Contribution/Results: We derive a fundamental single-head capacity bound of *O*(√*dₖ*), where *dₖ* is the key dimension, and uncover inherent trade-offs among model depth, width, and attention head redundancy. Furthermore, we propose a biologically plausible learning rule grounded in this framework, offering a novel theoretical foundation for designing efficient and neuroscientifically credible Transformer architectures.
📝 Abstract
Transformer architectures have revolutionized artificial intelligence (AI) through their attention mechanisms, yet the computational principles underlying their success remain opaque. We present a novel theoretical framework that reinterprets the core computation of attention as Pavlovian conditioning. Our model finds a direct mathematical analogue in linear attention, which simplifies the analysis of the underlying associative process. We demonstrate that attention's queries, keys, and values can be mapped to the three elements of classical conditioning: test stimuli that probe associations, conditional stimuli (CS) that serve as retrieval cues, and unconditional stimuli (US) that contain response information. Through this lens, we suggest that each attention operation constructs a transient associative memory via a Hebbian rule, where CS-US pairs form dynamic associations that test stimuli can later retrieve. Our framework yields several theoretical insights grounded in this linearized model: (1) a capacity theorem showing that attention heads can store O($sqrt{d_k}$) associations before interference degrades retrieval; (2) an error propagation analysis revealing fundamental architectural trade-offs of balancing model depth, width, and head redundancy to maintain reliability; and (3) an understanding of how biologically plausible learning rules could enhance transformer architectures. By establishing this deep connection, we suggest that the success of modern AI may stem not from architectural novelty alone, but from implementing computational principles that biology optimized over millions of years of evolution.