🤖 AI Summary
Existing self-attention and related models lack a unified theoretical framework, hindering systematic understanding and extension. This work proposes the “localization method”—a general machine learning framework grounded in localization kernels and local averages—that formalizes local modeling through rigorous theoretical constructs and localization techniques. For the first time, it subsumes state-of-the-art architectures such as Transformers under a unified local modeling paradigm. The framework not only reveals intrinsic connections among diverse models—including kernel methods, MeanShift, Hopfield networks, Locally Linear Embedding (LLE), fuzzy inference, denoising autoencoders, and Transformers—but also introduces scalable hierarchical local and non-local models, thereby establishing a novel paradigm for building data-adaptive learning systems.
📝 Abstract
This paper proposes a general machine learning framework called the localization method, which is fundamentally built on two core concepts: localization kernels and local means -- key components that underpin the self-attention mechanism. To establish a rigorous theoretical foundation, the framework is formally defined through two essential pillars: the formulation of the local(-ized) model and the localization trick. We systematically investigate the connections between the localization method and a wide range of existing machine learning models/methods, including (but not limited to) kernel methods, lazy learning, the MeanShift algorithm, relaxation labeling, Hopfield networks, local linear embedding (LLE), fuzzy inference, and denoising autoencoders (DAEs). By dissecting these relationships, we clarify the broader theoretical significance of the localization method and demonstrate its practical applicability across diverse machine learning tasks. Furthermore, we explore advanced extensions of the framework, such as adaptive kernels, hierarchical local models, and non-local models. Notably, we show that the Transformer -- a cornerstone of modern sequence modeling -- can be constructed using hierarchical local models, revealing the ability of the localization method to unify and generalize state-of-the-art architectures. This work not only provides a unified theoretical lens to reinterpret existing models but also offers new methodological tools for designing flexible, data-adaptive learning systems.