Quantization robustness from dense representations of sparse functions in high-capacity kernel associative memory

2026-04-22

AI Summary

This study addresses the challenge of deploying high-capacity kernel associative memory models, which are hindered by substantial computational overhead, by investigating their compressibility for hardware-efficient implementation. Leveraging spontaneous symmetry breaking and Walsh analysis, the authors develop a geometric framework that elucidates the behavior of kernel-based Hopfield networks under compression. Their analysis reveals a striking dichotomy: these models exhibit strong robustness to low-precision quantization yet extreme sensitivity to pruning. This asymmetry stems from their underlying encoding principle—“sparse functions with dense representations.” The work not only uncovers the theoretical origin of this disparity in robustness but also provides a principled and practical pathway toward hardware-friendly compressed kernel memory systems, supported by rigorous theoretical foundations.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Machine Learning: Kernel Methods
• Data Mining & Knowledge Management: Data Compression

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Models for Web evolution

Abstract

High-capacity associative memories based on Kernel Logistic Regression (KLR) are known for their exceptional performance but are hindered by high computational costs. This paper investigates the compressibility of KLR-trained Hopfield networks to understand the geometric principles of its robust encoding. We provide a comprehensive geometric theory based on spontaneous symmetry breaking and Walsh analysis, and validate it with compression experiments (quantization and pruning). Our experiments reveal a striking contrast: the network is extremely robust to low-precision quantization but highly sensitive to pruning. Our theory explains this via a ``sparse function, dense representation'' principle, where a sparse input mapping is implemented with a dense, bimodal parameterization. Our findings not only provide a practical path to hardware-efficient kernel memories but also offer new insights into the geometric principles of robust representation in neural systems.

Problem

• quantization robustness
• kernel associative memory
• Hopfield networks
• model compression
• sparse functions

Innovation

• quantization robustness
• dense representation
• sparse functions
• kernel associative memory
• spontaneous symmetry breaking