🤖 AI Summary
This work addresses the challenge of balancing accuracy and generalization in neural network quantization by proposing a soft quantization method during training. The approach introduces short-range attractive coupling among weights to encourage automatic discretization of the weight distribution, enabling mixed-precision compression without complex scheduling schemes. Relying on only two hyperparameters, the method offers both simplicity and flexibility, providing a novel tool for studying the trade-off between model compression and generalization. Experimental results on ResNet-20 with CIFAR-10 demonstrate that the proposed technique outperforms post-training quantization methods based on histogram equalization, achieving higher accuracy in compressed models.
📝 Abstract
We show that introducing short-range attractive couplings between the weights of a neural network during training provides a novel avenue for model quantization. These couplings rapidly induce the discretization of a model's weight distribution, and they do so in a mixed-precision manner despite only relying on two additional hyperparameters. We demonstrate that, within an appropriate range of hyperparameters, our"soft quantization''scheme outperforms histogram-equalized post-training quantization on ResNet-20/CIFAR-10. Soft quantization provides both a new pipeline for the flexible compression of machine learning models and a new tool for investigating the trade-off between compression and generalization in high-dimensional loss landscapes.