ShamAN-Q: Shampoo Augmented NanoQuant for Sub-1-bit LLM Weights

2026-09-29

AI Summary

This study addresses the significant accuracy degradation in sub-1-bit quantization of large language models caused by neglecting second-order curvature information. We introduce, for the first time, the dense curvature paradigm of the Shampoo optimizer into post-training quantization. Specifically, our method constructs a Kronecker-factored approximation of the empirical Fisher matrix via Kullback–Leibler divergence minimization and reformulates ADMM updates as Sylvester equations. Additionally, we propose layer-wise dynamic refreshing and cross-layer bit reallocation strategies. Evaluated on the Qwen3 series, our approach substantially reduces perplexity (e.g., from 27.56 to 22.96 for the 0.6B model), matching the performance of prior 1-bit methods at approximately 0.8 bits while preserving zero-shot accuracy.

Abstract

We introduce ShamAN-Q, a sub-1-bit post-training quantization method that extends NanoQuant by replacing each its diagonal reconstruction geometry with a tractable dense curvature metric, using a general paradigm popularized by the Shampoo optimizer. For each linear weight, ShamAN-Q fits a Kronecker product to the empirical Fisher information matrix of a small calibration set by Kullback--Leibler minimization, forming a Mahalanobis reconstruction loss from the result. The continuous ADMM updates from NanoQuant become solutions to Sylvester equations, while its discrete projection and deployment format remain unchanged. Because the curvature is local to a given set of weights, ShamAN-Q re-measures the input curvature statistic for each layer immediately before layer factorization, periodically refreshing all statistics on the partially quantized model. ShamAN-Q also redistributes the uniform rank from NanoQuant across layers at the same total number of bits. On Qwen3-Base, ShamAN-Q lowers WikiText-2 perplexity at $\approx$1 bpw from 27.56 to 22.96 (0.6B), 19.21 to 16.72 (1.7B), and 14.29 to 13.80 (4B) while matching or improving zero-shot accuracy on the Eleuther LM Evaluation Harness. On 0.6B, ShamAN-Q at $\approx$0.8 bpw matches the published perplexity of NanoQuant at $\approx$1.0 bpw.

Problem

• Sub-1-bit quantization
• Post-training quantization
• Large Language Models
• Curvature metric
• Weight reconstruction

Innovation

• Sub-1-bit quantization
• Post-training quantization
• Shampoo optimizer
• Kronecker product
• Mahalanobis reconstruction loss