🤖 AI Summary
This work addresses a critical limitation in existing neural network compression methods: their neglect of cross-layer redundancy arising from functional symmetries, such as permutation invariance among hidden units and attention heads. To overcome this, the authors propose a novel framework that aligns symmetric blocks across layers via motion compensation, transforming weight sequences into predictable structures. The approach introduces a lightweight per-layer predictor, a rate–distortion-optimized entropy model, and a keyframe scheduling mechanism to efficiently encode quantized residuals. During decoding, inverse alignment enables rapid weight reconstruction. By explicitly modeling cross-layer alignment for the first time, the method achieves substantial improvements over state-of-the-art quantization and learned compression techniques on Transformer-based language modeling and vision classification tasks, significantly advancing the rate–accuracy Pareto frontier while preserving inference speed.
📝 Abstract
Neural network weights are increasingly a bottleneck for deployment, yet most compression pipelines treat layers independently and overlook cross-layer redundancy induced by function-preserving symmetries. We propose Motion-Compensated Weight Compression (MCWC), a weight-only codec that aligns permutation-symmetric blocks (e.g., hidden units and attention heads) to maximize cross-layer correspondence, turning depth into a predictable sequence. In the aligned coordinate system, MCWC uses a lightweight layer-sequential predictor with periodic keyframes and encodes only quantized prediction residuals using a learned entropy model trained under a rate distortion objective. A simple decoder reconstructs deployable weights by entropy decoding, dequantization, predictor-driven reconstruction, and inverse alignment, enabling fast weight materialization for inference. Across Transformer language modeling and vision classification, MCWC improves the rate accuracy Pareto frontier over strong quantization and learned weight-codec baselines, while maintaining competitive decode time. Ablations confirm that alignment, prediction, entropy modeling, and keyframe scheduling are each necessary for the full gains. Our code is available via https://github.com/Ism-ail11/MCWC.