Theoretical Guarantees for One-Shot Magnitude Pruning and Compute-Adaptive Early Exit

2026-09-10

AI Summary

研究通过一次性幅度剪枝和自适应提前退出方法减少神经网络计算量，并证明了这些方法的有效性及误差随计算差距的变化规律。

Abstract

We study compute reduction in neural networks through a unified partial versus full computation view, captured by one-shot magnitude pruning in the static regime and early exit in the adaptive regime. In an asymptotic single-neuron model, we prove a concentration theorem for one-shot magnitude pruning with explicit rates. We also introduce the conditional perceptron for early exit and show that its excess generalization error decays as a power of the compute gap, with an exponent that grows to infinity as the alignment between partial and full computations tends to one. We then extend the analysis to deep networks, characterizing how pruning-induced distortions accumulate with depth and deriving a corresponding compute-accuracy tradeoff for frozen-backbone early exit under a neural network Gaussian process model. Numerical simulations corroborate the predicted scaling laws.

Problem

• one-shot magnitude pruning
• compute-adaptive early exit
• neural networks
• computation reduction
• generalization error

Innovation

• one-shot magnitude pruning
• compute-adaptive early exit
• conditional perceptron
• excess generalization error
• neural network Gaussian process