Neural Network Compression by Approximate Differential Equivalence

2026-05-31

AI Summary

This work proposes a novel global compression paradigm grounded in the dynamic behavioral equivalence of neurons, addressing the limitations of conventional neural network compression methods that rely solely on local weight importance—such as magnitude-based pruning—and often fail to preserve performance under high compression ratios. By encoding a trained network as a system of polynomial ordinary differential equations, the method identifies and merges functionally homogeneous neurons through approximate forward differential equivalence. A single tolerance parameter ε smoothly governs the trade-off between model size and accuracy. Moving beyond weight-centric pruning strategies, this approach achieves substantial parameter reduction while maintaining high fidelity on both synthetic nonlinear dynamical systems and standard regression benchmarks, consistently outperforming existing techniques like magnitude pruning and Wanda at equivalent compression rates.

Abstract

Neural network compression is commonly achieved by pruning parameters based on local importance scores, e.g., magnitude-based pruning. We propose a complementary approach that compresses models by aggregating neurons with similar functional behavior rather than removing weights independently. Our method encodes a trained network as a polynomial ODE system and applies a lumping method called Approximate Forward Differential Equivalence to identify neurons with approximately matching induced dynamics. A single tolerance parameter, $\varepsilon$, controls the compression level and induces a smooth trade-off between model size and predictive accuracy. We evaluate the method on synthetic datasets derived from nonlinear dynamical systems with known ground-truth behavior and on public regression benchmarks. Across both settings, the proposed approach achieves substantial parameter reduction while preserving accuracy, and consistently compares favorably with magnitude-based pruning and Wanda at similar compression levels. These results suggest that differential equivalence-based aggregation is a principled and effective alternative to conventional weight-centric pruning.

Problem

• neural network compression
• neuron aggregation
• differential equivalence
• model reduction
• predictive accuracy

Innovation

• neural network compression
• differential equivalence
• neuron aggregation
• polynomial ODE
• model reduction