Layerwise Decoupling for Stable Structured Sparsification of Fully Connected Layers

2026-09-17

AI Summary

提出了一种分层解耦方法，用于全连接层的结构稀疏化，通过逐层处理子网络并施加结构组惩罚来剪枝神经元，相比联合方法更稳健。

Abstract

We propose a decoupled, layerwise method for structurally sparsifying the fully connected layers of pretrained neural networks. Rather than penalizing all layers jointly, our approach extracts shallow two-layer subnetworks, normalizes the inner weights, and applies a structured group penalty to the outer weight matrix of each block, processing layers sequentially to prune neurons and reduce the width of each layer. We prove that the constrained decoupled objective is equivalent at optimality to a specific joint penalty on the inner and outer weights, for any positively homogeneous activation, and thus admits a clean projected and proximal formulation. Our central finding is that this decoupled reformulation is more robust than coupled methods. In numerical experiments it provides a wider usable range of the regularization strength and a lower rate of catastrophic over-pruning than the tested joint baseline while maintaining comparable accuracy. We establish these properties in controlled classification and sparse-recovery studies, and examine their scope in a high-dimensional PINN stress test and in the feed-forward layers of OPT-1.3B.

Problem

• structurally sparsifying
• fully connected layers
• pretrained neural networks
• catastrophic over-pruning

Innovation

• decoupled method
• structured sparsification
• fully connected layers
• neural networks
• robust regularization