CoCurve: Cross-Module Co-Pruning Curvature for Training-Free Structured LLM Pruning

2026-07-20

AI Summary

This work addresses a key limitation in existing training-free structured pruning methods for Transformers, which evaluate the importance of attention heads and feed-forward network (FFN) channels in isolation, thereby neglecting their joint dependencies induced by residual connections and often leading to the erroneous removal of critical component combinations. To overcome this, the authors propose a calibration-only, fine-tuning-free joint pruning approach that introduces, for the first time, a cross-module co-pruning curvature metric. This metric leverages off-diagonal elements of the Fisher information matrix to capture the collaborative impact of unit pairs on model loss. The full interaction matrix is efficiently reconstructed via Gram products of single-ablation features, and pruning is formulated as a budget-constrained quadratic programming problem derived from a second-order Taylor expansion of token-level KL divergence between the frozen model and its masked replica. The method significantly outperforms current training-free pruning techniques without requiring labels, fine-tuning, or recovery training.

Abstract

Structured pruning compresses large language models (LLMs) by removing whole computational units, such as attention heads and feed-forward (FFN) channel groups. Most training-free methods, however, rank these units independently, implicitly treating the loss from pruning a set as the sum of its individual losses. This view fails for Transformers, whose sublayers are coupled through a shared residual stream. Two individually weak units can thus be jointly indispensable, yet independent scoring is blind to such dependence and removes them together. We introduce CoCurve (Cross-Module Co-Pruning Curvature), a calibration-only, fine-tuning-free method that prunes attention and FFN units jointly. A second-order Taylor expansion of the token-level KL between the frozen model and its masked copy yields a single Fisher matrix whose diagonal is classical node saliency and whose off-diagonal entries are co-pruning curvature edges: the extra damage of removing two units together. Under a single-ablation additivity approximation this matrix reduces to a Gram product of single-unit ablation features, so the full M x M interaction is recovered from M forward passes, with no pairwise sweeps or gradients. Pruning then reduces to one budgeted quadratic program, solved in a single shot under a shared attention--FFN budget, with no labels, fine-tuning, or recovery.

Problem

• structured pruning
• large language models
• cross-module dependency
• training-free pruning
• unit interaction

Innovation

• CoCurve
• structured pruning
• training-free
• co-pruning curvature
• Fisher information