STARFISH: faST Accuracy Recovery in pruned networks From Internal State Healing

2026-05-31

AI Summary

Model pruning often leads to significant accuracy degradation, and existing recovery techniques offer limited effectiveness. This work proposes STARFISH, a novel method that, for the first time, restores performance by optimizing the internal states of the pruned network to align with the representations of the original unpruned model. Requiring only a minuscule unlabeled calibration set, STARFISH integrates internal representation alignment, label-free calibration, and a lightweight repair strategy. When applied to a DeiT-B model with 75% of its weights pruned, STARFISH recovers 82% of the original accuracy using merely 0.4% of ImageNet images—substantially outperforming current state-of-the-art methods, which achieve only around 40% recovery under the same high sparsity regime.

Abstract

Pruning is a process designed to reduce the number of weights in a large neural network. This can substantially speed up inference but might cause a considerable reduction in the model's accuracy, and thus it is usually followed by a healing process that regains some of the lost accuracy. In this paper, we propose a new healing method, STARFISH, that can recover (most of) the accuracy of any pruned network efficiently. The main idea of STARFISH is to optimize the pruned network to align with the original network's internal state representations using a tiny calibration set of unlabeled examples. For the common case of removing 50% of the weights, STARFISH healing improves the recovered accuracy by up to 22% over the state-of-the-art methods on ViT-based networks. Its advantage is even more pronounced under aggressive pruning. For example, after eliminating 75% of the weights in a DeiT-B network for ImageNet, STARFISH uses only 0.4% of the number of training images as a calibration set and recovers 82% of the original dense accuracy, whereas competing recovery techniques reach only 40% of the dense model accuracy.

Problem

• pruning
• accuracy recovery
• neural networks
• model compression
• internal state

Innovation

• pruning
• accuracy recovery
• internal state alignment
• calibration set
• model compression