Calibration Data Trade-offs Across Capability Dimensions: Why Multi-Source Mixing Matters for High-Sparsity LLM Pruning

2026-06-02

AI Summary

This work addresses the challenge that under high-compression sparsification, a single calibration dataset fails to adequately preserve the multidimensional capabilities—such as general reasoning, commonsense, coding, and mathematical skills—of large language models. The study reveals inherent trade-offs among different calibration sources across these capability dimensions and proposes an Information-Guided Self-Calibration Protocol (IGSP) that constructs multi-source mixed calibration data without requiring aligned corpora. Integrating Spearman correlation analysis, OIT-based information measurement, 4-gram aggregation minimization, perplexity balancing, and the SparseGPT pruning framework, IGSP achieves a total capability retention rate of 58.8% at 60% sparsity on LLaMA-3.1-8B, outperforming the best single-source baseline by 8.8 percentage points and surpassing Self-Cal by 2.4 percentage points.

Abstract

Post-training pruning compresses large language models to high sparsity using a small unlabelled calibration set, and recent work has concluded that the choice of calibration source has only modest impact on averaged post-pruning accuracy. We ask whether this conclusion survives once calibration impact is evaluated separately across distinct capability dimensions rather than aggregated. Decomposing post-pruning capability into General, Commonsense, Code, and Math, and analysing $n{=}15$ calibration sources via Spearman correlations between OIT information metrics and per-dimension retention, we uncover an opposite-sign trade-off: calibration perplexity correlates positively with General retention ($ρ{=}{+}0.71$) but negatively with Math and Code retention ($ρ{=}{-}0.53,\,{-}0.59$; $p{<}0.05$), so no single source can preserve all capabilities. We respond with multi-source calibration mixing, and propose IGSP, an information-guided self-calibration protocol that automates multi-source construction without capability-aligned corpora by minimising 4-gram aggregation and balancing perplexity across dimensions. On LLaMA-3.1-8B at SparseGPT 60% sparsity, a uniform multi-source mix reaches 58.8% total retention, outperforming the best single source (MetaMath, 50.0%) by $+8.8$ and the C4 default (40.0%) by $+18.8$; IGSP improves over Self-Cal by $+2.4$ and SGS by $+4.8$.

Problem

• calibration data
• capability trade-offs
• LLM pruning
• multi-source mixing
• high sparsity

Innovation

• multi-source calibration
• high-sparsity pruning
• capability trade-offs
• information-guided self-calibration
• IGSP