When Does Knowledge Distillation Hurt? Reliability-Aware Distillation for Low-Resource Language Summarization

2026-07-22

AI Summary

Standard knowledge distillation yields limited gains—and often harms performance—in low-resource language summarization. To address this, this work proposes two reliability-aware distillation approaches: CHAD evaluates sample utility via gradient alignment and incorporates a lightweight gating mechanism, while EWAD+CPDP combines entropy-weighted adaptive distillation with a capacity-proportional geometric constraint to handle lexically incompatible teachers. The study introduces the first method for quantifying distillation-induced harm at the sample level and integrates counterfactual gating with complementary signals from multiple teachers to overcome low-resource performance bottlenecks. On BanSum, CHAD and EWAD+CPDP improve ROUGE-L by +0.0173 and +0.0219, respectively—substantially outperforming standard distillation (+0.0003)—and achieve these gains with only 60M parameters, surpassing Qwen2.5-3B and outperforming baselines in 10 out of 15 languages.

Abstract

Knowledge distillation (KD) is a standard approach for compressing sequence-to-sequence models, but its per-sample effects are rarely examined. On the BanSum Bangla summarization benchmark, we find that standard KD improves ROUGE-L by only +0.0003 over a cross-entropy baseline, and that approximately 51.3% of training samples are estimated to actively harm student validation loss under standard KD. We propose two complementary reliability-aware distillation methods. CHAD (Counterfactual Harm-Aware Distillation) measures per-sample KD usefulness via gradient alignment with the validation loss direction and trains a lightweight gate that generalizes this counterfactual judgment to the full training set. EWAD+CPDP combines token-level entropy-weighted adaptive distillation with a capacity-proportional geometric constraint from a second, vocabulary-incompatible teacher. On BanSum, both methods substantially outperform standard KD: CHAD by +0.0173 ROUGE-L and EWAD+CPDP by +0.0219 ROUGE-L, where standard KD itself improves ROUGE-L by only +0.0003; despite using only 60M parameters, both outperform a fine-tuned Qwen 2.5-3B model (50x larger). We further evaluate the stronger method, EWAD+CPDP, across 15 typologically diverse XL-Sum languages organised into three sets, beating the CE-only baseline on 10/15 languages; gains are most reliable where the two teachers contribute complementary signal, and weakest where they have saturated or jointly weak target-language coverage. We release code and trained models to support reproducibility and further research on selective distillation.

Problem

• Knowledge Distillation
• Low-Resource Language Summarization
• Reliability-Aware Distillation
• Student Model Harm
• Per-Sample Effects

Innovation

• reliability-aware distillation
• counterfactual harm estimation
• entropy-weighted adaptive distillation
• cross-teacher geometric constraint
• low-resource summarization