Fast Speech Foundation Model Distillation Using Interleaved Stacking

2026-06-10

AI Summary

Existing distillation methods for speech foundation models often suffer performance degradation when employing stacking strategies to accelerate training. This work proposes a progressive depth-growing stacked training framework, innovatively incorporating an interleaved stacking mechanism that strictly preserves positional consistency across layers while accelerating the distillation process. By mitigating the performance deterioration typically induced by naive stacking, the proposed approach achieves both significantly improved training efficiency and maintained—or even enhanced—model performance, as validated on the SUPERB benchmark.

Abstract

Distilling a large speech foundation model (SFM) into an efficient student model has been successfully applied to low-resource environments. Although distillation reduces inference latency, it requires an additional student model training. However, the training efficiency of SFM distillation remains underexplored. In this work, we explore training acceleration of SFM distillation to speed up model deployment. We examine the potential of stacking, in which the model depth is progressively increased through training until the target model depth is reached. While existing stacking methods improve training speed, they suffer from performance degradation. To handle this limitation, we propose interleaved stacking, a novel stacking method that consistently preserves layer position throughout the stacking process. This property is particularly critical in SFMs, in which each layer encodes distinct layer-specific knowledge. We validate the effectiveness of the proposed method on SUPERB.

Problem

• speech foundation model
• model distillation
• training efficiency
• stacking
• model deployment

Innovation

• speech foundation model
• model distillation
• interleaved stacking
• training acceleration
• layer-specific knowledge