Efficient Supernet Training with Orthogonal Softmax for Scalable ASR Model Compression

2025-01-31

AI Summary

To address the challenge of adaptively balancing model size and performance for automatic speech recognition (ASR) under diverse hardware constraints, this paper proposes a supernet-based framework for jointly training variable-size encoders. The core innovation is OrthoSoftmax—a novel orthogonalized softmax mechanism that enables efficient, fine-grained subnetwork identification and selection without the computational overhead of conventional neural architecture search (NAS). OrthoSoftmax supports FLOPs-aware, multi-criteria, and multi-granularity component-level dynamic configuration, while uncovering structural regularities in subnetwork selection. Evaluated on LibriSpeech and TED-LIUM-v2, models of various sizes derived from a single supernet training run achieve word error rates (WER) on par with—or even superior to—those of individually trained counterparts. This approach significantly improves training efficiency and deployment flexibility across heterogeneous hardware platforms.

Technology Category

• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP
• Machine Learning: Hardware-aware ML

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

ASR systems are deployed across diverse environments, each with specific hardware constraints. We use supernet training to jointly train multiple encoders of varying sizes, enabling dynamic model size adjustment to fit hardware constraints without redundant training. Moreover, we introduce a novel method called OrthoSoftmax, which applies multiple orthogonal softmax functions to efficiently identify optimal subnets within the supernet, avoiding resource-intensive search. This approach also enables more flexible and precise subnet selection by allowing selection based on various criteria and levels of granularity. Our results with CTC on Librispeech and TED-LIUM-v2 show that FLOPs-aware component-wise selection achieves the best overall performance. With the same number of training updates from one single job, WERs for all model sizes are comparable to or slightly better than those of individually trained models. Furthermore, we analyze patterns in the selected components and reveal interesting insights.

Problem

• Automatic Speech Recognition (ASR)
• Model Scalability
• Performance Adaptation

Innovation

• Orthogonal Softmax
• Super Network Training
• Automatic Speech Recognition (ASR)