Problem
Research questions and friction points this paper is trying to address.
fairness
training-seed variability
speech LLM
automatic speech recognition
demographic bias
Fairness Beyond a Single Run: Training-Seed Variability in Speech LLM Adaptation
🤖 AI Summary
This study addresses the vulnerability of single-run evaluations in adapting large speech models, where training stochasticity can obscure fairness issues. We reveal for the first time that training seeds exert a substantially greater influence on demographic fairness than conventional factors such as audio compression rates. By fine-tuning Q-former projectors and LoRA adapters across multiple seeds on the Common Voice and Fair-Speech datasets, we propose a fairness attribution method based on 3×3 variance decomposition to quantify the contribution of seed variation. Our findings demonstrate that 85.3% of racial fairness disparities stem from seed variability, and we establish 0.30 as the minimum significant difference threshold that exceeds noise levels. This work corrects biases inherent in single-run evaluations, providing a more reliable benchmark for assessing fairness in large models.
📝 Abstract
Demographic fairness gaps in automatic speech recognition are almost always reported from a single training run. We fine-tune the Q-former projector and LoRA adapters of a speech LLM at five audio compression factors and six random seeds, holding the encoder, base decoder, data and decoding fixed, and evaluate every run on Common Voice and Fair-Speech. At 460 h of clean LibriSpeech, the seed moves fairness metrics more than compression does on most demographic axes. A balanced 3x3 decomposition attributes 85.3% of the variation in Fair-Speech ethnicity normalized gap to the seed against 8.3% to compression (p = 0.009), though compression explains more on age and gender. Held-out LibriSpeech word error rate spreads by 0.04 points across those seeds while Common Voice spreads by 8.57, so these are not failed runs, and the effect survives controlling for accuracy and dropout. Scaling and diversifying the adaptation set to 960 h damps the effect but does not remove it. On Fair-Speech ethnicity, two single-run systems must differ by more than 0.30 in normalized gap to exceed seed variability.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Training-Seed Variability
Speech LLM Adaptation
Demographic Fairness
LoRA Fine-tuning
Variance Decomposition
🔎 Similar Papers
No similar papers found.
