Federated Multilingual Speech-LLMs: Architecture and Aggregation Strategy Benchmarking

2026-09-20

AI Summary

研究通过比较不同架构和聚合策略，解决了联邦学习在多语言自动语音识别中的性能优化问题，提出独立调整各组件学习率可获得最佳效果。

Abstract

We present a comprehensive benchmark of Federated Learning (FL) for multilingual Automatic Speech Recognition (ASR), evaluating four Speech-LLM architectures on the Multilingual LibriSpeech dataset. We compare FedAvg and FedProx across frozen and unfrozen encoder configurations, demonstrating that optimized learning rates are critical for performance. Specifically, independently tuning the learning rates for the speech encoder, connector, and decoder yields the lowest error rates, with full three-component adaptation (LoRA for encoder and decoder, full training for the connector) producing the best FL results. We observe that FedProx efficacy is architecture-dependent, providing notable advantages in multilingual pre-trained architectures (e.g., EuroLLM over TinyLlama when keeping the encoder fixed); this indicates that LLM backbone capacity plays a key role in mediating resilience to heterogeneous data distributions. These findings offer concrete design guidance for deploying multilingual Speech-LLMs in privacy-sensitive, distributed environments.

Problem

• Federated Learning
• Multilingual ASR
• Speech-LLM
• Aggregation Strategy
• Performance Optimization

Innovation

• Federated Learning
• Learning Rate Optimization
• LoRA
• Architecture Dependence
• Heterogeneous Data