Towards More Expressive Spoken LLMs: Fine-Grained Intent Benchmarking and Acoustic-Lexical Decoupled Policy Optimization

2026-08-03

AI Summary

Current speech large language models struggle to simultaneously capture both explicit semantic content and implicit paralinguistic emotional intent in user utterances, and lack fine-grained evaluation benchmarks and joint optimization strategies for this dual objective. To address this gap, this work introduces ParaIntent, a Chinese fine-grained intent benchmark encompassing 14 categories of explicit and implicit intents, along with Acoustic-Lexical Policy Optimization (ALPO), a decoupled optimization approach. ALPO operates within a unified generative framework but employs separate textual and acoustic advantage functions to independently refine emotional expression in each modality. Experimental results demonstrate that, under identical reward functions and training budgets, ALPO outperforms standard GRPO on most automatic metrics and achieves the highest subjective ratings on both synthetic and human-recorded test sets, with particularly notable gains in emotional expressiveness.

Abstract

Spoken emotional dialogue requires a model to understand a user's spoken input and generate a response that is both semantically appropriate and emotionally expressive. This is challenging because communicative intent may be stated explicitly in lexical content or conveyed more implicitly through paralinguistic cues, which can complement or diverge from the words themselves. However, two limitations constrain progress in this area: the scarcity of benchmarks that distinguish these intent expressions, and the lack of reinforcement learning objectives that jointly account for response quality and emotional expression. To address the lack of suitable benchmarks, we introduce ParaIntent, a Chinese benchmark comprising 14 intent categories with balanced explicit and implicit samples, together with a multidimensional evaluation protocol covering intent fulfillment, response quality, and emotional expression. For policy optimization, existing approaches either use a shared objective for text and speech or apply reinforcement learning to only one modality, leaving modality-specific learning signals entangled within policy optimization. Motivated by this, we propose Acoustic-Lexical Decoupled Policy Optimization (ALPO), which computes independent textual and acoustic advantages and routes them to the corresponding text and speech tokens within a unified rollout. Under identical reward functions and training budgets, ALPO improves over standard GRPO on most automatic metrics and achieves the best subjective results among the fine-tuned variants, with particularly clear gains in emotional expressiveness on both the synthetic and human-recorded test sets.

Problem

• spoken dialogue
• intent expression
• emotional expressiveness
• benchmarking
• multimodal policy optimization

Innovation

• spoken LLMs
• intent benchmarking
• acoustic-lexical decoupling
• policy optimization
• emotional expressiveness