Latent-IM: Latent Interaction Management for Speech LLMs

2026-07-29

AI Summary

This work addresses the limited explicit control over dialogue acts in large language models (LLMs), which hinders their ability to reliably generate specific conversational behaviors such as confirmation or questioning. To overcome this, the authors propose the Latent-IM framework, which preserves the end-to-end architecture of LLMs while decoupling dialogue act selection from utterance generation, thereby recovering capabilities akin to traditional systems in state estimation and action control. By integrating contextual modeling, causal generation control, and latent-space intervention, Latent-IM enables plug-and-play guidance of dialogue acts. Experimental results demonstrate that, on a human dialogue act reproduction task, the method improves end-to-end dialogue act accuracy by 12.5 percentage points over an unguided baseline, achieving performance comparable to fine-tuned approaches.

Abstract

Classical spoken dialogue systems often separated dialogue management from response realization: a policy selected the next dialogue action, and a generation component expressed that action. As dialogue systems shift toward LLMs, this decomposition has largely disappeared into the model's hidden representations. We ask whether an LLM-internal analogue of state estimation and action control can be recovered for conversational moves such as acknowledging, checking, querying, explaining, and replying. We formulate move control as two coupled problems: selection, predicting the appropriate next move from the dialogue context, and realization, causally producing a chosen move at generation time. We introduce Latent-IM, an internal dialogue-management framework that provides a general interface for choosing and deploying conversational moves under different objectives. Here, we use this control to reproduce human move choices, improving average end-to-end move accuracy by 12.5 points over the unsteered backbone while performing comparably to fine-tuning.

Problem

• dialogue management
• conversational moves
• speech LLMs
• latent control
• response realization

Innovation

• Latent Interaction Management
• Speech LLMs
• Conversational Moves
• Dialogue Control
• Move Realization