HiPLEX: Hierarchical Policy Factorization for Full Duplex Speech Language Models

2026-10-06

AI Summary

Existing full-duplex speech models lack joint optimization of timing and content. This work proposes a hierarchical policy factorization framework that decouples dialogue policies into temporal control and content generation, thereby eliminating optimization conflicts in reinforcement learning. Methodologically, the framework integrates event-causal masking, an LLM-based judge, and hierarchical policy networks to enable synergistic training. Experimental results demonstrate that this approach significantly reduces interruption rates during user pauses and shortens response latency following interruptions. Furthermore, the temporal distribution of system turns more closely aligns with natural human conversational patterns, indicating improved turn-taking dynamics in full-duplex spoken dialogue systems.

Abstract

As human--AI interactions become more conversational, full-duplex speech language models capable of natural real-time dialogue are growing in importance. Beyond generating appropriate responses, these models must coordinate turn-taking, backchanneling, and floor management in real time. Reinforcement learning (RL) provides a way to refine these behaviors through direct feedback on interaction outcomes. However, existing RL methods either apply timing feedback to a token policy or optimize semantic content, leaving the joint improvement of timing and content unresolved. We introduce HiPLEX, an RL framework that factorizes a pretrained full-duplex text policy into a control policy that decides when to emit content and a conditional content policy that decides what to emit. The first factor selects among 'pad', 'epad', and 'con'. The second selects a token only when 'con' is chosen. This hierarchy describes conditional actions within each frame and uses the model's existing text head. We route timing advantages to the token-group factor through event-causal masks derived from generated speech episodes, and route an LLM-judge semantic advantage to the conditional content factor. Across three Moshi seeds on Full-Duplex-Bench v1, HiPLEX reduces takeover rates during natural user pauses and backchannel opportunities, and shortens post-interruption response latency relative to GRPO, while maintaining comparable judged interruption-response quality. On Moshi and PersonaPlex, HiPLEX better matches pooled human turn-timing and backchannel-rate marginals than GRPO.

Problem

• Full-duplex speech language models
• Reinforcement learning
• Turn-taking
• Joint optimization of timing and content
• Policy factorization

Innovation

• Full Duplex Speech Language Models
• Hierarchical Policy Factorization
• Reinforcement Learning
• Turn-taking
• Advantage Routing