Harness TTS: Towards Context-Aware Expressive Speech Synthesis with Harness Layer

2026-07-20

AI Summary

This work addresses the challenge of harmonizing explicit user commands with implicit interaction context in expressive speech synthesis for voice assistants. We propose Harness TTS, which formulates style control as a closed-set prompt-tool routing task grounded in large language models (LLMs). By introducing a structured prompt-tool registry, a priority-aware observation mechanism, and a lightweight control layer, our approach enables externalized, auditable, and low-latency context-aware modulation of TTS expressive behavior. Integrated with a Qwen3-4B planner and CosyVoice3/VoxCPM2 synthesizers, the system achieves Top-1 routing accuracies of 74.3%, 43.0%, and 64.6% in explicit, implicit, and conflicting scenarios, respectively. Synthesized speech significantly outperforms instruction-only control, yielding UTMOSv2 gains of 0.11–0.38 while maintaining first-token latency under 50 ms.

Abstract

Expressive speech synthesis for voice assistants requires flexible style control that adapts to explicit requests and broader interaction context. We propose Harness TTS, a lightweight control layer that wraps around a TTS engine to externalize and govern its expressive behavior. It reformulates style control as closed-set prompt-tool routing: offline, a compact registry of stylistic prompt tools is constructed with structured metadata; online, an LLM planner selects the appropriate tool based on a priority-aware observation schema, and the TTS executor synthesizes speech using the corresponding prompt audio. We evaluate Harness TTS on both routing and synthesis tasks. In routing, Qwen3-4B achieves Top-1 accuracies of 74.3%, 43.0%, and 64.6% on explicit, implicit, and conflict subsets. For synthesis, experiments on CosyVoice3 and VoxCPM2 show that Harness TTS outperforms instruction-only control, achieving higher instruction-following win rates (margins of 23.1-35.6 points on CosyVoice3 and 13.8-20.0 points on VoxCPM2) and improving UTMOSv2 scores by 0.11-0.38. Moreover, the 4B planner delivers its first tool recommendation in under 50 ms in standard mode, introducing negligible latency for real-time interaction. These results demonstrate that equipping TTS engines with a dedicated Harness layer offers a practical, auditable, and context-aware solution for voice assistant expression control.

Problem

• expressive speech synthesis
• style control
• context-aware
• voice assistants
• TTS

Innovation

• Harness TTS
• context-aware speech synthesis
• prompt-tool routing
• expressive TTS
• LLM-based planning