EmojiVoice: Towards long-term controllable expressivity in robot speech

2025-06-18

AI Summary

To address monotonous expressiveness and declining user engagement in long-term voice interactions with social robots, this paper proposes a lightweight, offline-deployable expressive text-to-speech (TTS) method. Our approach builds upon an enhanced Matcha-TTS architecture and introduces *emoji-prompting*—a novel mechanism that jointly embeds emoji semantics and models dynamic prosody for phase-level, temporally adaptive emotional control. To our knowledge, this is the first work enabling real-time, long-duration, fine-grained expressive control on resource-constrained robotic platforms. Experiments in storytelling tasks demonstrate significant improvements in speech naturalness and perceived expressiveness. Furthermore, we uncover scenario-dependent expressive preferences—e.g., between assistant and educational use cases—thereby empirically delineating the method’s applicability boundaries. The proposed framework bridges the gap between expressive TTS and practical deployment constraints in social robotics.

Technology Category

• Intelligent Robots: Embodied AI
• Natural Language Processing: Safety and Robustness
• Cognitive Modeling & Cognitive Systems: Affective Computing

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Humans vary their expressivity when speaking for extended periods to maintain engagement with their listener. Although social robots tend to be deployed with ``expressive'' joyful voices, they lack this long-term variation found in human speech. Foundation model text-to-speech systems are beginning to mimic the expressivity in human speech, but they are difficult to deploy offline on robots. We present EmojiVoice, a free, customizable text-to-speech (TTS) toolkit that allows social roboticists to build temporally variable, expressive speech on social robots. We introduce emoji-prompting to allow fine-grained control of expressivity on a phase level and use the lightweight Matcha-TTS backbone to generate speech in real-time. We explore three case studies: (1) a scripted conversation with a robot assistant, (2) a storytelling robot, and (3) an autonomous speech-to-speech interactive agent. We found that using varied emoji prompting improved the perception and expressivity of speech over a long period in a storytelling task, but expressive voice was not preferred in the assistant use case.

Problem

• Enable long-term expressive speech in robots
• Provide offline-deployable expressive TTS for robots
• Control robot speech expressivity via emoji-prompting

Innovation

• Emoji-prompting for fine-grained expressivity control
• Lightweight Matcha-TTS backbone for real-time speech
• Customizable TTS toolkit for variable expressive speech