PersonaTAB: Predicting Personality Traits using Textual, Acoustic, and Behavioral Cues in Fully-Duplex Speech Dialogs

2025-05-20

AI Summary

The absence of personality annotations hinders personalized adaptation in voice-based dialogue systems. Method: This paper proposes the first personality-aware modeling framework for full-duplex voice dialogue. It integrates textual, acoustic, and behavioral multimodal cues and introduces an ASR- and LLM-driven end-to-end pipeline for automatic personality annotation generation. To enhance temporal coherence, it incorporates sequential modeling of emotion and response types, coupled with a human-in-the-loop annotation protocol for fine-grained personality trait prediction. Contribution/Results: The work establishes the first multimodal collaborative personality annotation framework, overcoming the key bottleneck of implicit-label-free automatic personality modeling. Human evaluation demonstrates significantly higher inter-annotator agreement compared to state-of-the-art baselines, empirically validating strong alignment between predicted personality traits and actual conversational behaviors.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Cognitive Modeling & Cognitive Systems: Affective Computing

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Despite significant progress in neural spoken dialog systems, personality-aware conversation agents -- capable of adapting behavior based on personalities -- remain underexplored due to the absence of personality annotations in speech datasets. We propose a pipeline that preprocesses raw audio recordings to create a dialogue dataset annotated with timestamps, response types, and emotion/sentiment labels. We employ an automatic speech recognition (ASR) system to extract transcripts and timestamps, then generate conversation-level annotations. Leveraging these annotations, we design a system that employs large language models to predict conversational personality. Human evaluators were engaged to identify conversational characteristics and assign personality labels. Our analysis demonstrates that the proposed system achieves stronger alignment with human judgments compared to existing approaches.

Problem

• Predicting personality traits from speech dialogues using multimodal cues
• Creating annotated datasets for personality-aware conversational agents
• Improving alignment with human judgments in personality prediction

Innovation

• Preprocesses raw audio for annotated dialogue dataset
• Uses ASR and LLMs for personality prediction
• Engages human evaluators for personality labeling