Looking for Affect in Spontaneous Finnish Speech through Linguistic Interpretability

2026-07-27

AI Summary

This study addresses the unclear relative contributions of acoustic and linguistic features to human perception of emotional valence and arousal in spontaneous Finnish speech. Leveraging a newly released Finnish emotional speech corpus, it presents the first systematic investigation integrating textual and audio modalities through multimodal feature extraction and regression modeling to examine their joint role in affective perception. Results demonstrate that multimodal fusion significantly enhances valence prediction performance, whereas its benefit for arousal estimation remains limited—a finding consistent with cross-linguistic trends in affective computing. These outcomes underscore the critical role of linguistic content in modeling emotional valence, highlighting modality-specific contributions to dimensional emotion recognition in naturalistic spoken language.

Abstract

Existing research on affect in speech has shown how acoustic surface characteristics and content-related linguistic aspects of speech both relate to perceived emotional arousal and valence. However, it is not clear what the relative contributions of these two factors are in the perceptual process. This is especially true for Finnish, for which most existing studies focus on either acoustic-phonetic or text analysis. This paper presents a study where we systematically explore the combinatory role of text- and audio-based features in modeling the human perception of valence and arousal using a newly released affective speech corpus for spontaneous Finnish. We show that the combination of text- and audio-based features improves valence regression results over the individual modalities, whereas for arousal regression the complementary effect is not substantial. The results support prior findings from other languages, providing new data and knowledge on spontaneous Finnish speech.

Problem

• affect
• spontaneous speech
• Finnish
• valence
• arousal

Innovation

• multimodal affect modeling
• spontaneous Finnish speech
• valence-arousal prediction
• text-audio fusion
• linguistic interpretability