SubjECTive-QA: Measuring Subjectivity in Earnings Call Transcripts' QA Through Six-Dimensional Feature Analysis

2024-10-28

arXiv.org

AI Summary

This study addresses the lack of quantitative assessment for subjectivity in earnings call Q&A transcripts. We introduce the first six-dimensional subjectivity annotation dataset (Assertive, Cautious, Optimistic, Specific, Clear, Relevant), comprising 49,446 long-text QA pairs. A systematic, multi-dimensional subjectivity definition framework and a human-in-the-loop annotation protocol are proposed to fill a critical gap in financial-domain subjectivity evaluation resources. We benchmark RoBERTa-base and Llama-3-70b-Chat on multi-label classification: performance is comparable on low-subjectivity dimensions (e.g., Relevant; F1 difference = 2.17%), but diverges substantially on high-subjectivity ones (up to 10.01% F1 gap). Cross-domain evaluation on White House briefings and Gaggles yields an average weighted F1 of 65.97%. The dataset is publicly released under the CC BY 4.0 license.

Technology Category

• Natural Language Processing: Safety and Robustness
• Application Domains: Humanities & Computational Social Science
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Fact-checking is extensively studied in the context of misinformation and disinformation, addressing objective inaccuracies. However, a softer form of misinformation involves responses that are factually correct but lack certain features such as clarity and relevance. This challenge is prevalent in formal Question-Answer (QA) settings such as press conferences in finance, politics, sports, and other domains, where subjective answers can obscure transparency. Despite this, there is a lack of manually annotated datasets for subjective features across multiple dimensions. To address this gap, we introduce SubjECTive-QA, a human annotated dataset on Earnings Call Transcripts' (ECTs) QA sessions as the answers given by company representatives are often open to subjective interpretations and scrutiny. The dataset includes 49,446 annotations for long-form QA pairs across six features: Assertive, Cautious, Optimistic, Specific, Clear, and Relevant. These features are carefully selected to encompass the key attributes that reflect the tone of the answers provided during QA sessions across different domain. Our findings are that the best-performing Pre-trained Language Model (PLM), RoBERTa-base, has similar weighted F1 scores to Llama-3-70b-Chat on features with lower subjectivity, such as Relevant and Clear, with a mean difference of 2.17% in their weighted F1 scores. The models perform significantly better on features with higher subjectivity, such as Specific and Assertive, with a mean difference of 10.01% in their weighted F1 scores. Furthermore, testing SubjECTive-QA's generalizability using QAs from White House Press Briefings and Gaggles yields an average weighted F1 score of 65.97% using our best models for each feature, demonstrating broader applicability beyond the financial domain. SubjECTive-QA is publicly available under the CC BY 4.0 license

Problem

• Subjective Bias
• Corporate Earnings Calls
• Answer Quality

Innovation

• SubjECTive-QA Dataset
• RoBERTa-base Model
• Subjectivity and Complexity in QA