๐ค AI Summary
This study addresses the poorly understood sources of inter-annotator disagreement in narrative analysis. We propose decomposing the annotation task into two hierarchical subtasks: (1) judgment of narrative event existence and (2) identification of narrative elements. This constitutes the first structured decomposition and quantitative attribution of annotation disagreement in narrative analysis. Using a sentence-level dataset annotated for Complication, Resolution, and Success, we apply conceptual decomposition, variance decomposition, and qualitative case analysis. Results reveal significantly divergent contributions of the two subtasks to overall disagreement. Further analysis identifies three actionable root causes: textual ambiguity, ill-defined schema criteria, and annotator subjectivity. The proposed framework provides a principled, operational diagnostic tool and concrete pathways for improving annotation consistency in narrative analysis.
๐ Abstract
In this work, we explore sources of inter-annotator disagreement in narrative analysis, in light of the question of whether or not a narrative plot exists in the text. For this purpose, we present a method for a conceptual decomposition of an existing annotation into two separate levels: (1) whether or not a narrative plot exists in the text, and (2) which plot elements exist in the text. We apply this method to an existing dataset of sentences annotated with three different narrative plot elements: Complication , Resolution and Success . We then employ statistical analysis in order to quantify how much of the inter-annotator disagreement can be explained by each of the two levels. We further perform a qualitative analysis of disagreement cases in each level, observing several sources of disagreement, such as text ambiguity, scheme de๏ฌnition and personal differences between the annotators. The insights gathered on the dataset may serve to reduce inter-annotator disagreement in future annotation endeavors. We conclude with a broader discussion on the potential implications of our approach in studying and evaluating inter-annotator disagreement in other settings.