Annotation of Positive vs Negative User Interactions for Social Sign Prediction

2026-06-04

AI Summary

Existing methods for inferring social relationships often conflate content sentiment with relational dynamics, introducing systematic biases. This work proposes a relation-oriented zero-shot paradigm that leverages large language models—such as Qwen2.5-7B, Gemma2-9B, GPT-4o, and GPT-4o-mini—to directly recognize relational signals (e.g., praise and aggression) in interactions, bypassing conventional sentiment analysis. Through three progressively complex prompt designs, the approach achieves effective prediction of relational labels on two annotated datasets without any task-specific training data. Aggression detection proves robust across models and prompts, whereas praise identification exhibits greater sensitivity to model choice and prompting strategy. The study establishes a practical baseline and offers a novel direction for zero-shot annotation of social relationships.

Abstract

Inferring the sign of social relationships from online interactions is a fundamental challenge in social network analysis. Existing approaches typically rely on sentiment analysis to label individual interactions as positive or negative, then aggregate these labels to assign a sign to the relationship. However, sentiment analysis captures the valence of the content being discussed rather than the nature of the relational exchange itself, a conflation that can lead to systematic misclassification. In this paper, we propose a methodology that addresses this limitation by leveraging Large Language Models (LLMs) in a zero-shot setting to identify interaction-level relational signals (specifically, personal praise and personal attacks directed at the interlocutor) as more direct indicators of positive and negative social ties. We evaluate four models spanning open-weight and proprietary architectures (Qwen2.5:7b, Gemma2:9b, GPT-4o, GPT-5.4-mini) across three prompt designs of increasing complexity, on two human-annotated datasets of approximately 298 and 340 texts respectively. Results show that zero-shot LLMs achieve good classification performance on both tasks without any task-specific training data, establishing a practical baseline for relational annotation. Performance differs across tasks: attack detection is robust to prompt design and model choice, while praise detection is more sensitive to both, reflecting the greater subjectivity of positive relational gestures. These findings lay the groundwork for integrating LLM-based relational annotation into sign prediction pipelines.

Problem

• social sign prediction
• user interaction annotation
• relational signals
• sentiment analysis
• positive and negative ties

Innovation

• zero-shot LLMs
• relational signals
• social sign prediction
• interaction-level annotation
• personal praise and attacks