collaborative filtering

This work systematically evaluates the efficacy and limitations of large language models (LLMs) as annotators for subjective tasks. We survey 12 prior studies and empirically compare opinion distribution alignment between GPT-series models and human annotators across four subjective datasets—introducing, for the first time, a novel evaluation paradigm centered on *perspective diversity alignment*. Results reveal substantial distributional biases in LLMs, including underrepresentation of minority viewpoints, prompt sensitivity, English-language bias, and embedded societal prejudices; most existing annotation methods overlook such distributional discrepancies, and only a few strategies effectively capture opinion diversity. Our findings expose critical reliability risks in deploying LLMs for subjective annotation and establish a reproducible statistical framework—comprising quantitative metrics and methodological guidelines—for assessing annotation quality in subjective NLP tasks.

This work addresses the challenge of inconsistent preference annotation quality from users during LLM comparison interactions, which severely hampers alignment performance. We propose a dynamic quality-aware preference learning framework grounded in user behavioral modeling. Our method introduces (1) an asymmetric dual-model response generation mechanism to faithfully emulate real-world comparative decision-making; (2) a latent variable to represent annotation quality, jointly estimated with the preference model via the Expectation-Maximization (EM) algorithm; and (3) fully automated, label-free data filtering that adapts to intrinsic annotation reliability. Experiments demonstrate substantial improvements in preference data fidelity and alignment robustness, outperforming state-of-the-art methods across multiple alignment benchmarks—particularly under scenarios with noisy or low-quality user feedback. The approach establishes a novel paradigm for robust preference-based alignment without requiring manual quality labels.

This study systematically evaluates the zero-shot and few-shot binary classification performance of GPT-3.5, GPT-4, LLaMA-3, Mistral-7B, and Claude-2 on human rights violation content in Russian and Ukrainian social media texts, using dual human annotation (κ = 0.91) as the gold standard. Method: We introduce a novel cross-lingual (English-to-Russian) prompting framework and a fine-grained error analysis pipeline to rigorously assess model reliability on sensitive, subjective, and low-resource human rights annotation tasks. Contribution/Results: GPT-4 achieves the highest performance (F1 = 0.82), yet all models fall significantly short of human inter-annotator agreement in ambiguous contexts. Russian-language prompts improve non-English models’ accuracy by an average of 11%, underscoring the critical role of linguistic adaptation for robustness. This work establishes the first reproducible benchmark and practical guidelines for AI-assisted annotation in high-stakes, multilingual human rights monitoring.

The effectiveness and applicability boundaries of large language models (LLMs) in recommendation tasks remain poorly understood. Method: We propose a unified prompt engineering framework that reformulates recommendation as natural language inference, enabling zero-shot and cross-scenario generalization. We conduct controlled, multi-dimensional experiments on MovieLens and Amazon datasets to isolate the independent effects of LLM architecture, parameter scale, context length, and four prompt components—task description, user interest modeling, candidate item construction, and prompting strategy. Contribution/Results: Our study establishes a reproducible evaluation paradigm and demonstrates that LLMs possess intrinsic zero-shot recommendation capability. However, prompt quality and fidelity of user interest modeling constitute critical bottlenecks. Structurally optimizing prompts yields substantial performance gains. This work provides both an empirically grounded benchmark and a practical, deployable technical pathway for LLM-based recommender systems.

To address three key challenges in directly applying large language models (LLMs) to recommendation systems—scarce and non-textual recommendation data, incompatibility between interaction signals and linguistic modalities, and the risk of catastrophic forgetting of LLMs’ general capabilities—this paper proposes the Item-Language Model (ILM). ILM introduces a novel “language-aligned item encoder” that maps user interaction signals into semantic representations comprehensible to LLMs, thereby decoupling interaction modeling from language understanding. It adopts a parameter-efficient optimization paradigm combining frozen LLM backbones (e.g., LLaMA or Qwen), contrastive learning, and instruction tuning—without updating any LLM parameters. On multi-turn conversational recommendation tasks, ILM significantly outperforms both end-to-end fine-tuned LLMs and conventional recommendation models. Empirical results validate the dual necessity of language-aligned representation learning and effective injection of recommendation-specific knowledge.

Large language models exhibit inconsistent performance in social science text classification. This study systematically investigates the impact of three key prompt engineering components—label descriptions, instructional guidance, and few-shot examples—on classification accuracy. Through controlled experiments across multiple mainstream large language models, the authors find that moderately enriching prompt context significantly improves performance, whereas excessive augmentation can degrade it. Moreover, the optimal prompt configuration is highly dependent on the specific model, task, and data batch. The findings underscore the necessity of task- and model-specific validation and reveal a “less-is-more” principle in prompt design, offering practical guidance for achieving efficient and stable text classification in social science applications.

This study addresses the lack of systematic evaluation regarding the implementation choices of large language models (LLMs) in political text annotation and the unclear mechanisms underlying their effects. The authors propose a validation-first evaluation framework, conducting controlled multi-model experiments and ablation analyses under unified hardware, quantization settings, and prompt templates to systematically assess six open-source LLMs across four political science annotation tasks. Findings reveal no universally optimal model or prompting strategy, an inconsistent relationship between model scale and performance or cost-efficiency, and unreliable or even detrimental effects of certain widely adopted prompting techniques. Moreover, substantial efficiency differences emerge across model families. Challenging prevailing empirical “best practices” in LLM-based annotation, this work also releases an open-source toolchain to support reproducible research.

This study addresses the limitations of large language models (LLMs) in annotating complex social science constructs—such as climate mitigation pessimism—where autonomous labeling often yields suboptimal quality. To overcome this, the authors propose AnnotateThis, a human-centered interactive annotation system that introduces an innovative “LLM grounding” paradigm, deeply integrating expert knowledge into the LLM annotation pipeline. The system enables iterative co-evolution of conceptual definitions and model refinement through human–AI collaboration, interactive visualizations, and dynamic prompt optimization, functioning effectively both with and without ground-truth labels. Empirical evaluation demonstrates that, in labeled settings, AnnotateThis achieves a 0.15 improvement in F-Measure and a 0.23 gain in accuracy, significantly outperforming existing fully automated approaches.

Accurately extracting UK Research Excellence Framework (REF) ratings (1*–4*) from noisy, unstructured text containing missing or invalid values presents a significant challenge, requiring large language models (LLMs) to output only normalized integers (1–4) or a designated missing-value indicator (−1). To address this, this work introduces the first standardized prompt engineering benchmark for complex numerical extraction tasks, accompanied by a publicly available dataset of 1,446 short texts with gold-standard annotations. By integrating semantic understanding with explicit rule-based constraints, an initial prompting strategy achieves 72.6% accuracy. The study clarifies the definition of valid ratings and formalizes a mechanism for handling missing data, thereby advancing research into LLMs’ numerical reasoning and instruction-following capabilities, with the aim of fostering community-driven improvements in structured information extraction from noisy textual sources.

This study addresses the instability of large language models (LLMs) across different prompting styles, a phenomenon whose underlying mechanisms remain poorly understood. By systematically comparing instruction-based and exemplar-based prompts through attention head analysis, representational probing, and behavioral experiments, the work identifies for the first time a shared “lexical task attention head” that operates consistently across prompting paradigms. The activation strength of this attention head strongly correlates with model performance and reliably triggers task-relevant answer generation. Furthermore, the research reveals that competition among internal task representations is a key driver of prompt sensitivity. These findings offer an interpretable framework for understanding LLM internal dynamics and suggest novel directions for enhancing prompt robustness.