Who Do LLMs Trust? Human Experts Matter More Than Other LLMs

2026-02-14

AI Summary

This study investigates whether large language models (LLMs) exhibit credibility sensitivity and conformity behaviors akin to humans when exposed to social information sources such as human experts, other LLMs, or laypersons. By presenting four instruction-tuned LLMs with source-attributed peer responses in binary decision tasks spanning reading comprehension, multi-step reasoning, and moral judgment, the research systematically manipulates group accuracy, group size, and direct conflicts between human and LLM sources to assess trust preferences. The findings reveal, for the first time, that LLMs display a strong preference for answers labeled as coming from “human experts,” even when those answers are incorrect, often revising their own responses accordingly. This indicates the presence of a cross-task social influence mechanism grounded in credibility sensitivity, with the expert label functioning as a powerful prior signal.

Technology Category

Humans and AI: Learning Human Values and PreferencesCognitive Modeling & Cognitive Systems: Social Cognition And InteractionMachine Learning: Large Multimodal Models (LMMs)

Application Category

Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAISemantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactionsUser Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large language models (LLMs) increasingly operate in environments where they encounter social information such as other agents'answers, tool outputs, or human recommendations. In humans, such inputs influence judgments in ways that depend on the source's credibility and the strength of consensus. This paper investigates whether LLMs exhibit analogous patterns of influence and whether they privilege feedback from humans over feedback from other LLMs. Across three binary decision-making tasks, reading comprehension, multi-step reasoning, and moral judgment, we present four instruction-tuned LLMs with prior responses attributed either to friends, to human experts, or to other LLMs. We manipulate whether the group is correct and vary the group size. In a second experiment, we introduce direct disagreement between a single human and a single LLM. Across tasks, models conform significantly more to responses labeled as coming from human experts, including when that signal is incorrect, and revise their answers toward experts more readily than toward other LLMs. These results reveal that expert framing acts as a strong prior for contemporary LLMs, suggesting a form of credibility-sensitive social influence that generalizes across decision domains.

Problem

• large language models
• social influence
• credibility
• human experts
• source attribution

Innovation

• large language models
• social influence
• human experts
• credibility sensitivity
• source attribution