🤖 AI Summary
This study reveals that large language models struggle to effectively assess the veracity of statistical evidence when integrating multi-source information, exhibiting a tendency to rely on superficial stylistic cues in methodological text rather than numerical plausibility when judging source credibility. The work identifies a previously undocumented “cognitive alignment” bias—where models prefer sources with strong analytical register over those with content consistency. Employing interpretable techniques including causal tracing, linear probing (AUC: 0.83–0.92), and component-level attribution, the authors replicate this blind spot across five mainstream models through cross-model and cross-domain experiments. Further analysis localizes the issue to a methodology-register gating mechanism and demonstrates that neither prompt engineering nor post-training interventions adequately mitigate the bias, instead raising concerns about model generalization.
📝 Abstract
Language models increasingly act as epistemic proxies, synthesizing evidence from multiple sources to inform decisions. Whether they evaluate the quality of that evidence, or merely aggregate it based on surface presentation, remains poorly understood. We show that models possess the capability to detect fabricated statistics (correct identification rates of 0.76-1.00 for methodology in isolation) but do not recruit this capability during multi-source synthesis, producing similar numeric estimates whether the statistics are fabricated or valid. Specifically, source influence is governed by a methodology-register gate that responds to the distributional register of analytical text but not to numeric validity: for example, statistically impossible confidence intervals receive the same weight as valid ones. The behavioral dissociation replicates across five models from three families (Claude, Qwen, OLMo) and three professional domains. Mechanistic analyses, including causal tracing, linear probes, and component-level attribution, converge on the same account: the model encodes and causally uses a methodology-register representation that transfers across domains (probe AUC 0.83-0.92), while numeric-validity signals, decodable in isolation, are suppressed to chance during multi-source synthesis. Prompting-based mitigations, even an oracle checklist naming the exact statistical checks, produce blanket skepticism rather than selective discernment, and the post-training pipelines we examine reinforce the stylistic shortcut without building numeric verification. Unlike sycophancy, which tracks user preference, this failure tracks whether a source presents as analytically credible, not whether its claims are internally consistent. We term this epistemic alignment: like preference and safety alignment, the question is not capability but deployment.