🤖 AI Summary
This study addresses a critical confound in the calibration evaluation of large language models (LLMs), where discriminability (sensitivity) is often conflated with response bias (tendency). For the first time, signal detection theory (SDT) is fully integrated into LLM assessment through an unequal-variance model, z-ROC analysis, and a pre-registered experimental design. Analyzing 168,000 factual judgment trials across three leading models, the research demonstrates that temperature not only modulates confidence but concurrently enhances sensitivity (as measured by AUC) and shifts the response criterion. Critically, the evidence distributions exhibit pronounced unequal variance (z-ROC slopes ranging from 0.52 to 0.84), revealing that conventional calibration metrics fail to disentangle distinct combinations of sensitivity and bias. This work establishes a more nuanced psychometric framework for evaluating LLM performance.
📝 Abstract
Large language models (LLMs) are evaluated for calibration using metrics such as Expected Calibration Error that conflate two distinct components: the model's ability to discriminate correct from incorrect answers (sensitivity) and its tendency toward confident or cautious responding (bias). Signal Detection Theory (SDT) decomposes these components. While SDT-derived metrics such as AUROC are increasingly used, the full parametric framework - unequal-variance model fitting, criterion estimation, z-ROC analysis - has not been applied to LLMs as signal detectors. In this pre-registered study, we treat three LLMs as observers performing factual discrimination across 168,000 trials and test whether temperature functions as a criterion shift analogous to payoff manipulations in human psychophysics. Critically, this analogy may break down because temperature changes the generated answer itself, not only the confidence assigned to it. Our results confirm the breakdown with temperature simultaneously increasing sensitivity (AUC) and shifting criterion. All models exhibited unequal-variance evidence distributions (z-ROC slopes 0.52-0.84), with instruct models showing more extreme asymmetry (0.52-0.63) than the base model (0.77-0.87) or human recognition memory (~0.80). The SDT decomposition revealed that models occupying distinct positions in sensitivity-bias space could not be distinguished by calibration metrics alone, demonstrating that the full parametric framework provides diagnostic information unavailable from existing metrics.