surprisal is Not a Theory

2026-07-22

AI Summary

This study critically examines the common practice of directly employing surprisal values from large language models (LLMs) to test the Surprisal Theory in psycholinguistics, highlighting that such usage overlooks the dependence of surprisal on model-specific representational and algorithmic choices. The work demonstrates that LLM-derived surprisal is not theoretically neutral but is significantly shaped by architectural differences—such as Transformer versus RNN—and algorithmic factors, including sampling strategies and context handling. Through three systematic comparative experiments, the authors reveal substantial variations in surprisal estimates across modeling decisions, thereby challenging its validity as a universal cognitive metric. The findings underscore the necessity for computational psycholinguistics to explicitly articulate representational and algorithmic commitments, advocating for more rigorous paradigms in theory validation.

Abstract

Surprisal Theory is often characterized as a computational-level explanation per (Marr, 1982). We argue in this work that, even though a computational level narrative has been used to support "representation-agnostic research" within computational psycholinguistics, the movement toward black box systems embodied by large language models (LLMs) does not exempt modelers using the surprisal metric from the representational decisions required by computational-level characterizations. In fact, we argue that the uncritical use of LLM-surprisal obfuscates the representational and algorithmic-level commitments of different models. In three analyses, we show that the choice of algorithm and model architecture play significant roles in the computation of language model probabilities. We advise that researchers who wish to test Surprisal Theory re-evaluate the practice of treating large language model probabilities as interchangeable

Problem

• surprisal
• large language models
• computational-level theory
• representational commitments
• psycholinguistics

Innovation

• surprisal
• large language models
• computational-level theory
• model architecture
• algorithmic commitments