Reader Proficiency Shapes Layer-wise Surprisal Profiles

2026-09-29

AI Summary

This study investigates how reader vocabulary proficiency modulates the layer-wise mapping between surprisal derived from large language models (LLMs) and eye-tracking measures during reading. Leveraging the MECO L2 corpus, the authors integrate eye-tracking data with twelve LLMs to introduce a novel metric termed “predictive depth,” employing leave-one-out cross-validation to analyze fixation duration disparities and internal representational distributions across high- and low-proficiency readers. The findings reveal that first-pass gaze durations for low-proficiency readers correspond to deeper predictive layers, while total gaze durations generally align with deeper representations. This research demonstrates that layer-wise surprisal effectively accounts for variance in reading behavior, offering new insights into the divergences between LLM processing mechanisms and human cognition.

Abstract

Reading behaviour varies not only with linguistic input, but also with reader proficiency. In this study, we investigate whether the layer-wise relationship between surprisal from large language models (LLMs) and human gaze behaviour differs across readers with different levels of proficiency and across gaze measures. Using eye-tracking data from the MECO L2 corpus, we compare readers with high and low vocabulary proficiency on first-pass gaze duration (FPGD) and total gaze duration (TGD). We quantify the distribution of the predictive power of surprisal across model layers using Predictive Depth. Across 12 tested LLMs, we find that readers with lower vocabulary proficiency tend to show deeper Predictive Depth for FPGD, while this difference is smaller for TGD. Also, TGD itself shows deeper Predictive Depth than FPGD in both proficiency groups. These patterns suggest that where predictive power is concentrated across LLM layers may be related to the timing and breadth of the reading processes captured by different gaze measures, and that this relationship can vary with reader proficiency. Our leave-one-out analysis further shows that the advantage of informative internal layers extends to unseen texts, although the practical improvements in prediction are limited. Overall, our results show that layer-wise LLM surprisal provides a useful perspective on variation in reading behaviour across both reader groups and gaze measures.

Problem

• reader proficiency
• surprisal
• large language models
• eye-tracking
• gaze behaviour

Innovation

• layer-wise surprisal
• predictive depth
• reader proficiency
• eye-tracking
• large language models