Can Perplexity Serve as a Cognitive Signal for Code Understandability?

2026-08-01

AI Summary

This study addresses the limitations of existing approaches that rely on token-level perplexity to assess code understandability, which often fail to capture holistic or segment-level cognitive processes. The authors systematically investigate the effectiveness of segment-level perplexity as a proxy for human-judged understandability, conducting empirical analyses across multiple large language models and tokenizers using several human-annotated datasets. They evaluate various aggregation strategies—such as mean, median, and peak—and find that simple aggregations exhibit unstable correlations with human judgments. This instability stems from three key factors: highly skewed perplexity distributions, insufficient consensus among human annotators, and sensitivity differences across models and tokenizers. To overcome these challenges, the paper advocates for integrating code-aware aggregation, consensus-aware evaluation, and model-sensitivity analysis, thereby charting a new direction toward more reliable understandability metrics.

Abstract

Recent work suggests that token-level perplexity from large language models can align with localized human confusion during code comprehension. This raises a natural question: can perplexity also serve as a snippet-level signal for code understandability? We conduct an empirical study of this question across multiple human-grounded datasets, including method-level understandability judgments, output-prediction tasks, and accepted understandability-improvement patches. Despite prior token-level evidence, we find that simple snippet-level aggregations of token perplexity, such as average, median, or peak perplexity, do not reliably correlate with human understandability. We then investigate why this happens. First, token perplexity is highly skewed and heavy-tailed across code structures; extreme spikes arise not only from semantically meaningful constructs, but also from identifiers, literals, types, separators, and tokenization artifacts. Second, human understandability labels often lack consensus, making whole-snippet difficulty a noisy target. Third, perplexity distributions and their alignment with human difficulty vary substantially across models and tokenizers. These findings explain why prior token-level perplexity--confusion alignment does not directly transfer to snippet-level understandability. Overall, our study positions perplexity as a promising but delicate cognitive signal: useful for localized code confusion, but requiring code-aware aggregation, consensus-aware evaluation, and model-sensitivity analysis before it can support reliable code-understandability measurement.

Problem

• perplexity
• code understandability
• cognitive signal
• human confusion
• code comprehension

Innovation

• perplexity
• code understandability
• cognitive signal
• tokenization artifacts
• human-grounded evaluation