compute model perplexity

This study investigates the reliability of perplexity as a model selection metric, revealing its fundamental limitations in distinguishing predictive correctness. Through rigorous analysis grounded in the continuity theory of Transformers, the work establishes—for the first time—that even compact decoder-only models capable of accurate and confident predictions on certain sequences necessarily admit other sequences with low perplexity yet incorrect predictions. The analysis further demonstrates that perplexity reliably reflects performance improvement only when increases in model confidence are accompanied by concurrent gains in accuracy. By characterizing iso-perplexity curves and providing formal theoretical proofs, this research elucidates the inherent inconsistency between perplexity and predictive accuracy, offering new theoretical foundations for model evaluation and metric design.

To address the high computational overhead and inefficiency caused by redundant reasoning steps in Chain-of-Thought (CoT) inference, this paper proposes a layer-wise perplexity-change-based method for identifying critical reasoning steps—introducing, for the first time, dynamic perplexity change as an interpretable, quantitative metric for step importance. The method enables fine-grained step pruning and jointly optimizes few-shot example refinement and critical-step-driven supervised fine-tuning. Evaluated on multiple complex reasoning benchmarks, it compresses CoT inference while maintaining or improving accuracy, achieving average speedups of 37%–52%. Key contributions include: (1) a perplexity-change-driven criterion for critical step identification; (2) a dual-path optimization framework integrating example selection and step-aware fine-tuning; and (3) a lightweight, annotation-free inference compression mechanism. This approach significantly improves the accuracy-efficiency trade-off in CoT reasoning.

This work addresses the lack of rigorous theoretical interpretation for length-normalized importance ratios in the GSPO algorithm. We establish, for the first time, a strict equivalence between sequence-level importance weights and information-theoretic quantities—specifically, proving that such weights equal the product of perplexity ratios and exponential changes in cross-entropy. Building on this insight, we propose a novel gradient-weighting mechanism grounded in inverse perplexity ratios and geometric means in the log domain, offering a unified information-theoretic explanation for GSPO’s variance reduction and training stability. Our theoretical derivations are mathematically rigorous and empirically validated. Experiments demonstrate that this interpretation effectively accounts for GSPO’s superior performance and robustness in training mixture-of-experts models on mathematical reasoning tasks. The framework advances policy optimization by introducing an interpretable, analyzable paradigm rooted in information theory.

Current evaluations of large language models predominantly focus on task performance, offering limited insight into whether models rely on linguistically principled reasoning mechanisms and remaining susceptible to confirmation bias. This work proposes an interpretability framework based on token-level perplexity, which examines the distribution of perplexity across minimally contrasting sentence pairs that differ only at critical linguistic tokens. By doing so, it tests whether models condition their predictions on expected linguistic cues. This approach represents the first application of token-level perplexity for hypothesis-driven validation of linguistic mechanisms, circumventing the instability inherent in conventional feature attribution methods. Experiments across multiple open-source large language models reveal that while key tokens significantly influence model behavior, they alone cannot fully account for observed perplexity variations, indicating that models continue to rely on unintended heuristic strategies.

Perplexity is widely adopted for detecting large language model–generated code (LLMgCode), yet its practical effectiveness—particularly regarding accuracy, generalizability, and efficiency—remains inadequately characterized. Method: We conduct a systematic empirical evaluation of perplexity across a multilingual benchmark comprising over 24,000 code snippets, assessing detection accuracy, cross-language and cross-task generalization, and inference latency. Contribution/Results: We find that perplexity achieves the strongest generalization—outperforming alternatives across languages and programming tasks—and offers intrinsic interpretability. However, its average detection accuracy remains low (<65%), inference is 3–8× slower than feature-engineering–based methods, and performance degrades markedly on high-level languages (e.g., Python, JavaScript) versus lower-level ones (e.g., C, Java). Our analysis delineates the fundamental trade-offs among generalization, accuracy, and efficiency, establishing a three-dimensional evaluation framework that informs both theoretical understanding and practical selection of LLMgCode detection techniques.

该研究使用概率模型检测方法解决自回归神经序列模型在部署时的约束违反概率和输入满足领域需求比例的问题。

This study addresses the limitations of existing approaches that rely on token-level perplexity to assess code understandability, which often fail to capture holistic or segment-level cognitive processes. The authors systematically investigate the effectiveness of segment-level perplexity as a proxy for human-judged understandability, conducting empirical analyses across multiple large language models and tokenizers using several human-annotated datasets. They evaluate various aggregation strategies—such as mean, median, and peak—and find that simple aggregations exhibit unstable correlations with human judgments. This instability stems from three key factors: highly skewed perplexity distributions, insufficient consensus among human annotators, and sensitivity differences across models and tokenizers. To overcome these challenges, the paper advocates for integrating code-aware aggregation, consensus-aware evaluation, and model-sensitivity analysis, thereby charting a new direction toward more reliable understandability metrics.

This work addresses the limitations of conventional perplexity-based evaluation in knowledge distillation for autoregressive generation tasks, which often leads to suboptimal architectures and training strategies. To overcome this, the authors propose a generation-quality-centric distillation paradigm, introducing a Hybrid-KDA architecture and a multi-stage GenDistill distillation pipeline. Through systematic analysis across six design dimensions, they identify data selection, masking strategy, and attention layer freezing as critical factors influencing generative performance. Experimental results demonstrate that the proposed approach preserves 86–90% of the teacher model’s knowledge accuracy while reducing KV cache memory usage by 75% and achieving a 2–4× reduction in first-token latency at a context length of 128K tokens.

This work addresses the challenge of interpreting the contribution of input tokens to outputs in large language model generation by proposing the first model-agnostic probabilistic attribution method. The approach models text generation as a stochastic process and leverages Bayes’ rule to infer the conditional probability of a response given a prompt. Attribution scores are defined via the logarithm of probability ratios, while conditional entropy is introduced to quantify context sensitivity and generation uncertainty. Experiments across eight mainstream models and seven prompt categories demonstrate that the method effectively identifies anomalous generations, token-sensitive regions, and unstable behaviors, substantially enhancing users’ awareness and understanding of generative uncertainty.

Current non-autoregressive language models commonly rely on generation perplexity (gen-PPL) to evaluate text quality; however, this metric fails to adequately capture grammatical correctness and semantic coherence. This work proposes a zero-parameter naive sampler that achieves state-of-the-art gen-PPL on LM1B and OpenWebText yet produces clearly incoherent text, thereby systematically exposing the fundamental limitations of gen-PPL for the first time. To address this issue, we introduce direct evaluation methods based on distributional divergences—such as KL and Jensen–Shannon divergence—and scoring from pretrained autoregressive models. Our experiments demonstrate that these distribution-based metrics provide a more faithful and effective assessment of generation quality in unconditional text generation, establishing their necessity and superiority over conventional gen-PPL.