perplexity measurement

This work investigates whether pretraining metrics—such as perplexity—reliably predict downstream performance of large language models (LLMs) after fine-tuning, aiming to improve model selection efficiency under fixed computational budgets. The authors formulate checkpoint selection as a pairwise classification task and systematically evaluate 50 distinct 1B-parameter LLM variants across diverse downstream tasks, revealing that perplexity is frequently misleading. They propose novel unsupervised and supervised proxy metrics, which reduce prediction error rates by over 50% in multi-task supervised fine-tuning (SFT) evaluation. This study is the first to empirically demonstrate a non-monotonic relationship between pretraining metrics and downstream performance. The proposed proxies exhibit strong cross-task generalization and practical utility, offering a trustworthy, task-aware evaluation paradigm for optimizing pretraining strategies toward downstream objectives.

This paper addresses the problem that conventional calibration evaluation of deep learning models is vulnerable to spurious recalibration—i.e., trivial post-hoc adjustments that improve calibration metrics without enhancing generalization. To tackle this, we propose a novel joint evaluation paradigm integrating calibration and generalization. First, we derive a Bregman-divergence-based decomposition of calibration error, establishing the first theoretical connection between calibration metrics and generalization objectives (e.g., negative log-likelihood). Second, we design a new reliability diagram that jointly visualizes calibration bias and estimated generalization error. Third, we characterize multiple “pseudo-optimal” calibration phenomena and provide theoretically grounded, detectable criteria for identifying trivial recalibration. Experiments on standard benchmarks demonstrate that our approach significantly improves model diagnostic capability, yielding a more reliable and interpretable evaluation framework for calibration research.

Current time-series forecasting evaluation suffers from a fundamental flaw: conventional metrics conflate model performance with intrinsic data predictability, yielding biased assessments. To address this, we propose a spectral-coherence-based predictability-aligned evaluation framework. It introduces the Spectral Coherence Predictability (SCP) score and the Linear Utilization Ratio (LUR) diagnostic tool—revealing, for the first time, the phenomenon of “predictability drift.” Our method integrates fast Fourier transform (O(N log N)), frequency-resolved analysis, and linear system modeling to quantify task-inherent difficulty and assess how efficiently models exploit available information. Experiments demonstrate complementary strengths: complex models excel on low-SCP subtasks, while linear models dominate high-SCP regimes. This framework shifts evaluation from static ranking to predictability-aware dynamic diagnosis, enabling principled model selection and targeted improvement. (149 words)

This study addresses a critical confound in the calibration evaluation of large language models (LLMs), where discriminability (sensitivity) is often conflated with response bias (tendency). For the first time, signal detection theory (SDT) is fully integrated into LLM assessment through an unequal-variance model, z-ROC analysis, and a pre-registered experimental design. Analyzing 168,000 factual judgment trials across three leading models, the research demonstrates that temperature not only modulates confidence but concurrently enhances sensitivity (as measured by AUC) and shifts the response criterion. Critically, the evidence distributions exhibit pronounced unequal variance (z-ROC slopes ranging from 0.52 to 0.84), revealing that conventional calibration metrics fail to disentangle distinct combinations of sensitivity and bias. This work establishes a more nuanced psychometric framework for evaluating LLM performance.

This work identifies a significant likelihood bias in large language models (LLMs) when used as automatic evaluation metrics: LLMs systematically assign higher scores to surface-level features—such as word order and syntactic structure—favoring high-probability sentences while under-scoring semantically valid yet low-probability ones. To rigorously validate this phenomenon and mitigate the bias, we propose, for the first time, a “bias-to-correct-bias” few-shot in-context learning strategy: prompts are constructed from highly biased examples to calibrate LLM evaluation behavior without fine-tuning. Experiments across data-to-text generation and grammatical error correction tasks, involving multiple state-of-the-art LLMs, confirm the ubiquity of this bias. After calibration, average correlation between LLM scores and human judgments improves by 12.7%; moreover, evaluation robustness and fairness are substantially enhanced.

Current machine learning evaluation practices predominantly rely on surface-level performance metrics, often neglecting the internal mechanisms of models. This work proposes trustworthy interpretability as a central evaluation paradigm and, for the first time, systematically demonstrates that it satisfies core criteria from the philosophy of science—namely falsifiability, reproducibility, and predictive power. By constructing an evaluation framework that integrates causal analysis with mechanistic probing, the study delineates three functional pathways through which interpretability enables the identification of behavioral origins, detection of latent flaws, and prediction of potential failure modes. This approach advances model assessment beyond performance-oriented benchmarks toward a deeper understanding of underlying mechanisms.

This study addresses the evaluation and selection of source-level likelihood ratio (LR) systems for forensic evidence-to-reference comparison tasks by proposing an integrated analytical framework that balances performance and practical feasibility. The authors employ strictly proper scoring rules to quantify how effectively each system updates Bayesian prior odds and present the first systematic comparison among specific-source feature-based, common-source anchored, and unanchored score-based LR approaches. Their findings reveal that specific-source feature-based LRs achieve the highest performance but incur substantial experimental costs, whereas common-source feature-based methods offer strong discriminative power with significantly reduced implementation complexity. All LR systems substantially outperform a baseline relying solely on prior odds. This work thus provides both theoretical grounding and practical guidance for selecting LR systems in forensic practice.

This study addresses a critical gap in explainable AI evaluation by proposing a novel metric that quantifies the consistency of model explanations across inputs sharing the same label or subjected to label-preserving perturbations. Specifically, the method measures the cosine similarity of SHAP values among samples with identical labels, thereby capturing explanation stability under semantic-preserving variations. By integrating both class-wise consistency and robustness to minor input perturbations into a unified evaluation framework, this work pioneers a systematic approach to assessing cross-sample explanation coherence. Experiments on SST-2 and IMDB datasets using BERT, RoBERTa, and DistilBERT demonstrate that the proposed metric effectively identifies inconsistent explanatory behaviors—such as undue reliance on specific features—and offers superior diagnostic capability compared to conventional fidelity-based metrics, ultimately contributing to the development of more trustworthy AI systems.

Current evaluation practices for supervised learning models are often misleading due to an overreliance on single aggregate metrics, which neglect the alignment among data characteristics, task objectives, and real-world application contexts. This work reframes model evaluation as a context-dependent, decision-oriented process and systematically investigates—through controlled experiments—the impact of dataset properties, validation strategies, class imbalance, and asymmetric error costs on evaluation outcomes. Leveraging diverse benchmark datasets, multiple validation protocols, and multidimensional performance measures, the study uncovers common pitfalls such as the accuracy paradox, data leakage, and metric misuse. It proposes a structured evaluation framework explicitly aligned with operational goals, offering principled guidance for developing more robust, reliable, and trustworthy supervised learning systems.

This study addresses the long-standing challenge of calibrating local false discovery rates (lfdr) in multiple hypothesis testing when true labels are unavailable. The authors introduce, for the first time, a pseudo-labeling mechanism based on spacings between ordered p-values, reframing lfdr calibration as an unsupervised regression problem and thereby enabling the application of classical probability calibration tools. By integrating an empirical Bayes framework with posterior calibration techniques, the method reveals that the widely used q-value approach suffers from substantial miscalibration. Extensive empirical analyses in psychology and neuroscience demonstrate that the proposed approach significantly enhances the reliability and interpretability of lfdr estimates, underscoring its necessity and superiority over existing practices.