overfitting diagnostics

Conventional wisdom holds that overfitting degrades anomaly detection performance. This work challenges that assumption, proposing a “controllable overfitting” paradigm that repurposes overfitting as a tunable mechanism for detection enhancement. Method: (1) We introduce the Aberrance Retention Quotient (ARQ) to quantify overfitting severity and identify an optimal “sweet spot”; (2) we propose the Relative Anomaly Distribution Index (RADI) as a robust, pixel-level alternative to AUROC; (3) we theoretically justify and empirically validate the general efficacy of Gaussian-noise-based pseudo-anomaly synthesis. Contribution/Results: Our approach achieves state-of-the-art performance across standard benchmarks—including MVTec AD—demonstrating for the first time that overfitting can be precisely regulated to improve both anomaly sensitivity and discriminability. This work establishes a novel modeling paradigm for unsupervised anomaly detection.

High-complexity machine learning models lack reliable, theoretically grounded mechanisms for detecting overfitting. Method: We propose a statistical hypothesis test that operates solely on training data, dispensing with the need for an independent validation set or PAC-style uniform convergence assumptions. Our approach formalizes overfitting via empirical mean consistency and constructs a rigorous testing framework based on Hoeffding-type concentration inequalities. Contribution/Results: This is the first method to use empirical mean consistency as an overfitting criterion, enabling significance-based inference and implicit diagnosis sensitive to distributional shifts. We prove its validity under mild regularity conditions. Empirical evaluation demonstrates robust identification of overfitting transition points and latent distribution drift, substantially improving both the reliability and interpretability of model selection.

To address copyright infringement and transparency concerns arising from unauthorized use of third-party data in machine learning model training, this paper proposes the first general-purpose, task-agnostic data usage auditing framework for black-box models. Methodologically, it innovatively integrates arbitrary black-box membership inference techniques with a custom sequential probability ratio test (SPRT), enabling zero assumptions about downstream tasks, strict control over false positive rates (tunable within 0.5%–5%), and cross-model generalization. The framework features a model-agnostic interface, supporting heterogeneous architectures including image classifiers and multimodal large language models. Extensive experiments on ImageNet classifiers and multimodal foundation models demonstrate an average detection accuracy exceeding 92%, with false positive rates consistently meeting user-specified thresholds. This work significantly enhances the quantifiability and reliability of training data provenance auditing.

This work systematically exposes data leakage risks in machine learning and transfer learning arising from user operational errors—risks that severely distort model evaluation, affecting over 70% of “off-the-shelf” practices. To address this, we propose the first multidimensional leakage taxonomy, explicitly grounded in task type, learning paradigm (inductive vs. transductive), and transfer scenario; it identifies six high-frequency leakage patterns. Through workflow modeling, empirical case studies, and comparative analysis across learning paradigms, we characterize the mechanisms of illicit forward propagation of information during preprocessing, validation, and fine-tuning stages. Innovatively, we develop a leakage-prevention checklist and standardized procedural guidelines—delivering a practical, actionable methodology for robust and reliable model evaluation.

Existing misinformation detection models exhibit poor generalizability against human-crafted disinformation, suffer from non-independent evaluation protocols, and rely on datasets with systemic biases—leading to a severe disconnect between academic research and industrial deployment. Method: We conduct a cross-disciplinary methodology audit of 248 highly cited papers across security, NLP, and computational social science, introducing the first machine learning evaluation framework specifically designed for trust and safety applications. Our audit integrates bibliometric analysis, reproducibility assessment, three representative replication experiments, and systematic evaluation of dataset and code availability. Contribution/Results: We identify critical flaws in data curation practices and evaluation methodologies; empirical results demonstrate substantial performance degradation of fully automated detectors under real-world conditions. This work delivers a practical, actionable evaluation guideline and a concrete research roadmap for trustworthy AI governance.

This study addresses a critical yet overlooked issue in the evaluation of AI-driven keyphrase extraction models: severe performance overestimation due to data leakage between training and test sets caused by duplicated samples in benchmark datasets. For the first time, this work systematically quantifies and reveals the misleading impact of such leakage on model generalization assessments. Through large-scale duplicate detection, cross-set similarity analysis, and controlled experiments—augmented by code fingerprinting and semantic comparison to identify leaked instances—the authors propose a rigorous de-duplicated evaluation protocol. Re-evaluation under this protocol demonstrates that state-of-the-art models suffer substantial F1 score drops (exceeding 30% in some cases) once data leakage is eliminated, exposing significant risks in real-world deployment and establishing a more reliable, unbiased benchmark for future research in keyphrase extraction.

This study addresses the challenge of detecting information leakage solely from a model’s predictive outputs, without access to training code, external data, or domain knowledge. Framed within decision theory, the approach models leakage diagnosis as a functional of predictive risk and outcome distribution, linking proper scoring rules with decision curve analysis via threshold-weighted associations to enable detection without prior assumptions. The work introduces a novel tripartite classification of information leakage—miscalibration, generalized calibration, and determinism—and theoretically establishes that generalized calibration leakage is fundamentally unidentifiable, whereas near-deterministic subgroups can be efficiently detected. Empirical validation on UK Biobank demonstrates detection of temporal-window comorbidity leakage down to Δc*≈0.007 in under one second, while also revealing inherent structural limitations of purely output-driven leakage detection.

Risk valuation systems are susceptible to undetected errors caused by data failures, misconfigurations, or anomalies, potentially leading to significant operational losses. This work proposes EQAF, a hierarchical unsupervised ensemble framework for anomaly detection that uniquely integrates domain-specific deterministic rules with multiple complementary statistical outlier detection methods to enable real-time integrity monitoring of risk computation outputs. EQAF effectively identifies subtle anomalies—such as “frozen values”—that are often missed by conventional purely statistical approaches. Experimental evaluation on four real-world risk datasets demonstrates that EQAF achieves F1 scores between 61% and 79% and improves AUC-ROC by 4–6 percentage points over the best individual baseline method, substantiating its robustness and effectiveness.

Current evaluation practices for supervised learning models are often misleading due to an overreliance on single aggregate metrics, which neglect the alignment among data characteristics, task objectives, and real-world application contexts. This work reframes model evaluation as a context-dependent, decision-oriented process and systematically investigates—through controlled experiments—the impact of dataset properties, validation strategies, class imbalance, and asymmetric error costs on evaluation outcomes. Leveraging diverse benchmark datasets, multiple validation protocols, and multidimensional performance measures, the study uncovers common pitfalls such as the accuracy paradox, data leakage, and metric misuse. It proposes a structured evaluation framework explicitly aligned with operational goals, offering principled guidance for developing more robust, reliable, and trustworthy supervised learning systems.

Detecting overfitting—termed “anti-grokking”—during neural network grokking remains challenging when neither training nor test data are accessible. This work introduces random matrix theory to enable data-free overfitting detection by analyzing deviations of empirical spectral distributions of weight matrices from the Marchenko–Pastur law. The authors propose the concept of “correlation traps” as early indicators of overfitting, identified through such spectral anomalies. By combining weight randomization with Jensen–Shannon divergence measurements, they demonstrate that the number and magnitude of correlation traps strongly correlate with declining test accuracy. Similar patterns are observed across multiple foundational large models, suggesting that correlation traps serve as a universal signature of detrimental overfitting in the absence of data access.