characterize dataset shift

In AI deployment for medical imaging, data distribution shifts frequently cause abrupt performance degradation and increased misdiagnosis risk. Existing methods can only detect the presence of shift but fail to identify its specific type—e.g., covariate shift, prior (concept) shift, or compound shift—hindering root-cause analysis and targeted mitigation. This paper proposes the first unsupervised framework for data shift type identification. It introduces a novel joint shift detection mechanism that synergistically leverages self-supervised encoder representations and task-model outputs. By integrating feature distribution comparison, unsupervised clustering, and multimodal image modeling, the method achieves high-accuracy shift-type discrimination across three major imaging modalities—chest X-ray, mammography, and fundus photography—and five realistic shift scenarios. Evaluated on four large public medical imaging datasets, it significantly enhances the robustness and interpretability of clinical AI systems.

This study investigates the impact of expanding the historical training window on predictive performance and algorithmic fairness under temporal distribution shift—including covariate and concept drift. Using simulation experiments and empirical student retention prediction across multi-institution, multi-year educational datasets, we find that the common assumption “more data is better” fails under concept-drift-dominant regimes: extending the training window degrades overall accuracy and exacerbates fairness disparities across sociodemographic groups—particularly when marginalized subpopulations experience heterogeneous concept drift, leading to nonlinear bias amplification. Our key contributions are: (i) identifying concept drift—not covariate drift—as the primary driver of performance degradation; and (ii) the first systematic characterization of its nonlinear, fairness-amplifying mechanism. These findings provide both theoretical grounding and practical guidance for model update strategies in dynamic, real-world deployment environments.

This work addresses the critical challenge of dataset shift—distributional discrepancies arising from temporal or source variations—that undermines the reliability and safety of medical AI systems. To bridge the gap in existing analytical tools, which often lack usability and comprehensiveness, we introduce dashi, an open-source Python library that unifies information geometry with nonparametric statistical manifold methods within a single framework. This framework enables both unsupervised shift characterization and supervised performance degradation assessment. By leveraging metrics such as information-geometric temporal graphs, global probability divergence, and source-wise probability outlier scores, complemented by interactive visualizations, dashi quantifies and interprets shifts across time or data sources. Validation across three real-world and simulated clinical scenarios—gestational diabetes, COVID-19, and emergency dispatch—demonstrates that dashi substantially enhances the trustworthiness and robustness of AI systems throughout their lifecycle.

Existing methods struggle to align and interpret distribution shifts across heterogeneous, domain-consistent datasets—such as tabular, textual, visual, and time-series data—especially when scale and modality disparities are pronounced, resulting in poor interpretability. This paper introduces the first human-centric, cross-modal distribution discrepancy explanation framework, implemented as an interpretable dataset comparison toolbox. It integrates statistical hypothesis testing, feature importance decomposition, class activation mapping (CAM), contrastive representation learning, and interpretable generative modeling to enable fine-grained, semantically readable attribution and visualization of distributional shifts. Evaluated across diverse real-world scenarios, the framework significantly improves users’ efficiency in understanding shift causes and enhances the accuracy of intervention decisions—thereby overcoming the limitations of conventional black-box shift detection approaches.

To address the challenge of unsupervised model performance estimation under covariate shift—where ground-truth labels are unavailable or delayed post-deployment—this paper proposes the Probability-Adaptive Performance Estimation (PAPE) framework. PAPE requires neither access to true labels nor knowledge of the original model’s architecture or feature representations; it operates solely on the model’s probabilistic outputs and confidence scores. By jointly leveraging density ratio estimation and performance generalization bound theory, PAPE models prediction distributions and applies adaptive reweighting to yield unbiased estimates of arbitrary classification metrics—without assuming a specific shift form or resorting to feature learning or generative modeling. Extensive evaluation across 900+ real-world census dataset–model combinations demonstrates that PAPE reduces mean absolute error by 37% compared to state-of-the-art proxy metrics and drift detection methods, significantly enhancing the reliability and generality of model monitoring in production environments.

This work addresses the challenge of predicting performance changes when a source-domain model is replaced by a new one. To this end, the authors propose TRACE, a novel framework that, for the first time, decomposes the risk difference between two models under covariate shift into four interpretable components: two generalization gaps, a model change penalty, and a covariate shift penalty. The framework establishes a computable upper bound to diagnose the causes of performance degradation. TRACE estimates model sensitivity via high-quantile input gradients, quantifies data distribution shift using either optimal transport (OT) or maximum mean discrepancy (MMD), and measures model change through output distances on target samples. Experiments demonstrate that TRACE’s diagnostic scores exhibit strong monotonic correlation with actual performance degradation and achieve superior performance in deployment gating, as measured by AUROC and AUPRC, thereby enabling label-efficient and safe model replacement.

When predictive models are deployed in new environments, their performance often degrades due to covariate shift and selective labeling, which jointly obscure accurate assessment of the true target risk. This work proposes an unbiased risk estimation method that integrates double machine learning with influence functions to simultaneously address both sources of bias for the first time. The approach is model-agnostic and compatible with general loss functions, constructing a corrected target risk estimator via nonparametric and plug-in techniques. Experiments on eICU electronic health record data demonstrate that the proposed method significantly outperforms baselines that handle only one type of bias or naively combine existing approaches, yielding more accurate tracking of the true target risk.

This work addresses the performance degradation of pathological vision-language models (VLMs) under distribution shifts in clinical deployment, a challenge exacerbated by the absence of effective label-free degradation detection mechanisms. To this end, the authors propose a unified monitoring framework that jointly leverages input-level data shift detection and output-level prediction confidence analysis to identify performance deterioration without requiring ground-truth labels. The approach innovatively integrates multi-source unsupervised shift indicators with dynamic confidence metrics within DomainSAT, a lightweight visualization tool. Extensive experiments on large-scale histopathological tumor classification datasets demonstrate that the proposed framework reliably and interpretably detects VLM performance degradation under distributional shifts, thereby significantly enhancing the robustness and trustworthiness of clinical deployment.

This study addresses a critical limitation in traditional reproducible research, where sharing only code and results fails to expose the implicit assumptions, expectations, and premises underlying an analyst’s reasoning—thereby hindering thorough evaluation of analytical quality. To overcome this, the paper proposes a formal modeling framework that explicitly translates the analyst’s tacit reasoning process into structured logical representations, statically capturing the construction logic of the analysis. This approach enables systematic scrutiny of the analytical chain of reasoning, assumption sensitivity, and conclusion robustness—even in the absence of the original data. Empirical validation on representative data analysis tasks demonstrates the framework’s effectiveness, achieving both logical visualization and data-free static assessment of analytical integrity.

Existing ML monitoring systems detect data distribution shifts but struggle to automatically identify their root causes—such as data defects, software failures, or genuine concept drift—relying instead on manual root-cause analysis. To address this, we propose a causal graph framework for ML systems, the first to integrate causal modeling with hierarchical system abstraction. It constructs a multi-granularity dependency graph spanning environmental variables, data pipelines, and model components. By analyzing propagation paths, the framework enables traceable attribution of distribution shifts back to their origins. This systematically establishes a mapping between observed distributional changes and their underlying causes, advancing ML monitoring from merely detecting *whether* a shift occurs to explaining *why* it occurs. The framework provides both theoretical foundations and a practical technical pathway toward interpretable, traceable, and automated ML monitoring. (149 words)