out-of-distribution testing

This work investigates the impact of concept shift—temporal variation in the input-label mapping—on the generalization performance of machine learning models. To address this form of distributional shift, the authors develop a high-dimensional asymptotic framework and derive, for the first time, a closed-form expression for the prediction risk of ridge regression under concept drift. Key theoretical findings are: (1) concept drift induces data-dependent, non-monotonic generalization curves—even in settings devoid of double descent; and (2) robust and non-robust features contribute to test error in opposing directions and with heterogeneous magnitudes. Methodologically, the study integrates high-dimensional statistical analysis, formal concept drift modeling, and rigorous theoretical derivation. Empirical validation on MNIST and FashionMNIST confirms the ubiquity of non-monotonic generalization behavior and demonstrates strong agreement between theoretical predictions and experimental results. This work advances the understanding of out-of-distribution generalization failure mechanisms and provides interpretable, theoretically grounded tools for enhancing model robustness.

Domain generalization (DG) aims to train models that generalize to unseen test domains. This paper identifies a critical disparity in empirical risk minimization (ERM) performance across distribution shift types: while ERM remains robust under covariate shift, it suffers severe degradation under posterior shift. To address this, we propose Domain-Aware Feature Enhancement (DAFE), a method that explicitly incorporates domain information into the ERM framework to mitigate posterior shift effects. Theoretical analysis establishes how domain-aware feature augmentation tightens the generalization error bound by reducing the discrepancy between domain-specific posterior distributions. Extensive multimodal experiments—spanning vision and language tasks—demonstrate that DAFE significantly outperforms standard ERM and state-of-the-art DG methods on datasets dominated by posterior shift. Our work introduces an interpretable, lightweight, and deployment-friendly DG paradigm, underscoring the necessity of shift-type diagnosis and shift-specific modeling in generalization-aware learning.

Domain adaptation (DA) faces practical challenges including difficulty in problem identification and lack of principled guidance for method selection. Method: This paper proposes the first problem-oriented DA framework, introducing a novel five-dimensional scenario taxonomy that systematically characterizes the causes and patterns of data distribution shift, complemented by a scenario identification guide and a method recommendation mechanism. Grounded in design science research, the framework undergoes iterative empirical evaluation across synthetic and real-world datasets, as well as a 100-participant user study. Contribution/Results: Results demonstrate significant improvements in interpretability, generality, and usability. The framework substantially enhances non-expert users’ accuracy in understanding DA tasks and their rationality in method selection, thereby addressing a critical gap in problem-driven adaptive decision-making research.

This study addresses the challenge of predicting generalization performance of deep learning models under distribution shift (non-IID settings). It presents the first systematic evaluation of over 40 generalization measures that rely solely on training data and model parameters, assessing their robustness across diverse architectures, training strategies, and types of distribution shifts. Through extensive hyperparameter search (>10,000 configurations) and comprehensive correlation analyses, the work uniquely incorporates calibration-based and information criterion–based metrics into the non-IID generalization evaluation framework. The findings reveal that the majority of existing measures substantially degrade under distribution shift, with only a few demonstrating consistent reliability. These results provide empirical grounding and practical guidance for evaluating generalization in non-IID scenarios.

Existing synthetic data evaluation lacks unified, transferable quantitative metrics. This paper proposes a novel evaluation framework grounded in generalized cross-validation (GCV) and domain transfer learning. It constructs a cross-dataset performance matrix and defines two core metrics: *fidelity*, quantifying distributional similarity between synthetic and real data; and *generalization coverage*, measuring the task-transfer capability of synthetic data across diverse real-world source domains. The framework is model-agnostic and enables normalized, comparative evaluation of detectors such as YOLOv5s across heterogeneous datasets—including Virtual KITTI, KITTI, and BDD100K. Experiments demonstrate that the method effectively quantifies synthetic data quality, significantly enhancing evaluation generality, comparability, and utility for model optimization. It establishes a scalable, reproducible, and standardized evaluation paradigm for synthetic data development.

This work addresses the critical challenge of evaluating model generalization in high-stakes scenarios with scarce labels, where existing methods lack reliable, label-free metrics for pre-deployment model selection and post-deployment performance monitoring. To bridge this gap, the study introduces, for the first time, the internal causal circuit mechanisms of Vision Transformers into generalization assessment, proposing two novel unsupervised metrics: Dependency Depth Bias and Circuit Shift Score. The former quantifies depth-wise biases in representational dependency structures, while the latter measures changes in circuit stability under distribution shifts. Extensive experiments across diverse tasks demonstrate that these metrics achieve substantially higher correlations with true generalization performance—improving by 13.4% and 34.1% on average over current approaches—thereby significantly enhancing the reliability of generalization prediction without requiring ground-truth labels.

This work addresses the unclear internal mechanisms underlying the failure of deep neural networks to generalize on unseen samples, particularly the lack of effective characterization of shifts in internal decision logic. The authors propose a novel perspective termed “Decision Pattern Shift” (DPS), which for the first time links generalization performance to the stability of internal decision processes. Specifically, they construct channel contribution vectors via GradCAM to represent the decision logic of individual samples and quantify generalization failure by measuring the deviation of these vectors from class-wise average patterns. The proposed framework offers a unified explanation for diverse generalization degradation scenarios, enabling early risk detection and precise defect localization. Experiments demonstrate that DPS exhibits a strong linear correlation with generalization gap (Pearson r > 0.8) and organizes various degradation cases into a continuous spectrum, revealing a systematic drift mechanism behind generalization failure.

This study addresses the safety risks posed by fine-tuning large language models, which can induce unexpected or even hazardous behaviors across domains—a phenomenon linked to "grokking generalization." The authors systematically reproduce and extend this effect across diverse models and datasets, demonstrating its strong dependence on specific model–data configurations. They propose a general-purpose prompting intervention that requires no prior knowledge of the particular risk. Through extensive fine-tuning experiments, cross-domain evaluations, and contextual prompting trials, they show that such anomalous generalization can be significantly mitigated with simple prompts. The findings indicate that the proposed method offers a practical and broadly applicable solution for enhancing AI alignment and safety without necessitating task-specific safeguards.

This study addresses the challenge of distribution shift in sensor-based human activity recognition (HAR), which arises from variations in devices, sensor placements, sampling rates, and user behaviors. It presents the first systematic investigation into four concrete types of such shifts and introduces a unified open-source benchmark platform alongside diverse datasets. The authors comprehensively evaluate 28 domain generalization methods against an empirical risk minimization baseline. Experimental results demonstrate that current domain generalization approaches offer only marginal improvements over the baseline, highlighting their limited efficacy in handling real-world distribution shifts. These findings underscore the shortcomings of existing techniques and provide critical insights and open resources to guide future research in robust HAR systems.

This work addresses the limited generalization of prompt-based large language model (LLM) classifiers in data-scarce scenarios, where insufficient fine-tuning often hinders performance. The authors propose a multi-task prompt fine-tuning approach that designs task-specific prompts while integrating general instruction tuning, substantially improving classification accuracy on unseen domains and novel prompts. Notably, they find that supervised classification training without explicit reasoning capabilities can effectively generalize to reasoning-intensive tasks such as summarization. To mitigate performance degradation caused by prompt variations, a hybrid training strategy is introduced. Experimental results demonstrate strong performance on related unseen tasks, highlighting the potential of classification-oriented training for building versatile, general-purpose monitoring systems.