Learning to be Reproducible: Custom Loss Design for Robust Neural Networks

2026-01-02

arXiv.org

AI Summary

This work addresses the inherent instability and irreproducibility of deep learning models under identical training conditions, which often arise from random factors such as weight initialization and data shuffling. To mitigate this issue, the study explicitly incorporates training stability into the optimization objective and proposes a novel custom loss function (CLF) that significantly reduces model sensitivity to stochasticity while preserving predictive accuracy. Extensive experiments across diverse tasks—including image classification and time series forecasting—and various architectures demonstrate that the proposed method consistently enhances training robustness, effectively achieving a unified balance between high accuracy and high reproducibility.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Reasoning under Uncertainty: Stochastic Optimization
• Computer Vision: Learning & Optimization for CV

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

To enhance the reproducibility and reliability of deep learning models, we address a critical gap in current training methodologies: the lack of mechanisms that ensure consistent and robust performance across runs. Our empirical analysis reveals that even under controlled initialization and training conditions, the accuracy of the model can exhibit significant variability. To address this issue, we propose a Custom Loss Function (CLF) that reduces the sensitivity of training outcomes to stochastic factors such as weight initialization and data shuffling. By fine-tuning its parameters, CLF explicitly balances predictive accuracy with training stability, leading to more consistent and reliable model performance. Extensive experiments across diverse architectures for both image classification and time series forecasting demonstrate that our approach significantly improves training robustness without sacrificing predictive performance. These results establish CLF as an effective and efficient strategy for developing more stable, reliable and trustworthy neural networks.

Problem

• reproducibility
• robustness
• training stability
• deep learning
• model reliability

Innovation

• Custom Loss Function
• Training Stability
• Reproducibility
• Robust Neural Networks
• Stochastic Sensitivity