Rate-In: Information-Driven Adaptive Dropout Rates for Improved Inference-Time Uncertainty Estimation

2024-12-10

AI Summary

Fixed dropout rates in Monte Carlo Dropout lead to biased uncertainty estimates in high-stakes applications such as medical diagnosis. Method: We propose the first inference-time, unsupervised, layer- and sample-adaptive dropout rate scheduling method. It dynamically computes per-layer, input-specific information sensitivity—quantified via functional information loss on feature maps—to generate input-aware and layer-aware dropout rates, without requiring labels or assumptions about training data distribution. Grounded in information-theoretic modeling and stochastic neural network inference, the method operates entirely at test time. Contribution/Results: Evaluated on multimodal medical imaging tasks, our approach significantly improves uncertainty calibration and resolution while preserving predictive accuracy. It further demonstrates strong generalization across diverse network architectures and out-of-distribution scenarios.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Stochastic Optimization
• Planning, Routing, and Scheduling: Scheduling under Uncertainty

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Accurate uncertainty estimation is crucial for deploying neural networks in risk-sensitive applications such as medical diagnosis. Monte Carlo Dropout is a widely used technique for approximating predictive uncertainty by performing stochastic forward passes with dropout during inference. However, using static dropout rates across all layers and inputs can lead to suboptimal uncertainty estimates, as it fails to adapt to the varying characteristics of individual inputs and network layers. Existing approaches optimize dropout rates during training using labeled data, resulting in fixed inference-time parameters that cannot adjust to new data distributions, compromising uncertainty estimates in Monte Carlo simulations. In this paper, we propose Rate-In, an algorithm that dynamically adjusts dropout rates during inference by quantifying the information loss induced by dropout in each layer's feature maps. By treating dropout as controlled noise injection and leveraging information-theoretic principles, Rate-In adapts dropout rates per layer and per input instance without requiring ground truth labels. By quantifying the functional information loss in feature maps, we adaptively tune dropout rates to maintain perceptual quality across diverse medical imaging tasks and architectural configurations. Our extensive empirical study on synthetic data and real-world medical imaging tasks demonstrates that Rate-In improves calibration and sharpens uncertainty estimates compared to fixed or heuristic dropout rates without compromising predictive performance. Rate-In offers a practical, unsupervised, inference-time approach to optimizing dropout for more reliable predictive uncertainty estimation in critical applications.

Problem

• Uncertainty Estimation
• Monte Carlo Dropout
• Neural Networks

Innovation

• Rate-In Algorithm
• Dynamic Dropout Rate Adjustment
• Predictive Uncertainty Estimation