Learning Data-Driven Uncertainty Set Partitions for Robust and Adaptive Energy Forecasting with Missing Data

2025-03-26

AI Summary

To address the sharp accuracy degradation in short-term energy forecasting caused by real-time feature missingness (e.g., sensor failures, cyberattacks) without prior knowledge of missing patterns, this paper proposes an adaptive robust forecasting framework. Methodologically, it introduces a data-driven, dynamically partitioned uncertainty set, integrating adversarial learning with adaptive robust optimization and supporting both linear and neural network models—eliminating the need for exhaustive combinatorial retraining and enabling millisecond-level online inference. Key contributions include: (i) the first integration of adversarial learning into adaptive robust optimization for missingness-aware prediction; and (ii) a lightweight uncertainty set learning algorithm that achieves theoretical near-optimal performance of full retraining using only a small subset of scenarios. Evaluated on wind power forecasting tasks spanning 15 minutes to 4 hours, the framework significantly outperforms imputation-based methods under prolonged missingness and maintains competitive accuracy during transient missingness.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Computer Vision: Adversarial Attacks & Robustness
• Search and Optimization: Adversarial Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Energy management for devices in mobile Web and WoT environments
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Short-term forecasting models typically assume the availability of input data (features) when they are deployed and in use. However, equipment failures, disruptions, cyberattacks, may lead to missing features when such models are used operationally, which could negatively affect forecast accuracy, and result in suboptimal operational decisions. In this paper, we use adaptive robust optimization and adversarial machine learning to develop forecasting models that seamlessly handle missing data operationally. We propose linear- and neural network-based forecasting models with parameters that adapt to available features, combining linear adaptation with a novel algorithm for learning data-driven uncertainty set partitions. The proposed adaptive models do not rely on identifying historical missing data patterns and are suitable for real-time operations under stringent time constraints. Extensive numerical experiments on short-term wind power forecasting considering horizons from 15 minutes to 4 hours ahead illustrate that our proposed adaptive models are on par with imputation when data are missing for very short periods (e.g., when only the latest measurement is missing) whereas they significantly outperform imputation when data are missing for longer periods. We further provide insights by showcasing how linear adaptation and data-driven partitions (even with a few subsets) approach the performance of the optimal, yet impractical, method of retraining for every possible realization of missing data.

Problem

• Develop robust energy forecasting models with missing data
• Handle missing features without historical pattern reliance
• Improve accuracy under short and long data gaps

Innovation

• Uses adaptive robust optimization for missing data
• Combines linear adaptation with data-driven partitions
• Neural network models adapt to available features