Marginalize, Rather than Impute: Probabilistic Wind Power Forecasting with Incomplete Data

2024-03-06

AI Summary

In wind power probabilistic forecasting, sensor missing data are pervasive; however, the prevailing “impute-then-predict” paradigm introduces estimation bias and fails to propagate missingness uncertainty. To address this, we abandon the two-stage approach and propose an end-to-end framework based on joint generative modeling: input features and target variables are jointly modeled as a single distribution, and missing features are marginalized out via Bayesian integration—ensuring faithful uncertainty propagation. Our method employs expressive generative models—such as conditional variational autoencoders or normalizing flows—that natively handle incomplete inputs and implicitly learn the missingness mechanism during training. Experiments demonstrate that our approach achieves statistically significant improvements in Continuous Ranked Probability Score (CRPS) over diverse imputation-based baselines. Moreover, it incurs substantially lower inference overhead than existing uncertainty-aware forecasting methods, delivering superior accuracy, robustness to missingness patterns, and computational efficiency.

Technology Category

• Reasoning under Uncertainty: Relational Probabilistic Models
• Machine Learning: Probabilistic Circuits and Graphical Models
• Intelligent Robots: State Estimation

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Machine learning methods are widely and successfully used for probabilistic wind power forecasting, yet the pervasive issue of missing values (e.g., due to sensor faults or communication outages) has received limited attention. The prevailing practice is impute-then-predict, but conditioning on point imputations biases parameter estimates and fails to propagate uncertainty from missing features. Our approach treats missing features and forecast targets uniformly: we learn a joint generative model of features and targets from incomplete data and, at operational deployment, condition on the observed features and marginalize the unobserved ones to produce forecasts. This imputation-free procedure avoids error introduced by imputation and preserves uncertainty aroused from missing features. In experiments, it improves forecast quality in terms of continuous ranked probability score relative to impute-then-predict baselines while incurring substantially lower computational cost than common alternatives.

Problem

• Addresses missing values in wind power forecasting data
• Proposes marginalization over imputation to reduce bias
• Improves forecast accuracy and computational efficiency

Innovation

• Uses joint generative model for features and targets
• Marginalizes missing features instead of imputing them
• Reduces computational cost while improving forecast accuracy