From Prediction to Practice: A Task-Aware Evaluation Framework for Blood Glucose Forecasting

2026-05-01

AI Summary

Current glucose prediction models rely on aggregate error metrics that fail to capture their practical utility in critical clinical tasks such as hypoglycemia warning and insulin dosing decisions, potentially masking severe failures—especially in high-risk scenarios. This work introduces, for the first time, a task-oriented evaluation paradigm comprising a dual-arm framework: event-level hypoglycemia alert assessment using multicenter clinical data, and counterfactual insulin intervention decision evaluation based on the FDA-recognized UVA/Padova simulator. Operational metrics—including event-level recall, daily false alarm rate, and clinical cost functions—reveal that even high-accuracy models exhibit significant performance degradation during postprandial peaks and consistently fail to predict both the direction and relative magnitude of required insulin adjustments, thereby exposing fundamental limitations in their clinical applicability.

Abstract

Clinical time-series forecasting is increasingly studied for decision support, yet standard aggregate metrics can obscure whether a model is actually useful for the task it is meant to serve. In safety-critical settings, low average error can coexist with dangerous failures in exactly the high-risk regimes that matter most. We present a task-aware evaluation framework for blood glucose forecasting built around two downstream uses: hypoglycemia early warning and insulin dosing decision support. For early warning, we evaluate on real data from three clinical cohorts using event-level recall and false alarms per patient-day, metrics that reflect operational alarm burden rather than aggregate accuracy. We show that models appearing acceptable overall, with recall above 0.9 on the full test set, can fail badly in the post-bolus slice, where insulin-on-board is elevated and missed warnings carry the greatest clinical consequences. Standard forecasting evaluation, however, does not test whether a model can reason about the effects of actions, a requirement for supporting insulin dosing decisions. We therefore add a second, interventional arm using the FDA-accepted UVA/Padova simulator, where we evaluate whether forecasters can predict glucose responses to altered insulin plans in paired factual/counterfactual scenarios. We show that models that look strong on real-data forecasting often fail to predict the direction, magnitude, or ranking of intervention effects, and choose poor insulin doses when evaluated under a clinically motivated cost. Taken together, the two arms reveal a consistent gap between forecasting accuracy and task-relevant usefulness. We release the benchmark, the standardized preprocessing pipeline for public cohorts, and the simulator-based interventional dataset as a reproducible toolkit.

Problem

• blood glucose forecasting
• task-aware evaluation
• hypoglycemia early warning
• insulin dosing
• clinical decision support

Innovation

• task-aware evaluation
• blood glucose forecasting
• interventional simulation
• hypoglycemia early warning
• counterfactual reasoning