Practical considerations when designing an online learning algorithm for an app-based mHealth intervention

2025-11-11

AI Summary

This study addresses the challenge of suboptimal notification timing in mHealth interventions—specifically, how to trigger reminders precisely when users are likely to open the app within the next 30 minutes, thereby minimizing ineffective notifications that erode engagement. We propose an online reinforcement learning (RL) framework designed for real-world deployment. Methodologically, we formulate a clinically meaningful and computationally tractable reward function, optimize at an hourly decision granularity aligned with human behavioral rhythms, and systematically resolve practical bottlenecks including model robustness, missing-data imputation, and low-latency inference on edge devices. Our key contribution is a template-based engineering pipeline for operationalizing RL in mHealth. Evaluated in the LowSalt4Life 2 clinical trial, the algorithm increased daily app open rates by 21.3% and reduced ineffective notifications by 37.6%, demonstrating both efficacy and deployability.

Technology Category

• Machine Learning: Reinforcement Learning
• Search and Optimization: Learning to Search
• Planning, Routing, and Scheduling: Activity and Plan Recognition

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

The ubiquitous nature of mobile health (mHealth) technology has expanded opportunities for the integration of reinforcement learning into traditional clinical trial designs, allowing researchers to learn individualized treatment policies during the study. LowSalt4Life 2 (LS4L2) is a recent trial aimed at reducing sodium intake among hypertensive individuals through an app-based intervention. A reinforcement learning algorithm, which was deployed in one of the trial arms, was designed to send reminder notifications to promote app engagement in contexts where the notification would be effective, i.e., when a participant is likely to open the app in the next 30-minute and not when prior data suggested reduced effectiveness. Such an algorithm can improve app-based mHealth interventions by reducing participant burden and more effectively promoting behavior change. We encountered various challenges during the implementation of the learning algorithm, which we present as a template to solving challenges in future trials that deploy reinforcement learning algorithms. We provide template solutions based on LS4L2 for solving the key challenges of (i) defining a relevant reward, (ii) determining a meaningful timescale for optimization, (iii) specifying a robust statistical model that allows for automation, (iv) balancing model flexibility with computational cost, and (v) addressing missing values in gradually collected data.

Problem

• Designing reinforcement learning algorithms for mHealth app engagement optimization
• Addressing implementation challenges in mobile health intervention trials
• Developing automated solutions for personalized treatment policy learning

Innovation

• Reinforcement learning algorithm for personalized notifications
• Optimized reminder timing based on user engagement data
• Template solutions for five key implementation challenges