Restless Multi-Process Multi-Armed Bandits with Applications to Self-Driving Microscopies

2025-12-16

AI Summary

High-content live-cell imaging faces challenges in dynamically sampling regions of interest (ROIs) efficiently while avoiding missed critical biological events; existing static or heuristic approaches fail to jointly optimize phototoxicity, computational cost, and temporal constraints. This paper proposes a restless multi-process multi-armed bandit (RMPMAB) framework that, for the first time, models each ROI as an ensemble of heterogeneous Markov processes to capture asynchrony in cell-cycle progression and heterogeneity in drug response. We derive closed-form analytical solutions for both transient and steady-state dynamics and design a Whittle index policy with sublinear computational complexity. Simulation results demonstrate over 37% reduction in cumulative regret. In real live-cell experiments, the method increases the capture rate of biologically relevant events by 93%, significantly enhancing event discovery throughput under stringent phototoxicity constraints.

Technology Category

• Machine Learning: Online Learning & Bandits
• Reasoning under Uncertainty: Stochastic Optimization
• Intelligent Robots: Learning & Optimization for ROB

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

High-content screening microscopy generates large amounts of live-cell imaging data, yet its potential remains constrained by the inability to determine when and where to image most effectively. Optimally balancing acquisition time, computational capacity, and photobleaching budgets across thousands of dynamically evolving regions of interest remains an open challenge, further complicated by limited field-of-view adjustments and sensor sensitivity. Existing approaches either rely on static sampling or heuristics that neglect the dynamic evolution of biological processes, leading to inefficiencies and missed events. Here, we introduce the restless multi-process multi-armed bandit (RMPMAB), a new decision-theoretic framework in which each experimental region is modeled not as a single process but as an ensemble of Markov chains, thereby capturing the inherent heterogeneity of biological systems such as asynchronous cell cycles and heterogeneous drug responses. Building upon this foundation, we derive closed-form expressions for transient and asymptotic behaviors of aggregated processes, and design scalable Whittle index policies with sub-linear complexity in the number of imaging regions. Through both simulations and a real biological live-cell imaging dataset, we show that our approach achieves substantial improvements in throughput under resource constraints. Notably, our algorithm outperforms Thomson Sampling, Bayesian UCB, epsilon-Greedy, and Round Robin by reducing cumulative regret by more than 37% in simulations and capturing 93% more biologically relevant events in live imaging experiments, underscoring its potential for transformative smart microscopy. Beyond improving experimental efficiency, the RMPMAB framework unifies stochastic decision theory with optimal autonomous microscopy control, offering a principled approach to accelerate discovery across multidisciplinary sciences.

Problem

• Optimizes dynamic imaging decisions for live-cell microscopy under resource constraints.
• Models biological heterogeneity using Markov chains to capture asynchronous processes.
• Improves throughput and reduces regret compared to existing heuristic methods.

Innovation

• Modeling experimental regions as ensembles of Markov chains
• Deriving closed-form expressions for process behaviors
• Designing scalable Whittle index policies with sub-linear complexity