Reinforcement learning meets bioprocess control through behaviour cloning: Real-world deployment in an industrial photobioreactor

2025-09-08

AI Summary

Addressing the challenge of pH regulation in open photobioreactors—characterized by nonlinear dynamics, environmental disturbances, and poor robustness—this paper proposes an adaptive control strategy integrating reinforcement learning (RL) and behavior cloning (BC). To our knowledge, this is the first application of RL to such systems. We introduce a hybrid training framework comprising offline pretraining—unifying PID-based trajectory generation, RL, and BC—with online daily fine-tuning. The method significantly enhances responsiveness to rapid disturbances and adaptability across varying operating conditions. Simulation results demonstrate an 8% reduction in integral absolute error (IAE) and a 54% decrease in control energy consumption. An 8-day experimental validation confirms superior robustness and reliability under dynamic environmental conditions. The approach achieves high-precision, low-cost intelligent pH regulation, advancing autonomous operation of open photobioreactors.

Technology Category

Intelligent Robots: Behavior Learning & ControlMachine Learning: Reinforcement LearningHumans and AI: Human-Aware Planning and Behavior Prediction

Application Category

Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for rankingEconomics, Online Markets and Human Computation: Cost models of using LLMs in production systemsUser Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

The inherent complexity of living cells as production units creates major challenges for maintaining stable and optimal bioprocess conditions, especially in open Photobioreactors (PBRs) exposed to fluctuating environments. To address this, we propose a Reinforcement Learning (RL) control approach, combined with Behavior Cloning (BC), for pH regulation in open PBR systems. This represents, to the best of our knowledge, the first application of an RL-based control strategy to such a nonlinear and disturbance-prone bioprocess. Our method begins with an offline training stage in which the RL agent learns from trajectories generated by a nominal Proportional-Integral-Derivative (PID) controller, without direct interaction with the real system. This is followed by a daily online fine-tuning phase, enabling adaptation to evolving process dynamics and stronger rejection of fast, transient disturbances. This hybrid offline-online strategy allows deployment of an adaptive control policy capable of handling the inherent nonlinearities and external perturbations in open PBRs. Simulation studies highlight the advantages of our method: the Integral of Absolute Error (IAE) was reduced by 8% compared to PID control and by 5% relative to standard off-policy RL. Moreover, control effort decreased substantially-by 54% compared to PID and 7% compared to standard RL-an important factor for minimizing operational costs. Finally, an 8-day experimental validation under varying environmental conditions confirmed the robustness and reliability of the proposed approach. Overall, this work demonstrates the potential of RL-based methods for bioprocess control and paves the way for their broader application to other nonlinear, disturbance-prone systems.

Problem

• Control pH in open photobioreactors with fluctuating environments
• Handle nonlinear bioprocesses with external disturbances
• Reduce control error and operational costs efficiently

Innovation

• Reinforcement Learning combined with Behavior Cloning
• Offline training from PID controller trajectories
• Daily online fine-tuning for disturbance adaptation