🤖 AI Summary
This work addresses the challenge of thermal energy storage scheduling, which requires multi-constraint planning several hours ahead and is difficult to generalize across buildings using conventional methods. The authors propose Reinforcement Learning with Verifiable Rewards (RLVR), a novel approach that, for the first time, transforms action values generated by dynamic programming (DP) into dense rewards to fine-tune an open-source large reasoning model with only 30 few-shot examples. This enables the model to function as a high-level scheduler that outputs heat pump setpoints while stabilizing its internal planning mechanisms—such as candidate comparison, lookahead, and feasibility checking. The method reduces carbon emissions from 70.5 to 61.2 kg-CO₂, closely approaching the DP-optimal value of 60.8, and demonstrates strong generalization under prediction errors, novel storage configurations, and cross-task scenarios.
📝 Abstract
Buildings are expected to shift cooling loads in response to grid conditions. Thermal energy storage (TES) enables this shift, but scheduling it well requires planning hours ahead under storage constraints. Model predictive control (MPC) and reinforcement learning are difficult to scale across buildings. This study instead adapts an open-weight reasoning model through reinforcement learning with verifiable rewards (RLVR). We convert exact offline dynamic-programming (DP) action values into dense rewards for every candidate action. Using only 30 training prompts, reinforcement fine-tuning (RFT) trains the model as an upper-level scheduler that outputs hourly heat-pump setpoints from text-based states and forecasts. Evaluation uses a deliberately simple office-building TES benchmark where exact DP is tractable and the optimum is known. RFT reduces the open-weight model's emissions from 70.5 to 61.2 kg-CO2, close to the DP optimum of 60.8 kg-CO2. GPT-5 nearly matches DP and MPC without task-specific training, while GPT-4o, a non-reasoning LLM, produces higher emissions than the no-storage baseline, so inference-time reasoning appears important. Trace analysis shows that RFT mainly stabilizes observable planning patterns (candidate comparison, look-ahead, and feasibility checking) rather than creating a new strategy. Robustness and generalization tests clarify what transfers: the reinforced planning patterns persist under forecast errors and an unseen TES condition and carry over to a battery task, but its different structure limits the gains. DP-based verifiable rewards offer a practical way to adapt open-weight reasoning models to building storage scheduling. These results motivate higher-fidelity tests of whole-building control and scalable verifiers for city-scale energy management.