🤖 AI Summary
This work addresses the limitation of current large language model (LLM) planning agent evaluations, which predominantly focus on task success while neglecting the dynamic impact of other agents’ responses and physical constraints in cyber-physical systems. The authors introduce the first physically verifiable benchmark platform for demand response in smart grids, evaluating the strategic efficacy of four planning architectures—predefined, sequential, hierarchical, and search-based—within a simulated environment comprising 40 heterogeneous prosumers and a radial feeder. They propose an evaluation protocol based on paired forced counterfactuals, common random responses, and event-level deadline feasibility, combined with typed policy declarations and short instruction constraints to explicitly model schedule generation, prosumer dynamics, and power flow computation. Experiments show that three architectures yield feasible, near-optimal solutions; incorporating deadline feasibility prediction reduces average regret from 90.7 to 29.0, outperforming fixed sequential strategies by 61.1%, underscoring the substantial influence of planning architecture and highlighting solution quality selection among feasible outcomes as a key challenge.
📝 Abstract
Evaluations of LLM planning agents largely ask whether a task succeeds or a declared plan is followed. In strategic cyber-physical systems, a stronger question is whether the planning architecture remains appropriate after autonomous participants respond and physics constrains the outcome. We introduce a controlled, physics-grounded benchmark built around planning-induced control trajectories: the ordered planning operations and directives through which an execution architecture acts on other agents and the physical process. It implements predefined, sequential, hierarchical, and search executors in a smart-grid demand-response system with 40 heterogeneous prosumers and an independently simulated radial feeder. The LLM is bounded to typed policy declaration and short operator messages, while schedule construction, prosumer dynamics, and power flow remain explicit code. The protocol uses paired forced-mode counterfactuals, common random response draws, and event-level deadline feasibility. Three properties follow. Architecture materially changes outcomes: forced search is the oracle in all five baseline seeds. Execution fidelity needs more than mode agreement: objective substitution holds agreement at 1.0 while increasing voltage shortfall by 2.68x. A 144-scenario, 576-episode bank has feasible oracles from three of the four architectures. A prespecified stress-held-out ridge has mean regret 90.7 (95% interval [73.8, 108.6]) and no detectable value over fixed sequential; applying known deadline feasibility before quality prediction cuts regret to 29.0 and improves over fixed sequential by 61.1. An all-feasible ablation does not beat fixed search, localising the remaining challenge to within-feasible quality selection. A five-model extension separates stress-conditioned, state-blind, and invariant declarers; latency tails show that live feasibility should be treated probabilistically.