🤖 AI Summary
This work addresses systematic limitations in existing large language model evaluation frameworks—particularly their inadequacies in distributional coverage, temporal dynamics, scope, and procedural fidelity—which hinder effective assessment of embodied agents’ long-term reasoning and behavior and exacerbate reward hacking in reinforcement learning from human feedback (RLHF). To overcome these issues, the authors propose the Grounded Continuous Evaluation (GCE) framework, which introduces a simulation-based ISOPro system that replaces learned reward models with deterministic ground-truth verifiers, thereby structurally eliminating reward hacking. GCE enables LoRA weight updates on the CPU, drastically lowering hardware requirements, and pioneers a continuous evaluation paradigm that embeds assessment directly into training, implicitly inducing curriculum formation without manual design. Using only 0.216% trainable parameters, GCE achieves threefold higher accuracy than zero-shot baselines on resource scheduling tasks and demonstrates, for the first time on consumer-grade hardware, emergent capabilities contingent on continuous evaluation.
📝 Abstract
We argue that current evaluation frameworks for large language models (LLMs) suffer from four systematic failures that make them structurally inadequate for assessing deployed, agentic systems: distributional invalidity (evaluation inputs do not reflect real interaction distributions), temporal invalidity (evaluations are post-hoc rather than training-integrated), scope invalidity (evaluations measure single-turn outputs rather than long-horizon trajectories), and process invalidity (evaluations assess outputs rather than reasoning). These failures compound critically in RLHF, where reward models are evaluated under conditions that do not hold during RL training, making reward hacking a predictable consequence of evaluation design rather than a training pathology. We propose the Grounded Continuous Evaluation (GCE) framework and present ISOPro, a simulation-based fine-tuning and evaluation system. ISOPro replaces the learned reward model with a deterministic ground-truth verifier, eliminating reward hacking by construction in verifiable-reward domains, and operates on LoRA adapter weights updatable on CPU, reducing the hardware barrier by an order of magnitude. We validate ISOPro on a resource-constrained scheduling domain with six difficulty tiers, demonstrating capability emergence visible only through continuous evaluation, an implicit curriculum that forms without researcher curation, and a 3x accuracy improvement over zero-shot baselines, all on consumer hardware with 0.216% trainable parameters.