From Uncertainty to Action: Learning to Steer LLM Agents

2026-10-06

AI Summary

This study addresses the challenges of selecting intervention timing and mechanisms for LLM agents, as well as the limitations of uncertainty signals in precisely identifying effective steps. We propose an optimization framework based on step-level intervention value modeling. By constructing step-level outcome tables to analyze counterfactual trajectories, we design a Value of Step (VoS) monitor to learn intervention values and introduce a harm-budget-constrained triggering mechanism to balance success rates against intervention perturbation. This work pioneers step-level intervention value modeling using large-scale counterfactual data, integrating offline and online monitoring with reinforcement learning-style value estimation across multiple benchmarks. Experiments demonstrate an average improvement of 7.8 points across 12 settings, surpassing the strongest baseline by 2.9 points, thereby validating the effectiveness of counterfactual data and budget constraints.

Abstract

Steering an LLM agent means deciding whether to correct it, at which step, and with which mechanism. Uncertainty is often used to decide when to correct an agent, but whether it can guide these decisions remains unclear. We steer agent trajectories separately at every non-terminal step with each of four mechanisms and run each continuation to completion. The resulting stepwise outcome table (SOT) holds about 82,000 counterfactual continuations of 1,864 trajectories from three benchmarks and two agents. It shows that uncertainty can identify failing trajectories, but that no single signal reliably locates the step at which steering helps. We therefore propose VoS (Value of Steering), a trajectory-level monitor, offline or online, that learns from SOT the value of steering at each step and decides where to steer by it. A harm-budgeted trigger decides whether to steer, limiting the fraction of successful trajectories that VoS disturbs. VoS improves on unmodified execution in all 12 settings of benchmark, agent, and offline or online use, by 7.8 points on average, and outperforms the strongest of five existing uncertainty-triggered methods in 11, by 2.9 points on average. Ablations show that training on measured outcomes and a tight harm budget are both essential.

Problem

• LLM agents
• agent steering
• uncertainty
• trajectory correction
• Value of Steering

Innovation

• Value of Steering
• Stepwise Outcome Table
• LLM Agents
• Uncertainty
• Harm-budgeted Trigger