Policy Gradient Steering: Interventions from Behavioral Objectives

2026-07-29

AI Summary

Existing activation intervention methods struggle to effectively steer agent behavior even in simple decision-making tasks. This work formulates behavioral intervention as a reinforcement learning problem for the first time, constructing removable, composable, and reversible task vectors by accumulating policy gradients toward temporary behavioral objectives over a small number of trajectories. The resulting framework enables dynamic behavioral modulation and supports cross-task customization and composition of behaviors. Empirical validation demonstrates calibrated and reversible interventions in grid-world environments, flexible composition of tactical objectives in chess, and successful modification of team-specific behaviors in a football simulation setting, with effective generalization across diverse opponents.

Abstract

Activation steering has emerged in large language models as a lightweight alternative for dynamically changing a model's behavior at inference time. However, we show that existing steering methods fail to steer even a simple policy in a two-route gridworld environment. To address this limitation, we propose Policy Gradient Steering (PGS), which formulates steering as a reinforcement learning problem. PGS accumulates gradients of a temporary behavioral objective over a small set of rollouts or demonstrations to construct a removable task vector. We first demonstrate the calibration and reversibility of PGS in a two-route gridworld environment. Using chess puzzles, we then evaluate independently fitted PGS vectors both in isolation and in combination, finding that compatible tactical objectives accumulate constructively. Finally, in competitive football, we show that PGS can alter specific team behaviors and that its effects transfer across opponents. Together, these results show that policy gradients provide a natural interface for constructing temporary and composable behavioral adaptations across diverse decision-making domains.

Problem

• activation steering
• behavioral control
• policy gradient
• inference-time intervention
• reinforcement learning

Innovation

• Policy Gradient Steering
• behavioral intervention
• task vector
• reinforcement learning
• composable adaptation