Problem
Research questions and friction points this paper is trying to address.
In-Context Reinforcement Learning
Large Language Models
Reward Signal
In-Context Learning
LLM Agents
Do LLMs Learn from Rewards in Context? : Rethinking the role of reward in In-Context Reinforcement Learning
🤖 AI Summary
This study investigates whether large language models genuinely leverage reward signals in in-context reinforcement learning (ICRL). By reformulating ICRL as a special case of in-context learning (ICL), the authors conduct systematic evaluations through multi-benchmark testing, meta-prompt engineering, and trajectory perturbation ablation experiments. The findings reveal that flipping or removing rewards exerts negligible influence on performance improvement curves, indicating that input distributions rather than reward signals drive inference-time enhancements, while the semantic content of trajectories is similarly non-essential. These results demonstrate that the underlying mechanism of existing approaches is fundamentally ICL, thereby challenging conventional paradigms in agent memory design that rely on reward shaping and exploration.
📝 Abstract
LLM agents increasingly improve at inference time by accumulating experience in context rather than by updating parameters. This process is often described as in-context reinforcement learning (ICRL). Whether in-context learning (ICL) can actually play the role of RL, however, has not been tested. We study this question in its simplest form, direct ICRL, where the model conditions directly on raw trajectory-reward pairs, and ask whether the reward acts as a learning signal. Through controlled experiments on four benchmarks across six models, we find that the reward is read, but its effect is small: flipping, randomizing, or removing the reward leaves the improvement curve almost unchanged, and this holds even under meta-prompts that explicitly instruct the model to explore, exploit, or reason over rewards. Trajectories drive improvement, but not through their semantic content: shuffled or corrupted trajectories work as well as real ones. These patterns closely mirror those known in ICL, suggesting that direct ICRL is better understood as a special case of ICL than as inference-time RL. This reframing has implications for agent memory design: ICL factors such as input distribution and demonstrations may matter more than RL elements such as reward shaping and exploration.
Innovation
Methods, ideas, or system contributions that make the work stand out.
In-Context Reinforcement Learning
Large Language Models
Reward Signal
In-Context Learning
Agent Memory Design
🔎 Similar Papers
No similar papers found.
