RL + Transformer = A General-Purpose Problem Solver

2025-01-24

AI Summary

This work addresses the challenge of enhancing zero-shot generalization of AI agents to unseen tasks. We propose In-Context Reinforcement Learning (ICRL), a paradigm that fine-tunes pretrained Transformers via multi-round online reinforcement learning, enabling autonomous policy refinement solely from a few interaction trajectories within the context—without parameter updates. Our key contributions are threefold: (1) the first empirical demonstration that RL-finetuned Transformers exhibit emergent self-iterative meta-learning capabilities; (2) a context-aware behavior concatenation mechanism to structure trajectory history; and (3) a non-stationary environment adaptation strategy for robust policy optimization. Experiments show ICRL significantly outperforms conventional RL algorithms under zero-shot evaluation, achieving several-fold improvements in sample efficiency. Moreover, it demonstrates strong in-distribution and out-of-distribution generalization, resilience to high observation noise, and adaptability to dynamic environments—marking a step toward universal problem solving.

Technology Category

• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Multiagent Systems: Adversarial Agents
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

What if artificial intelligence could not only solve problems for which it was trained but also learn to teach itself to solve new problems (i.e., meta-learn)? In this study, we demonstrate that a pre-trained transformer fine-tuned with reinforcement learning over multiple episodes develops the ability to solve problems that it has never encountered before - an emergent ability called In-Context Reinforcement Learning (ICRL). This powerful meta-learner not only excels in solving unseen in-distribution environments with remarkable sample efficiency, but also shows strong performance in out-of-distribution environments. In addition, we show that it exhibits robustness to the quality of its training data, seamlessly stitches together behaviors from its context, and adapts to non-stationary environments. These behaviors demonstrate that an RL-trained transformer can iteratively improve upon its own solutions, making it an excellent general-purpose problem solver.

Problem

• Artificial Intelligence
• Self-learning
• Adaptability

Innovation

• ICRL
• Adaptive Learning
• Generalization