PARL: Prompt-based Agents for Reinforcement Learning

2025-10-24

AI Summary

This work investigates the zero-shot decision-making capability of large language models (LLMs) as reinforcement learning (RL) agents in non-linguistic, structured tasks—exemplified by grid-world navigation. To address LLMs’ inherent inability to natively process non-textual state-action spaces, we propose PARL: a prompting-based framework that encodes states, actions, and rewards into natural language via prompt engineering, integrates in-context learning, and maps discrete actions to linguistic tokens—enabling end-to-end, fine-tuning-free trial-and-error learning. To our knowledge, this is the first systematic effort to deploy LLMs directly on RL tasks devoid of linguistic priors. Experiments across three canonical RL benchmarks demonstrate that PARL matches or surpasses conventional RL methods, substantiating LLMs’ viability as general-purpose agents. However, the approach exhibits pronounced limitations on tasks demanding complex mathematical reasoning.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Large language models (LLMs) have demonstrated high performance on tasks expressed in natural language, particularly in zero- or few-shot settings. These are typically framed as supervised (e.g., classification) or unsupervised (e.g., clustering) problems. However, limited work evaluates LLMs as agents in reinforcement learning (RL) tasks (e.g., playing games), where learning occurs through interaction with an environment and a reward system. While prior work focused on representing tasks that rely on a language representation, we study structured, non-linguistic reasoning - such as interpreting positions in a grid world. We therefore introduce PARL (Prompt-based Agent for Reinforcement Learning), a method that uses LLMs as RL agents through prompting, without any fine-tuning. PARL encodes actions, states, and rewards in the prompt, enabling the model to learn through trial-and-error interaction. We evaluate PARL on three standard RL tasks that do not entirely rely on natural language. We show that it can match or outperform traditional RL agents in simple environments by leveraging pretrained knowledge. However, we identify performance limitations in tasks that require complex mathematical operations or decoding states and actions.

Problem

• Using LLMs as reinforcement learning agents without fine-tuning
• Enabling trial-and-error learning through structured prompt encoding
• Evaluating performance on non-linguistic reasoning tasks with limitations

Innovation

• LLMs as RL agents without fine-tuning
• Encoding states and rewards in prompts
• Learning through trial-and-error interaction