Learning to Sell: Reinforcement Learning for Strategic Large Language Model Agents in Multi-Product Markets

2026-09-27

AI Summary

This study addresses the challenge of sequential decision-making for LLM-based agents in multi-product markets characterized by information asymmetry and constrained interaction budgets. To tackle this, the authors formulate a Partially Observable Markov Decision Process (POMDP) framework and introduce a structured messaging protocol that maps natural language into the decision space. Furthermore, they design a reinforcement learning with verifiable rewards (RLVR) post-training methodology to optimize seller agent strategies. The resulting trained agents match or surpass trillion-parameter frontier models in both surplus extraction and allocation quality, while exhibiting robust generalization capabilities across previously unseen market structures.

Abstract

Autonomous large language model (LLM) agents operating in multi-product markets must make sequential decisions under information asymmetry and resource constraints. We develop a machine learning approach for training such agents to act effectively as sellers in a multi-item bargaining environment, where a seller concurrently negotiates a catalog of substitutable assets across a pool of independent buyers. Buyers hold private, heterogeneous valuations across products, and each can purchase at most one item. Facing limits on total communication turns, the seller must dynamically match buyers with the most profitable products considering their private valuations, while strategically allocating its limited interaction budget toward combinations of greater potential value. We formalize this problem as a Partially Observable Markov Decision Process using a structured, four-part message protocol that maps natural language into a parsable and regulated decision space. Using this formalization, we design a post-training method using Reinforcement Learning from Verifiable Rewards (RLVR). To evaluate this framework, we construct a multidimensional metric suite that quantifies constraint adherence, seller surplus extraction, and allocation quality. Our trained seller agent learns to match limited inventory to buyers more effectively, matching or outperforming trillion-parameter frontier models in both seller surplus extraction and buyer-product allocation quality. Finally, these learned strategies generalize robustly to unseen market structures, correlated valuation distributions, and price ranges not encountered during training.

Problem

• Large Language Model Agents
• Multi-Product Markets
• Information Asymmetry
• Resource Allocation
• Bargaining

Innovation

• Reinforcement Learning from Verifiable Rewards
• Partially Observable Markov Decision Process
• Large Language Model Agents
• Multi-Product Bargaining
• Structured Message Protocol