Towards Improving Sequential Decision-Making in LLM Agents via Experience Memory

2026-08-04

AI Summary

This work addresses the limited ability of large language models (LLMs) to approximate optimal policies in sequential decision-making tasks, even in well-defined two-player zero-sum games. The authors propose a fine-tuning-free agent framework that integrates post-hoc reflection, rule extraction, and experience memory mechanisms, coupled with Monte Carlo tree search for evaluation and retrieval. This approach effectively mitigates the credit assignment problem and, for the first time, substantially enhances LLMs’ game-playing performance without altering model weights. The method significantly narrows the gap between LLM behavior and optimal strategies in tasks such as Tic-Tac-Toe and demonstrates robustness to superficial variations in game representation.

Abstract

Large language models have improved substantially on single-shot reasoning tasks, but their performance in sequential decision-making is less well understood. We study this on fully-observable two-player zero-sum games, which provide ground-truth evaluation: outcomes are determined by the rules, and optimality of individual moves can be computed or approximated, without relying on a judge model. Across model tiers, LLMs play suboptimally in simple games such as tic-tac-toe or Connect Four, and lose to MCTS opponents. Obfuscations that preserve the game tree but rewrite its surface form leave performance largely unchanged, indicating the gap is not fully explained by recall of memorized strategies. Motivated by this performance gap, we introduce an agentic framework enhanced with an experience memory designed for the sequential setting and addressing common challenges of sequential decision-making such as credit assignment. We show that post-game reflection and rule extraction yield measurable improvements on tic-tac-toe without modifying the model weights.

Problem

• sequential decision-making
• large language models
• zero-sum games
• credit assignment
• experience memory

Innovation

• experience memory
• sequential decision-making
• post-game reflection
• credit assignment
• LLM agents