Technology Category
Machine Learning: Large Multimodal Models (LMMs)Game Theory and Economic Paradigms: Adversarial LearningSearch and Optimization: Learning to Search
PokerBench: Training Large Language Models to become Professional Poker Players
🤖 AI Summary
This work addresses the evaluation and enhancement of higher-order intelligence in large language models (LLMs) within incomplete-information games—exemplified by professional poker. To this end, we introduce PokerBench, the first poker-specific benchmark comprising 11,000 expert-level pre-flop and flop scenarios, systematically assessing mathematical reasoning, strategic planning, and opponent modeling. We establish poker as a novel paradigm for evaluating LLMs’ higher-order cognition; propose a verifiable evaluation protocol wherein PokerBench scores exhibit strong correlation with actual win rates against human experts (r > 0.85); and identify fundamental limitations of supervised fine-tuning, advocating reinforcement learning and game-theoretic alignment as key optimization pathways. Experiments across GPT-4, Llama, and Gemma show that state-of-the-art models significantly underperform theoretical optimal strategies out-of-the-box, yet achieve substantial gains post-fine-tuning—with higher PokerBench scores predicting superior real-world win rates in human–AI matches. The dataset, code, and evaluation framework are publicly released.
Technology Category
Application Category
📝 Abstract
We introduce PokerBench - a benchmark for evaluating the poker-playing abilities of large language models (LLMs). As LLMs excel in traditional NLP tasks, their application to complex, strategic games like poker poses a new challenge. Poker, an incomplete information game, demands a multitude of skills such as mathematics, reasoning, planning, strategy, and a deep understanding of game theory and human psychology. This makes Poker the ideal next frontier for large language models. PokerBench consists of a comprehensive compilation of 11,000 most important scenarios, split between pre-flop and post-flop play, developed in collaboration with trained poker players. We evaluate prominent models including GPT-4, ChatGPT 3.5, and various Llama and Gemma series models, finding that all state-of-the-art LLMs underperform in playing optimal poker. However, after fine-tuning, these models show marked improvements. We validate PokerBench by having models with different scores compete with each other, demonstrating that higher scores on PokerBench lead to higher win rates in actual poker games. Through gameplay between our fine-tuned model and GPT-4, we also identify limitations of simple supervised fine-tuning for learning optimal playing strategy, suggesting the need for more advanced methodologies for effectively training language models to excel in games. PokerBench thus presents a unique benchmark for a quick and reliable evaluation of the poker-playing ability of LLMs as well as a comprehensive benchmark to study the progress of LLMs in complex game-playing scenarios. The dataset and code will be made available at: url{https://github.com/pokerllm/pokerbench}.
Problem
Research questions and friction points this paper is trying to address.
Large Language Models
Complex Strategy Games
Performance Enhancement
Innovation
Methods, ideas, or system contributions that make the work stand out.
PokerBench
Complex Game Strategy
Model Evaluation
🔎 Similar Papers
No similar papers found.
