Problem
Research questions and friction points this paper is trying to address.
LLM-based agents
opponent-aware planning
Pokémon Battles
knowledge augmented generation
Porimon: An LLM-Based Pok\'emon Battle Agent Enhanced by Long/Short-Term Knowledge Augmented Generation
🤖 AI Summary
This study addresses the lack of opponent-awareness in large language model (LLM) agents during adversarial planning by proposing LSTKAG, a Long-Short-Term Knowledge-Augmented Generation framework. By integrating short-term state tracking with long-term experience retrieval and incorporating external APIs, this work constructs Porimon, an agent that operates without fine-tuning. Evaluated across 15,000 Pokémon tournament-style battles, Porimon significantly outperforms both PokéLLMon and rule-based baselines. These results validate the effectiveness of external information retrieval mechanisms in enhancing opponent-aware planning capabilities under zero fine-tuning conditions.
📝 Abstract
In this paper, we use Pok\'emon Battles as a case study to investigate how to improve the performance of LLM-based agents in tasks that require opponent-aware planning without additional fine-tuning. We propose Long/Short-Term Knowledge Augmented Generation (LSTKAG), a mechanism that enables LLM-based agents to leverage past states of the current task and retrieve experience summaries from similar previous task instances based on the current state. Based on LSTKAG, we design Porimon, an LLM-based agent structure for Pok\'emon Battles. For optimization, we introduce an external API for precise damage calculation and more detailed information about the game. We conduct tournament-like evaluation experiments comprising 15,000 battles for hyperparameter optimization, ablation studies, and performance evaluation. The results indicate that Porimon-based players with hyperparameter optimization significantly outperform players based on Pok\'eLLMon, an LLM-based agent structure proposed in previous research, and the rule-based heuristic player. Furthermore, our ablation study shows that Porimon variants outperform the one without extension in game information retrieval, which shows the contribution of that extension. However, the current experiment results are inconclusive regarding the contribution of Long-Term KAG. These results suggest that introducing external resources, information from previous states of the current task, and experience summaries from similar previous task instances could elevate the performance of LLM-based agents designed for tasks requiring opponent-aware planning.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Long/Short-Term Knowledge Augmented Generation
LLM-based Agent
Pokémon Battle
External API Integration
Opponent-aware Planning
🔎 Similar Papers
No similar papers found.
