Bridging Inference-Time Scaling and Episodic Memory with Action-Centric Graphs

2026-07-29

AI Summary

This work addresses the prevalent issues of statelessness and redundant search in existing test-time scaling methods for autonomous agents, as well as the high computational overhead of conventional memory mechanisms that rely heavily on large language model (LLM) inference. To overcome these limitations, the authors propose GAMER, a novel framework that uniquely integrates test-time scaling with episodic memory. GAMER dynamically stores historical reasoning trajectories in an action-centric graph and employs a dual-stream temporal difference learning mechanism to separately evaluate the positive and negative utilities of action nodes, enabling bidirectionally guided, efficient graph search. By decoupling the memory mechanism from the LLM, GAMER substantially reduces both context length and computational cost. Experiments demonstrate that GAMER achieves a 20.81% improvement in success rate and a 6.17% gain in task progression across multiple benchmarks, significantly outperforming current baselines.

Abstract

Recent advancements in inference-time scaling have significantly unlocked the complex reasoning capabilities of Large Language Models~(LLMs). However, for agents, these approaches suffer from a critical inefficiency, operating in a stateless manner and engaging in redundant search processes. Existing memory mechanisms largely rely on the reasoning capabilities of LLMs, leading to prohibitive computational costs. In this paper, we propose a novel framework, \textit{GAMER}~(Graph-based Action-centric Memory with Episodic Reasoning), that bridges the gap between inference scaling and episodic memory. Our approach models historical reasoning as a dynamic \textit{Action-Centric Graph}. By decoupling the memory mechanism from LLMs, our method can save token/money usage by providing less memory context than memory mechanism baselines. To extract knowledge from the graph effectively, we use a dual-stream Temporal Difference learning mechanism to estimate the positive~(suggestion) and negative~(avoidance) value of action nodes based on past successes and failures. During the inference phase, this learned value function optimizes decision-making bi-directionally, so that positive values provide action suggestions, while negative values indicate high-risk actions. By performing efficient searches on the graph, our method significantly improves the efficiency of inference scaling. Experiments on multiple benchmarks demonstrate that \textit{GAMER} achieves superior performance by \textbf{20.81\%/6.17\%} for success/progress rate compared to vanilla baselines.

Problem

• inference-time scaling
• episodic memory
• Large Language Models
• computational efficiency
• redundant search

Innovation

• Action-Centric Graph
• Inference-Time Scaling
• Episodic Memory
• Temporal Difference Learning
• Memory Decoupling