🤖 AI Summary
Current AI agents exhibit significant degradation in procedural memory retrieval when encountering novel tasks containing unseen vocabulary, revealing a fundamental bottleneck in cross-instance recognition of functionally equivalent programs.
Method: We introduce the first diagnostic benchmark for procedural memory retrieval in language agents, decoupling memory retrieval from task execution to isolate and rigorously evaluate agents’ understanding of functional equivalence. Leveraging ALFWorld, we construct a dual-source trajectory corpus—comprising expert-annotated and LLM-generated demonstrations—and design a hierarchical query protocol to systematically assess six retrieval methods via controlled ablation studies.
Results: State-of-the-art embedding methods suffer sharp performance drops in novel scenarios, exposing inherent limitations in modeling temporal structure and cross-context generalization. In contrast, LLM-generated program abstractions demonstrate superior generalizability: their transfer stability and scalability with corpus size substantially outperform representation-learning enhancements, highlighting abstraction—not just embedding—as a critical axis for advancing procedural memory.
📝 Abstract
Current AI agents excel in familiar settings, but fail sharply when faced with novel tasks with unseen vocabularies -- a core limitation of procedural memory systems. We present the first benchmark that isolates procedural memory retrieval from task execution, evaluating whether agents can recognize functionally equivalent procedures that span different object instantiations. Using ALFWorld, we construct dual corpora of expert and LLM-generated trajectories and evaluate six retrieval methods using systematically stratified queries. Our results expose a clear generalization cliff: embedding-based methods perform strongly on familiar contexts, yet degrade considerably on novel ones, while LLM-generated procedural abstractions demonstrate reliable cross-context transfer. Controlled ablations show that although embeddings capture some lexical-level abstraction, they fundamentally treat procedures as unordered bags of words, discarding temporal structure necessary for cross-context transfer. Corpus scale delivers far larger gains than representation enrichment, revealing an architectural ceiling in current encoders. Our benchmark offers the first diagnostic framework separating genuine procedural understanding from surface-level memorization and gives tools for developing retrieval systems capable of dependable generalization. Resources available at our GitHub repository (https://github.com/qpiai/Proced_mem_bench).