UniMem: Complementary Episodic-to-Parametric Memory for Boundary-Agnostic Task Streams

2026-07-28

AI Summary

This work addresses the fundamental stability-plasticity dilemma faced by large language model agents in continual task streams lacking explicit task boundaries: external memory retrieval incurs high overhead and struggles to internalize recurring patterns, while parameterized memory is constrained by fixed budgets and relies on known task demarcations. To resolve this, the authors propose UniMem, a framework inspired by human memory mechanisms that employs learnable routing tokens to autonomously coordinate complementary memory pathways—retaining novel or sparse tasks in an episodic buffer for retrieval-augmented execution, while progressively consolidating reliable, repeated patterns into an expandable parameterized memory. UniMem enables on-demand memory expansion without task labels, preventing uncontrolled parameter growth. Experiments demonstrate that UniMem improves average exact match (EM) scores by 4.0 across three backbone models, significantly outperforming baselines while maintaining high execution fidelity.

Abstract

Memory is essential for LLM agents to accumulate task experience and reuse task-specific execution strategies. However, real-world deployment over boundary-agnostic and evolving task streams exposes a fundamental stability-plasticity dilemma. External retrieval-based memory can rapidly absorb new evidence, but it often fails to internalize recurring execution patterns and incurs inference-time retrieval overhead. Parametric memory enables stable and efficient execution once learned, but typically relies on explicit task boundaries and fixed parameter budgets. Inspired by the human brain, which balances plasticity and stability through complementary episodic storage and gradual consolidation, we propose UniMem, a self-routing framework for autonomous memory management. UniMem uses learnable routing tokens as memory controllers, enabling adaptive coordination between complementary memory pathways: novel or sparse tasks are retained in an episodic buffer for retrieval-augmented execution, while recurring and reliable patterns are consolidated into expandable parametric memory. By decoupling task identification from task execution with routing tokens and parametric memory blocks, UniMem expands memory on demand without task labels during deployment or uncontrolled parameter growth. Experiments on long-horizon streaming task sequences show that UniMem consistently outperforms baselines while maintaining execution fidelity, achieving an average gain of 4.0 EM points across three backbone models.

Problem

• stability-plasticity dilemma
• boundary-agnostic task streams
• episodic memory
• parametric memory
• memory consolidation

Innovation

• UniMem
• episodic memory
• parametric memory
• memory routing
• boundary-agnostic learning