Complementary Roles of Activation and Parametric Memory in Few-Shot Learning

2026-09-23

AI Summary

研究通过控制实验探讨了激活记忆和参数记忆在少样本学习中的互补作用，发现两者结合对于解决条件算术任务至关重要。

Abstract

At test time, large language models (LLMs) can encode historical information in activation memory (i.e., KV caches) and parametric memory (i.e., updated parameters). While activation memory is generally considered effective for factual recall and parametric memory for learning new tasks, their interplay remains unclear. In this work, we systematically investigate the role of memory in few-shot learning through controlled experiments. We find that activation memory is superior for recalling facts, whereas parametric memory does not consistently outperform activation memory in task learning. Moreover, our experiments show that the composite task, Conditional Arithmetic, requires the synergy of both memory types. Through neuron-level analysis, we find that the model activates distinct sets of neurons when accessing the same historical information through activation versus parametric memory. When both memory types are combined, the model recruits neurons from both sets, which is crucial for solving Conditional Arithmetic. These findings suggest that neither memory mechanism alone is sufficient for this composite task, highlighting the importance of their collaboration.

Problem

• Few-Shot Learning
• Activation Memory
• Parametric Memory
• Conditional Arithmetic

Innovation

• Activation Memory
• Parametric Memory
• Few-Shot Learning
• Conditional Arithmetic
• Neuron-level Analysis