Beyond Memory Leaderboards: Evaluating Scientific Memory as Budgeted Context Restoration

2026-07-18

AI Summary

This work addresses the limitation of existing large model memory evaluations, which predominantly rely on dialogue or summarization and fail to assess scientific agents’ ability to recover evidence from full research papers. The authors propose the first full-text–oriented scientific memory evaluation paradigm, introducing two new benchmarks—PAIM and PTr—that formalize scientific memory as a budget-constrained, modality-aware context recovery task. Their approach combines sparse (BM25) and dense retrieval in a hybrid strategy, incorporating multi-granularity text ingestion, preservation of original content, controlled retrieval budgets, and consistency calibration across multiple evaluators, including humans. Experiments reveal that under strict retrieval budget constraints, the performance advantage of leading systems vanishes; hybrid retrieval substantially improves results; and LLM-based evaluators align closely with human judgments, achieving a resolution of 0.1 points.

Abstract

Long-term memory is becoming a core component of LLM agents, but most memory benchmarks evaluate conversations or compact summaries, while research agents need to restore evidence from full scientific papers. We introduce two full-text scientific-memory benchmarks, Public AI Memory (PAIM; 81 papers, 66 questions) and Public Transformers (PTr; 252 papers, 98 questions). We evaluate eight memory/retrieval systems, including our own proposed Theoria, plus a no-retrieval baseline. Our results show that memory leaderboards are not interpretable without the full protocol: ingestion granularity, raw-text preservation, retrieval budget, retrieval modality, rubric audit, and judge choice all affect the outcome. For example, on PAIM Graphiti wins convincingly but uses 2.6M characters of retrieved context per query, and after controlling for retrieval budget the lead disappears. On PTr, for the systems where BM25 retrieval can be added cleanly, the sparse-dense hybrid is the single most significant intervention: hybrid variants of Simple RAG, Mem0, and Theoria tie for the lead within 0.03 points. Multi-judge and human side-by-side calibration show that LLM-as-a-judge rankings are consistent across frontier judges and agree with human evaluation, with an effective resolution of roughly one point on a ten-point scale. We argue that scientific memory should be evaluated as budgeted, modality-aware context restoration rather than as an unconstrained architecture leaderboard, and we release the datasets, harness, raw outputs, judgments, and scripts to reproduce our results and serve as tools for such evaluation. Our code is available at http://gitlab.com/quantellence/research/scientific-recall-bench , and the datasets are available at http://huggingface.co/datasets/quantellence/srb-data .

Problem

• scientific memory
• context restoration
• retrieval budget
• memory evaluation
• long-term memory

Innovation

• scientific memory
• budgeted retrieval
• context restoration
• hybrid sparse-dense retrieval
• memory benchmarking