ContextWeave: A Real-World Workflow Benchmark

2026-08-05

AI Summary

Existing evaluation frameworks struggle to assess the memory capabilities of language agents in long-term, stateful real-world workflows. To address this gap, this work proposes ContextWeave—the first longitudinal benchmark constructed from de-identified office workflows of 14 participants over multiple months, comprising 1,005 executable tasks. Leveraging containerized environments, task trajectory logging, and fine-grained scoring, the framework systematically evaluates the impact of diverse memory mechanisms on agent workflow performance. Experiments demonstrate that an optimized memory configuration improves the Workspace Score from 68.08 to 78.20 and elevates the Preference Score from 41.50 to 70.60. Empirical memory outperforms summarization-based approaches but proves more susceptible to misleading recall. This study is the first to quantitatively measure memory’s contribution to agent efficacy in realistic, extended scenarios, introducing preference alignment and multidimensional diagnostic metrics.

Abstract

Memory is essential as language agents move from isolated tasks to long-horizon, stateful workflows, yet existing evaluations often reduce it to retrieval or question answering. We introduce ContextWeave, a longitudinal benchmark that evaluates whether recalled experience improves downstream agent performance in realistic office-work streams. ContextWeave reconstructs privacy-preserved, multi-month workflows of 14 participants into 1,005 executable tasks, including 568 core evaluation tasks, with instructions, containerized environments, trajectories, and task-specific rubrics. It measures workspace quality and alignment with participant-specific preferences, complemented by diagnostics of relevance, continuity, solvability, and robustness to misleading recall. Across six memory components under a fixed model, the strongest configuration raises Workspace Score from 68.08 to 78.20 and Preference Score from 41.50 to 70.60. With a fixed memory component, recall improves both outcomes for all five tested base models, although gains vary substantially. Our analysis shows that actionable, experience-rich memory supports workflow continuation and reduces redundant exploration more effectively than compact summaries, while it can also be more susceptible to misleading recall. These findings motivate memory systems that optimize not only retrieval relevance but also reliable use during execution.

Problem

• memory
• language agents
• workflow benchmark
• long-horizon tasks
• real-world evaluation

Innovation

• ContextWeave
• memory evaluation
• long-horizon workflows
• language agents
• real-world benchmark