Is Grep All You Need? How Agent Harnesses Reshape Agentic Search

2026-05-14

AI Summary

This study addresses the lack of systematic analysis on the interaction between retrieval strategies and agent architectures or tool-invocation paradigms, particularly regarding robustness under irrelevant contextual interference. For the first time, it evaluates the performance of traditional grep-based retrieval against vector retrieval within a unified benchmark (LongMemEval), across multiple LLM agent frameworks—including Chronos, Claude Code, Codex, and Gemini CLI—and under varying tool-output presentation formats. The work further assesses system stability as irrelevant dialogue history is incrementally introduced. Experimental results demonstrate that grep consistently outperforms vector retrieval in accuracy across most configurations, while both agent architecture and tool-invocation paradigm significantly influence overall effectiveness, underscoring the critical importance of their co-design.

Abstract

Recent advances in Large Language Model (LLM) agents have enabled complex agentic workflows where models autonomously retrieve information, call tools, and reason over large corpora to complete tasks on behalf of users. Despite the growing adoption of retrieval-augmented generation (RAG) in agentic search systems, existing literature lacks a systematic comparison of how retrieval strategy choice interacts with agent architecture and tool-calling paradigm. Important practical dimensions, including how tool outputs are presented to the model and how performance changes when searches must cope with more irrelevant surrounding text, remain under-explored in agent loops. This paper reports an empirical study organized into two experiments. Experiment 1 compares grep and vector retrieval on a 116-question sample from LongMemEval, using a custom agent harness (Chronos) and provider-native CLI harnesses (Claude Code, Codex, and Gemini CLI), for both inline tool results and file-based tool results that the model reads separately. Experiment 2 compares grep-only and vector-only retrieval while progressively mixing in additional unrelated conversation history, so that each query is embedded in more distracting material alongside the passages that matter. Across Chronos and the provider CLIs, grep generally yields higher accuracy than vector retrieval in our comparisons in experiment 1; at the same time, overall scores still depend strongly on which harness and tool-calling style is used, even when the underlying conversation data are the same.

Problem

• agentic search
• retrieval strategy
• tool-calling paradigm
• irrelevant context
• agent architecture

Innovation

• grep retrieval
• vector retrieval
• agentic search
• tool-calling paradigm
• retrieval-augmented generation