Lost in the Request: How Communication Variation Disrupts Retrieval and Action in Email Agents

2026-10-01

AI Summary

This study addresses the robustness deficiencies of email assistants, which frequently suffer from retrieval failures and action omissions due to variations in user expression styles or dialects. To this end, we propose a multidimensional communication style variation testing framework that systematically evaluates the reliability of retrieval-augmented generation (RAG) and tool-use agents under stylistic and dialectal perturbations while preserving semantic content through rule-based variant construction. Our findings reveal that verbose requests degrade lexical retrieval performance, whereas indirect and formal requests predominantly cause action omission rather than erroneous execution. Furthermore, dialectal variants lead to task failure even when retrieval succeeds. Accordingly, this work introduces a novel evaluation paradigm that decouples retrieval assessment from action completion measurement, thereby exposing critical limitations in existing benchmarking methodologies.

Abstract

An email assistant should not complete less work simply because a user phrases the same request differently. Yet most benchmarks test each task with only one canonical request, leaving this form of robustness largely unmeasured. We test whether email assistants remain reliable when the requested information, available evidence, and expected outcome stay fixed, but the communication style or English variety changes. We construct validated variants along five communication-style axes and four rule-based dialect conditions, and evaluate them on three benchmarks: a retrieval-augmented generation (RAG) pipeline and two tool-using agents. Indirect requests reduce performance on all three benchmarks, while formal requests reduce performance on both agentic benchmarks. Examining the systems more closely shows that these failures have different causes. Verbose requests mainly hurt a lexical retriever by making the relevant email harder to find. By contrast, indirect and dialect variants remain harmful even when the relevant email is retrieved. In the agentic setting, indirect and formal requests mainly cause the agents to omit required actions, not to take more unsupported actions. These results show that a successful response is not enough to establish robustness: evaluations should vary how requests are expressed and separately measure whether agents complete the requested work.

Problem

• email agents
• communication variation
• robustness evaluation
• retrieval-augmented generation
• tool-using agents

Innovation

• Communication Variation
• Email Agents
• Robustness Evaluation
• Retrieval-Augmented Generation
• Tool-Using Agents