Evidence-in-the-Loop: Trace-Driven Optimization for Customer-Service LLM Agents

2026-07-20

AI Summary

This work addresses the challenge of enhancing response quality in large language models within customer service scenarios, where outputs must adhere to policy constraints and human-handoff boundaries. The authors propose an evidence-based, controllable agent workflow that integrates multi-channel retrieval (BM25 and vector recall), weighted reciprocal rank fusion (RRF), cross-encoder reranking, and a fixed DAG orchestrated via LangGraph. This framework unifies FAQs, business rules, dialogue memory, and clarification states to enable policy-guided evidence reasoning. Key contributions include three reusable patterns: hybrid RAG evidence construction, an evidence-driven mechanism for question/action decision-making, and trajectory-diagnosis-informed reranking optimization—enabling fine-grained attribution and forgetting risk assessment. Deployed in real-world settings, the system delivers auditable, traceable inference, with reranker fine-tuning simultaneously improving domain performance and resistance to catastrophic forgetting.

Abstract

Production customer-service bots must improve answer quality across iterative releases, yet large language models must not bypass evidence boundaries, policy rules, or human-handoff safeguards. We present an \textbf{Evidence-Grounded Customer-Service Agent Workflow} deployed in a real-world customer-service setting. BM25 recall, issue-title-vector recall, issue-description-vector recall, weighted RRF fusion, and cross-encoder reranking construct grounded FAQ evidence for controlled LLM decisions. Policy-guided orchestration then combines this RAG evidence with scenario-specific rule evidence, conversation memory, and clarification state inside a fixed LangGraph DAG~\cite{langgraph2024}. The paper contributes three reusable deployment patterns: \textbf{hybrid RAG evidence construction}, where multi-channel retrieval and reranking produce auditable FAQ candidates; \textbf{evidence-grounded issue/action decision}, where an Evidence-Grounded Decision Module selects an issue/action from typed FAQ evidence and scenario-specific rule evidence; and \textbf{trace-driven RAG and reranker improvement}, where traces diagnose whether failures come from recall, ranking, final candidate selection, clarification, rule-derived evidence, or action policy, and where reranker fine-tuning is evaluated not only for in-domain gain but also for forgetting risk.

Problem

• customer-service LLM agents
• evidence boundaries
• policy compliance
• answer quality improvement
• safeguards

Innovation

• Evidence-Grounded Agent
• Hybrid RAG
• Trace-Driven Optimization
• Policy-Guided Orchestration
• Reranker Fine-Tuning