Building Customer Support AI Agents at 100M-User Scale: An Evaluation-Driven Framework

2026-06-07

AI Summary

This work addresses the deployment blind spots commonly encountered in developing production-grade AI customer service agents for hundreds of millions of users, which often stem from fragmented workflows across evaluation, context engineering, training, and online metrics. To bridge this gap, the authors propose the first unified development framework centered on evaluation-driven design. The framework integrates structured context engineering, human-in-the-loop prompt iteration, an LLM-based judging mechanism with consistency guarantees (including GEPA optimization), and an end-to-end validation pipeline, achieving strong alignment between offline metrics and online performance. Evaluated across five real-world customer service scenarios, the approach substantially enhances user experience—evidenced by a 37-percentage-point increase in AI Net Promoter Score and a 29-percentage-point rise in self-service rate in the card delivery scenario—with most scenarios approaching expert human-level performance.

Abstract

The rapid rise in LLM capabilities has made AI agents increasingly viable across a broad range of tasks. Among the most promising applications is building production-ready customer-facing agents, a challenge that demands coordinated excellence in evaluation methodology, context engineering, training, and online measurement. Yet these critical pillars are typically developed in isolation, creating blind spots that only surface after deployment. In this paper, we present a unified framework that bridges offline development with online impact for customer support AI agents at Nubank, a company with 100M+ users. Our approach integrates several key components: (1) structured context engineering tailored to customer support agents, (2) systematic human-in-the-loop prompt iteration, (3) rigorous LLM judge evaluation with measured inter-rater agreement and GEPA optimization for consistency, and (4) ideation-to-production validation. A central insight is that evaluation-pipeline quality directly determines iteration velocity. We present results from five production deployments spanning distinct domains: card delivery, debt management, credit-limit support, card management, and product explanation. These deployments deliver consistent customer-satisfaction gains while substantially accelerating iteration. In our card-delivery deployment, large-scale A/B testing yields a 37 percentage-point improvement in AI transactional Net Promoter Score and a 29 percentage-point gain in self-service rate over prior agent variants, alongside a strong correlation between offline simulation metrics and online outcomes, demonstrating that eval-driven development reliably predicts production impact. On most use cases, AI satisfaction reaches within a few percentage points of expert human agents.

Problem

• AI agents
• customer support
• evaluation framework
• large-scale deployment
• LLM evaluation

Innovation

• evaluation-driven development
• context engineering
• human-in-the-loop prompting
• LLM judge evaluation
• online-offline alignment