Screen Before You Serve: Simulation for Production Customer Experience AI Agents at 140M Scale

2026-09-24

AI Summary

This study addresses the insufficient test coverage for customer service AI agents in regulated industries and the risk that online experimentation poses to user trust. To overcome these challenges, this work proposes a hypothesis-driven simulation evaluation workflow. By leveraging the Snowglobe simulator and synthetic data generation techniques, the method enables multi-turn interaction simulation without invoking production backends. Furthermore, end-to-end binary evaluators are employed to pre-screen candidate agents for safety, facilitating large-scale configuration exploration with zero user risk. Experimental results demonstrate that simulated scores align closely with production environments. In A/B testing, the proposed approach yields a 36.69-point improvement in tNPS and an 8.82% increase in self-service rates, thereby achieving safe, iterative development of AI agents within regulated settings.

Abstract

Customer experience (CX) agents use tools and large language models to address customer requests and guide conversational interactions with an organization's products. Improving these agents, especially in regulated industries, is difficult: they must detect intent, follow complex operational policies and use tools reliably. Manual end-to-end testing offers limited coverage, while live experiments expose customers to failures that can erode trust. We present a hypothesis-driven simulation workflow for screening candidate CX agents before deployment. Synthetic customers react to agent responses and simulated tool outputs enable multi-step agentic workflows without invoking production backends. We use the Snowglobe simulator on Nubank's Card Delivery agent and its expanded successor, Card Management - Nubank's highest-volume chat-support agent in Brazil. Across 4 deployed versions, simulated and production version-level binary evaluator scores show high correlation. Simulation-guided iteration increased transactional net promoter score (tNPS) by 36.69 points in a live A/B test. We also screened open-weight configurations in over 16,000 simulated conversations. In a subsequent live A/B test, the selected model increased self-service rate (SSR) by 8.82 percentage points to the highest level observed at Nubank, with no statistically significant change in tNPS. Simulation made broad exploration of models, reasoning settings, and prompts feasible without customer exposure, enabling production improvements that would have been impractical to pursue through live experimentation alone.

Problem

• Customer Experience AI Agents
• Simulation-based Screening
• Regulated Industries
• Pre-deployment Evaluation

Innovation

• Simulation-based Screening
• Customer Experience AI Agents
• Synthetic Customers
• Agentic Workflows
• Large Language Models