A/B Agent: A Self-Evolving Agent for Strategy Iteration in Industrial A/B Testing

2026-08-05

AI Summary

Industrial recommendation systems often rely on manual, iterative hyperparameter tuning, leading to fragmented historical experimental knowledge that is difficult to reuse. Existing approaches are limited in retrieval accuracy and cross-scenario transferability due to their neglect of hierarchical scenario structures. To address these challenges, this work proposes a closed-loop, self-evolving A/B testing agent that organizes historical strategies into a hierarchical experience tree. It leverages multi-path Tree-RAG to enable goal-aware strategy generation and continuously refines strategies while updating the knowledge base using online A/B test feedback. Evaluated on a short-video e-commerce recommendation system, the method achieves a significant 4.829% increase in GMV, with all guardrail metrics showing positive trends, demonstrating its effectiveness and robustness.

Abstract

Industrial recommendation strategy iteration heavily relies on large-scale A/B experimentation. Traditional tuning requires experts to repeatedly design strategies, configure experiments, analyze results, and adjust parameters, making the process labor-intensive and time-consuming. Meanwhile, valuable knowledge from historical experiments is often fragmented, making systematic reuse difficult through manual expert effort alone. Existing RAG agents partially alleviate this burden by retrieving prior strategies, but typically organize experience in a flat manner, overlooking the hierarchical relationships among business scenarios, recommendation stages, optimization objectives, and experimental contexts. This often results in mismatched retrieval and limited cross-scenario transfer, while preventing agents from continuously refining strategies and parameters through sequential A/B feedback. % To address these limitations, we propose A/B Agent, a closed-loop A/B agent for industrial recommendation strategy optimization. The framework comprises three tightly coupled core components: Historical Strategy Knowledge Organization, Autonomous Target-Aware Strategy Generation, and Experiment-Guided Strategy Self-Evolution. It organizes historical strategies into a hierarchical experience tree, retrieves transferable evidence through multi-path Tree-RAG to generate executable strategies, and continuously analyzes online A/B feedback to guide autonomous tuning and update the experience tree for self-evolution. Extensive offline and online evaluations demonstrate its effectiveness, including a 4.829% improvement in GMV in a real-world short-video e-commerce recommendation system while maintaining positive gains across all guardrail metrics.

Problem

• A/B testing
• strategy iteration
• recommendation systems
• knowledge organization
• agent-based optimization

Innovation

• A/B Agent
• Tree-RAG
• self-evolution
• hierarchical experience tree
• strategy iteration