a/b testing

This study addresses the challenge of biased effect estimation in online controlled experiments caused by overlapping tests on shared traffic, which hinders accurate assessment of feature interactions. To resolve this, the authors propose Multi-Experiment Analysis (MEA), a method grounded in statistical modeling and causal inference that consistently estimates joint effects under arbitrary partial or full overlap and multi-variant settings—without requiring predefined factorial designs or constrained traffic allocation. MEA uniquely enables, without coordination overhead, the simultaneous modeling of bias-corrected individual effects, joint effects for any combination of variants, and conditional effects. Simulations confirm the estimator’s consistency and nominal confidence interval coverage, and the approach has been successfully deployed in large-scale production systems across multiple real-world business applications.

In A/B testing, control variates and regression adjustment are widely used variance reduction techniques, yet their theoretical relationship remains unclear, their methodological frameworks are disjointed, and both have long been confined to design-driven paradigms. Method: This paper establishes, for the first time, a formal equivalence between these two approaches and proposes a novel grouped coefficient estimation method that unifies design-based and model-based estimation frameworks—enabling a paradigm shift from design-driven to model-driven inference. Contribution/Results: Theoretical analysis demonstrates improved estimation accuracy and statistical power. Empirical validation on millions of real-world experiments at ByteDance confirms efficacy: the proposed method has been fully deployed in its online experimentation platform, yielding an average 12.3% increase in statistical significance and a 19.6% improvement in detection sensitivity.

This study addresses the risk of inference bias and potential failure of the CUPED method in online A/B testing under complex experimental conditions. The authors systematically investigate five critical issues related to variance reduction with CUPED, and for the first time delineate its applicability boundaries in designs such as multi-arm experiments and two-stage sampling. To overcome these limitations, they propose a robust variance estimation approach tailored to such settings. Through rigorous theoretical analysis and large-scale empirical validation, the proposed method significantly improves inference accuracy. The solution has been successfully deployed in ByteDance’s experimentation platform, demonstrating its practical effectiveness and scalability.

This work addresses the challenge of safely harnessing large language models (LLMs) in high-throughput experimental optimization, where direct LLM use risks unsafe exploration yet complete exclusion forfeits their optimization potential. To reconcile this trade-off, the authors propose the CARE framework, which employs a non-LLM default optimizer as the primary pathway while leveraging the LLM to generate candidate strategies. Adoption of these candidates is governed by an evidence-based intervention gating mechanism that audits proposals against publicly available evidence, ensuring decisions are auditable, controllable, and traceable. By synergistically integrating LLM-driven creativity with evidence-guided safety constraints, CARE achieves state-of-the-art performance on the Minerva/Olympus and ChemLex benchmarks, improving peak scores from 80.0 to 88.5 and from 83.9 to 92.1, respectively.

Online A/B testing often necessitates interim analyses due to resource constraints; however, frequent “peeking” at accumulating data inflates Type I error rates and compromises conclusion validity. To address this, we propose a Bayesian predictive probability–based framework for safe interim evaluation. Our method is the first to enable efficient, closed-form computation of Bayesian predictive probabilities—bypassing numerical integration—thereby supporting scalable deployment and real-time experiment health monitoring. It rigorously balances statistical validity with engineering practicality. Evaluated on large-scale, real-world A/B tests from Instagram, the system significantly reduces false positive rates while ensuring reliable early stopping decisions. Deployed as a production-grade infrastructure within Meta’s experimentation platform, it enhances both experimental fidelity and resource efficiency across thousands of concurrent experiments.

This work addresses the lack of a general, auditable dynamic control mechanism in existing training systems, which typically rely on framework-specific code. The authors propose the first cross-framework, open-source control plane that exposes training interfaces through a unified protocol, integrating declarative configuration, request validation, and secure control-point scheduling within the Aim workspace to enable metric monitoring, real-time intervention, and operational traceability. The system supports safe human and automated controller interventions during training while fully logging all operational trajectories. Experiments across five NLP and reinforcement learning tasks demonstrate its effectiveness, and the open-source implementation provides a foundation for reproducible human-in-the-loop training.

This work addresses the lack of governance support—particularly for multi-party collaboration, controlled access, and traceable workflows—in existing AI experimentation environments. The authors propose and implement a governance-aware, multi-tenant AI sandbox platform based on a layered reference architecture that decouples presentation, control, execution, and data management layers. Integrated approval workflows and audit logging mechanisms structurally capture experimental context and governance decisions. The platform enables controlled onboarding, cross-project collaboration, and compliance verification, and for the first time facilitates the generation of reusable evaluation evidence, thereby enhancing experiment comparability and auditability. Its effectiveness has been validated in industrial–academic collaborative settings, yielding key insights into deployment and evolutionary practices for such governance-integrated platforms.

This work addresses the limitations of existing evaluation methods, which focus narrowly on task completion and fail to ensure trustworthy deployment of embodied agents in multi-step, externally impactful scenarios, while also lacking coordination among evaluation, governance, orchestration, and runtime assurance. To bridge this gap, the paper proposes an integrated four-layer framework that establishes, for the first time, a closed-loop mechanism linking governance obligations to verifiable execution. Guided by the ODTA principles—Observability, Decidability, Timeliness, and Attestability—the framework introduces runtime localization testing and minimal action evidence bundles. Through a human-in-the-loop evidence synthesis approach, it formally connects policy requirements to concrete agent behaviors, exposing critical gaps such as the inability of static permissions and prompts to govern path-dependent actions. Validation via an enterprise procurement agent demonstrates the framework’s capacity to unify safety, robustness, and trajectory-level evaluation.

This study addresses the challenges of control design in complex industrial processes characterized by multivariable coupled dynamics by proposing an automated control strategy generation framework that integrates large language models (LLMs) with Bayesian optimization. The approach decomposes control design into structured code generation steps, ensuring physical consistency through execution-based validation and feedback-driven repair. It pioneers the automatic synthesis of decentralized PI controller architectures and their tuning environments directly from dynamic process models. Evaluated on a nonlinear gas preheater benchmark, the generated control schemes—subsequently refined via Bayesian optimization—achieve a 26.5% improvement in closed-loop performance and significantly enhance the transient response of pressure loops, thereby demonstrating the method’s effectiveness and novelty.

This work addresses the challenges posed by dynamic AI systems, where models, input distributions, evaluation datasets, and objectives continually evolve, rendering traditional static configurations inadequate. To tackle this, the study proposes integrating tunable elements—such as model selection and prompt structures—into software engineering governance, establishing an engineering paradigm centered on versioned and controlled program spaces. By leveraging statistical methods including sampled evaluation sets, effect size bounds, and confidence/risk thresholds, the framework enables evidence-based mechanisms for updating and deploying system variables. This approach facilitates ongoing validation of relative system effectiveness under shifting conditions, thereby fostering AI engineering practices that are maintainable, verifiable, and capable of sustainable evolution, significantly enhancing system robustness and adaptability.