workload generation

Cloud analytics system evaluation commonly relies on static benchmarks (e.g., TPC-H/TPC-DS), which fail to capture key statistical characteristics of real production workloads—such as performance metric distributions, operator frequencies, and temporal query patterns. Meanwhile, existing real-world execution traces lack reproducible SQL queries and database metadata. Method: We formulate the novel problem of *statistically grounded synthetic workload generation*, introducing three core techniques: (1) multi-objective optimization–driven component selection, (2) progressive timestamp modeling, and (3) LLM-enhanced statistical fidelity augmentation—all operating via recombination of benchmark queries and database objects. Contribution/Results: Evaluated on real cloud traces, our approach reduces statistical approximation error by up to 6× over state-of-the-art methods, significantly improving evaluation authenticity, reproducibility, and ecosystem compatibility.

Existing TPC-style benchmarks (e.g., TPC-H/DS) fail to capture two critical characteristics of real-world cloud data warehouse workloads—query repetitiveness and string-intensive operations—leading to inaccurate system evaluation. To address this, we propose Redbench, a novel synthetic benchmark generation framework grounded in empirical cloud workload trace analysis. Rather than relying on superficial execution metrics, Redbench models intrinsic workload signals—including query pattern distributions, repetition cycles, and string-operation intensity—via integrated techniques for query pattern extraction, repetitiveness modeling, and string-aware enhancement. It enables end-to-end, reproducible, and customizable synthesis of realistic workloads from production traces. Experimental evaluation demonstrates that Redbench-generated workloads significantly improve fidelity and effectively expose performance disparities across four major commercial cloud data warehouses under diverse optimization strategies, thereby providing a high-fidelity, reproducible foundation for rigorous system assessment and optimization.

This work addresses the inefficiency in notebook-based distributed workflows, where minor modifications often trigger full re-execution, severely hindering iterative development and reproducibility. To overcome this limitation, the authors propose NBRewind, a system that, for the first time, enables fine-grained incremental execution and cross-platform portability while preserving reproducibility. NBRewind integrates a dual-kernel architecture—comprising auditing and replay components—with cell-level incremental checkpoints and inter-cell dataflow analysis. It further leverages standardized notebook packaging to facilitate efficient partial re-execution. Evaluation in real-world high-performance computing (HPC) scenarios demonstrates that NBRewind incurs minimal overhead for incremental checkpointing and substantially improves both execution efficiency and cross-site reproducibility.

Existing benchmarks (e.g., TPC-H, TPC-DS) fail to capture distributional shifts and dynamic evolution characteristic of real production workloads, hindering the development and evaluation of learned database components. This paper introduces Redbench—the first high-fidelity benchmark built from 30 real-world cloud service production query workloads. Through systematic collection, statistical modeling, and pattern categorization, Redbench is the first to faithfully align with and reproduce the observed query distribution characteristics and temporal evolution patterns in Redset. It incorporates a workload alignment mechanism that ensures fidelity across critical dimensions: distributional shift, hotspot drift, and long-tail structure. Redbench supports reproducible query sampling and comprehensive workload feature analysis, significantly improving training effectiveness and generalization capability of learned components—such as indexes and query optimizers—in realistic deployment scenarios.

Existing approaches struggle to effectively quantify the similarity and quality between synthetic and real data in evaluating tool-augmented agents. To address this gap, this work proposes SynAE, a novel framework that establishes the first multi-axis evaluation system tailored for multi-turn tool-use scenarios. SynAE introduces four fine-grained metric categories—assessing task instructions, tool invocations, final outputs, and downstream evaluation performance—to systematically measure synthetic data across dimensions of validity, fidelity, and diversity. Integrating natural language processing, trajectory modeling, and controllable generation techniques, the framework enables a reproducible evaluation pipeline and successfully identifies several representative failure modes in synthetic data generation. Empirical results demonstrate that such multidimensional assessment is essential for enhancing the reliability of agent evaluations.

This work addresses the lack of a unified and comparable benchmark for fairly evaluating rule-based, learning-based, and large language model (LLM)-driven autoscaling strategies in big data batch processing scenarios. To this end, we propose BatchBench, an open-source, workload-aware benchmarking framework. BatchBench introduces a taxonomy encompassing six representative batch workload types, features a parameterized workload generator whose fidelity is validated via two-sample Kolmogorov–Smirnov tests and Earth Mover’s Distance, and defines a five-dimensional evaluation protocol covering cost, SLA compliance, responsiveness, scaling jitter, and interpretability. Notably, it enables, for the first time, side-by-side comparison of all three autoscaling strategy categories while incorporating LLM inference cost accounting. The framework’s design is complete, and its reference implementation will be open-sourced to establish a standardized experimental foundation for autoscaling research.

Current evaluations of agent tool use often conflate workload specifications, action generation, and evidentiary criteria, lacking a unified and auditable framework. This work proposes an evaluation paradigm centered on “evidence admissibility gating,” which explicitly decouples workloads, drivers, and verification evidence through a shared evidence admissibility contract. The framework integrates diverse environments—including WebArena Verified, a subset of SWE-Gym, and MiniWoB++—and employs a standardized reporting pipeline comprising a universal workload adapter, declarative drivers, task manifests, event schemas, and replay/freeze strategies. It uniformly logs multidimensional metrics such as latency, invalid actions, and patching costs, enabling consistent differentiation of controller performance under identical workloads while ensuring relevance, reproducibility, and auditability in agent evaluations.

This work addresses the reproducibility challenges in computational notebooks caused by implicit dependencies introduced through out-of-order cell execution, which leads to interactive results inconsistent with sequential execution. To ensure strong reproducibility without requiring precise dependency tracking, the authors propose a dynamic analysis technique that captures read and write sets at cell boundaries and verifies whether the current output can be reproduced by re-executing cells sequentially from an empty state. This approach circumvents the traditional trade-offs among expressiveness, precision, and performance inherent in static or fine-grained dependency analyses. The implemented FlowBook system incurs only a median overhead of 70 milliseconds, significantly enhancing the reliability and reproducibility of notebook computations.

This work addresses the limitation of existing agent evaluation benchmarks, which fail to capture the authentic interaction dynamics between developers and AI agents and inadequately reflect the distribution, diversity, and complexity of real-world tasks. To bridge this gap, the authors introduce a dynamic benchmark constructed from a large-scale corpus of real OpenClaw developer sessions. By leveraging conversation reconstruction, execution environment snapshots, and a deterministic validation mechanism, raw user requests are transformed into reproducible, automatically evaluable tasks. Sampling bias is mitigated through Jensen–Shannon divergence–based control to preserve the true task distribution. The resulting benchmark comprises 281 executable tasks; evaluations of 14 leading models reveal that even the best-performing system solves only 65.8% of them, highlighting a significant capability gap in current agents when operating in realistic software development scenarios.