batching strategies

To address severe contention and scalability bottlenecks caused by fetch-and-add operations on a single memory location under high concurrency, this paper proposes Aggregating Funnels—a novel mechanism that distributes atomic operations across multiple memory locations to enable cross-location batch aggregation and decoupled result computation. Our approach leverages dual-location coordinated batching, lock-free concurrency control, and fine-grained memory layout optimization, building an efficient aggregation path directly atop hardware-supported fetch-and-add instructions. Unlike conventional single-point or combining funnels, Aggregating Funnels overcomes fundamental scalability limits inherent in prior designs. Experimental evaluation demonstrates significantly higher throughput compared to state-of-the-art Combining Funnels. When integrated into mainstream concurrent queues, it delivers substantial end-to-end performance improvements by eliminating critical serialization bottlenecks.

To address CPU bottlenecks in data preprocessing, imbalanced resource allocation, and insufficient fault tolerance during ML training and inference on CPU-GPU heterogeneous systems, this paper proposes a Streaming-Batch hybrid execution model. It synergistically integrates the low-latency benefits of streaming execution with the high-throughput advantages of batched processing, enabling shard-wise incremental execution, lineage-based lightweight fault recovery, and dynamic heterogeneous resource scheduling—thereby eliminating reliance on homogeneous hardware. Implemented atop Ray Data, the model achieves 3–8× higher throughput for heterogeneous batch inference. In Stable Diffusion training, it improves end-to-end training throughput by 31% while preserving per-node data loading throughput.

This work addresses the lack of a unified and comparable benchmark for fairly evaluating rule-based, learning-based, and large language model (LLM)-driven autoscaling strategies in big data batch processing scenarios. To this end, we propose BatchBench, an open-source, workload-aware benchmarking framework. BatchBench introduces a taxonomy encompassing six representative batch workload types, features a parameterized workload generator whose fidelity is validated via two-sample Kolmogorov–Smirnov tests and Earth Mover’s Distance, and defines a five-dimensional evaluation protocol covering cost, SLA compliance, responsiveness, scaling jitter, and interpretability. Notably, it enables, for the first time, side-by-side comparison of all three autoscaling strategy categories while incorporating LLM inference cost accounting. The framework’s design is complete, and its reference implementation will be open-sourced to establish a standardized experimental foundation for autoscaling research.

This work addresses the high overhead and latency in machine learning training caused by numerous independent GET requests during data loading. The authors propose GetBatch, the first system to treat batched object retrieval as a first-class primitive in storage systems. By replacing multiple independent requests with a single, deterministic, fault-tolerant streaming execution, GetBatch enables efficient and low-latency multi-object fetching. The approach integrates distributed object storage, request aggregation, and streaming transfer, achieving up to 15× higher throughput for small objects, a 2× reduction in P95 batch latency in production environments, and a 3.7× reduction in P99 tail latency.

Existing inference engines struggle with extreme inter- and intra-sequence load imbalances in large-batch processing, resulting in low GPU utilization and reduced throughput. This work proposes a sequence coroutine computation model, representing each sequence as a fine-grained, event-driven coroutine that enables dynamic task reorganization at runtime. Introducing the sequence coroutine abstraction for the first time, the model achieves dynamic load balancing, expert-level large-batch scheduling, and cross-device task redistribution while maintaining high resource utilization even on memory-constrained devices. Built upon this model, the resulting cluster-scale batch inference system reduces end-to-end job completion time by up to 2.3× on a 128-GPU cluster and delivers up to 9.6× higher performance over the strongest offloading baseline on memory-limited accelerators.

This work addresses the challenge of achieving both high accuracy and scalability in differentially private query answering over high-dimensional data by proposing QuerySmasher. The method introduces a divide-and-conquer strategy into the matrix mechanism for the first time, decomposing complex query workloads into mutually orthogonal low-dimensional sub-workloads, which are solved independently and then fused to produce the final result. Theoretical analysis demonstrates that QuerySmasher strictly outperforms existing approaches—including ResidualPlanner, RP+, and WFF—in terms of total squared error. Experimental results further confirm that QuerySmasher significantly improves query accuracy while preserving unbiasedness and exhibits strong scalability across varying data dimensions.

Coordinating coarse-grained parallelism (e.g., tensor/pipeline/model parallelism) with fine-grained operator-level sharding dimensions in distributed LLM inference remains challenging, especially for meeting stringent service-level objectives (SLOs). Method: This paper proposes the first reinforcement learning–based joint optimization framework. It employs an attention mechanism to model high-performing historical strategies and integrates elite experience replay to efficiently navigate an ultra-large configuration space—overcoming limitations of static, heuristic-based approaches. Contribution/Results: The framework enables fully automated parallelization policy generation for ultra-large models—including MoE architectures up to 1.6 trillion parameters—on H100 GPU clusters. Experiments demonstrate throughput improvements of 3.5× over meta-heuristic baselines and 1.06× over Megatron’s heuristic policies, significantly enhancing inference performance and SLO compliance under complex hardware topologies.

This work addresses the critical limitation of existing serverless federated learning systems, which struggle to train large models due to strict memory constraints imposed by serverless functions. To overcome this barrier, the authors propose GradsSharding, a novel approach that shards gradient tensors and processes them in parallel across serverless functions, with each function aggregating only its assigned shard. This method achieves mathematically equivalent results to conventional tree-based aggregation while attaining constant memory consumption per function—decoupled from the number of participating clients—for the first time. Implemented on AWS Lambda, the sharded FedAvg algorithm demonstrates effectiveness across model sizes ranging from 43 MB to 5 GB, reduces training costs by 2.7× on VGG-16, and stands as the only serverless federated learning solution capable of surpassing the 10 GB memory ceiling.

In structured stochastic optimization, there exists a fundamental trade-off between expensive full-batch queries (e.g., gradients or matrix-vector products) and cheaper sample-based queries (e.g., stochastic function evaluations, row accesses, or generative model calls). Method: We propose the “pseudo-independent algorithms” framework—a generalization of pseudo-determinism—that systematically reuses random samples across iterative subproblems to reduce their variance. Our approach unifies finite-sum minimization, principal eigenvector computation, and MDP policy optimization by sharing randomness across subproblems while preserving convergence guarantees. Contribution/Results: This work introduces the first generic variance-reduction mechanism applicable to diverse stochastic optimization algorithms. It achieves optimal or near-optimal query complexity for gradient evaluations, matrix-vector products, and generative model calls—significantly reducing total query counts and improving practical computational efficiency without sacrificing theoretical rigor.

This work addresses the inefficiency and suboptimal energy consumption of data aggregation operations on heterogeneous hardware platforms. To bridge this gap, the authors propose a hybrid hardware acceleration framework that synergistically combines unified abstractions with platform-specific optimizations, effectively balancing programmability, portability, and architectural specialization across CPUs, GPUs, and FPGAs. By introducing a common abstraction layer while incorporating tailored optimization strategies for each hardware target, the approach achieves significant improvements in both performance and energy efficiency across all three mainstream architectures. The evaluation demonstrates consistent gains not only in device-level computation but also in end-to-end processing metrics, thereby establishing an effective trade-off between generality and high performance for data aggregation workloads.