🤖 AI Summary
To address low resource scheduling efficiency, suboptimal hardware utilization, and high operational costs in cloud and grid environments, this paper proposes GlideinBenchmark—a novel system that extends the pilot-based architecture of GlideinWMS for automated resource discovery and fine-grained performance benchmarking. It supports user-defined metrics and enables data-driven resource matching. Built as a web application, GlideinBenchmark integrates distributed task scheduling with the HEPCloud decision engine to perform lightweight, scalable benchmarking across heterogeneous cloud and grid resources. Experimental evaluation demonstrates that our approach significantly improves resource matching accuracy, reduces average job completion time by 19.3%, and lowers computational cost by 22.7%. The system provides a reusable, intelligent scheduling infrastructure for high-performance computing environments, advancing adaptive and cost-efficient resource orchestration in hybrid infrastructures.
📝 Abstract
Choosing the right resource can speed up job completion, better utilize the available hardware, and visibly reduce costs, especially when renting computers in the cloud. This was demonstrated in earlier studies on HEPCloud. However, the benchmarking of the resources proved to be a laborious and time-consuming process. This paper presents GlideinBenchmark, a new Web application leveraging the pilot infrastructure of GlideinWMS to benchmark resources, and it shows how to use the data collected and published by GlideinBenchmark to automate the optimal selection of resources. An experiment can select the benchmark or the set of benchmarks that most closely evaluate the performance of its workflows. GlideinBenchmark, with the help of the GlideinWMS Factory, controls the benchmark execution. Finally, a scheduler like HEPCloud's Decision Engine can use the results to optimize resource provisioning.