Intelligent resource prediction for SAP HANA continuous integration build workloads

2026-04-14

AI Summary

This work addresses the inefficiencies in continuous integration (CI) systems caused by static resource allocation, which often leads to memory over-provisioning, low cluster utilization, and high operational costs. Leveraging a dataset of 300,000 historical builds, the authors propose an ensemble quantile regression model that integrates LightGBM and XGBoost to accurately predict per-task memory requirements. The predicted demands are then used to dynamically optimize resource allocation via a microservice-based framework. Evaluated under stringent reliability constraints—maintaining a memory under-allocation rate below 0.3%—the approach achieves an average memory saving of 36 GB per build without compromising build execution time, thereby substantially improving overall resource efficiency in CI environments.

Technology Category

• Machine Learning: Hardware-aware ML
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories
• Cognitive Modeling & Cognitive Systems: Computational Creativity

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Virtualization and resource management in Web systems and infrastructures
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Large enterprises often operate extensive Continuous Integration (CI) pipelines on large, heterogeneous compute clusters, where conservative, statically defined resource requirements are used to ensure build reliability. This practice leads to substantial system memory over-allocation, reduced cluster utilization, and increased operational costs. In this paper, we motivate the need for intelligent resource prediction by analyzing over 300,000 historical build executions from a production CI environment with more than one thousand compute nodes. Our analysis shows that, on average, more than 60% of allocated system memory remains unused. We then compare multiple machine learning approaches for predicting build task memory usage, including classification-based methods and regression-based quantile prediction. Our final solution employs a LightGBM-XGBoost quantile regression ensemble optimized to minimize under-allocation while reducing over-provisioning. We integrate this solution into the production CI pipeline via a microservice-based orchestration layer, achieving average memory savings of approximately 36GB per build and reducing under-allocation rates to below 0.3% without negatively impacting build execution times.

Problem

• resource prediction
• continuous integration
• memory over-allocation
• build workloads
• cluster utilization

Innovation

• intelligent resource prediction
• quantile regression ensemble
• CI memory optimization
• LightGBM-XGBoost
• microservice-based orchestration