🤖 AI Summary
To address insufficient multi-objective load balancing in stream processing systems under complex workloads, this paper proposes a multi-tier collaborative scheduling framework. The framework introduces dynamic inter-layer coordination mechanisms and lightweight interfaces among schedulers, enabling seamless integration of novel scheduling policies. It jointly optimizes computational resource utilization, end-to-end latency, and throughput by integrating multi-objective optimization, distributed resource management, and real-time feedback control. Its key innovation lies in shifting hierarchical scheduling from static decoupling to dynamic collaboration—preserving scalability while significantly enhancing adaptability. Evaluated in Meta’s production environment, the system reliably processes TB-scale data with sub-second latency; it improves critical resource utilization by 27% and reduces tail latency by 41%.
📝 Abstract
Stream processing is a computing paradigm that supports real-time data processing for a wide variety of applications. At Meta, it's used across the company for various tasks such as deriving product insights, providing and improving user services, and enabling AI at scale for our ever-growing user base. Meta's current stream processing framework supports processing TerraBytes(TBs) of data in mere seconds. This is enabled by our efficient schedulers and multi-layered infrastructure, which allocate workloads across various compute resources, working together in hierarchies across various parts of the infrastructure. But with the ever growing complexity of applications, and user needs, areas of the infrastructure that previously required minimal load balancing, now must be made more robust and proactive to application load. In our work we explore how to build and design such a system that focuses on load balancing over key compute resources and properties of these applications. We also showcase how to integrate new schedulers into the hierarchy of the existing ones, allowing multiple schedulers to work together and perform load balancing, at their infrastructure level, effectively.