FREESH: Fair, Resource- and Energy-Efficient Scheduling for LLM Serving on Heterogeneous GPUs

2025-11-02

AI Summary

To jointly optimize energy efficiency and fairness for LLM inference services on heterogeneous GPU clusters, this paper proposes FREESH—a novel framework that pioneers the integration of spatiotemporal computational flexibility into multidimensional co-optimization, unifying load routing, parallelism strategy selection, query scheduling, and dynamic GPU frequency scaling. FREESH combines GPU power-throughput characterization, predictable query-length matching, least-laxity-first scheduling, and load-balanced routing to minimize carbon emissions or energy consumption while satisfying SLO latency and fairness constraints. Evaluated on real production workloads, FREESH reduces energy consumption by 28.6% and carbon emissions by 45.45% within one hour, while significantly improving SLA compliance and request-level fairness.

Technology Category

• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The ever-increasing computation and energy demand for LLM and AI agents call for holistic and efficient optimization of LLM serving systems. In practice, heterogeneous GPU clusters can be deployed in a geographically distributed manner, while LLM load also observes diversity in terms of both query traffic and serving patterns. LLM queries running on advanced GPUs during a high-emission hour at one location can lead to significantly higher carbon footprints versus same queries running on mid-level GPUs at a low-emission time and location. By observing LLM serving requirements and leveraging spatiotemporal computation flexibility, we consider the joint routing and scheduling problem, and propose FREESH to cooperatively run a group of data centers while minimizing user-specified carbon or energy objectives. FREESH identifies the optimal configurations of balanced load serving by matching distinct GPU instance's power-throughput characteristics with predictable LLM query length and workloads. To ensure both latency and fairness requirements, FREESH identifies optimized parallelism and query routing schedules together with dynamic GPU frequency scaling for power saving, and Least-Laxity-First (LLF) serving strategy for query scheduling. During the 1-hour serving on production workloads, FREESH reduces energy by 28.6% and emissions by 45.45% together with improvements in SLO attainment and fairness.

Problem

• Optimizing LLM serving for carbon and energy efficiency
• Scheduling queries across heterogeneous distributed GPU clusters
• Balancing latency requirements with fair resource allocation

Innovation

• Leverages spatiotemporal flexibility for carbon optimization
• Matches GPU power characteristics with query workloads
• Combines dynamic frequency scaling with LLF scheduling