Lit Silicon: A Case Where Thermal Imbalance Couples Concurrent Execution in Multiple GPUs

2025-11-13

AI Summary

Thermal imbalance in GPU clusters causes significant performance fluctuations, severely hindering the efficiency of AI workloads such as large language model (LLM) training. To address this, we identify and formalize the “Lit Silicon Effect”—a previously uncharacterized performance degradation phenomenon arising from the coupling of thermally induced tail latency in multi-GPU nodes with compute-communication overlap mechanisms. Leveraging C³ analysis and joint thermal-power modeling, we design a lightweight detection method and a zero-hardware-cost mitigation strategy. Our key innovation is node-level dynamic power management, which jointly enforces GPU power capping and redistributes constrained power to CPUs. Evaluations on AMD MI300X systems demonstrate up to 6% higher throughput and 4% improved energy efficiency, while substantially reducing datacenter PUE and TCO. This work establishes a deployable, thermal-aware systems optimization paradigm for large-scale AI infrastructure.

Technology Category

Machine Learning: Hardware-aware MLHumans and AI: Other Foundations of Human Computation & AIConstraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphsEconomics, Online Markets and Human Computation: Cost models of using LLMs in production systemsSystems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

GPU systems are increasingly powering modern datacenters at scale. Despite being highly performant, GPU systems suffer from performance variation at the node and cluster levels. Such performance variation significantly impacts both high-performance computing and artificial intelligence workloads, such as cutting-edge large language models (LLMs). We analyze the performance of a single-node multi-GPU system running LLM training, and observe that the kernel-level performance variation is highly correlated with concurrent computation communication (C3), a technique to overlap computation and communication across GPUs for performance gains. We then take a further step to reason that thermally induced straggling coupling with C3 impacts performance variation, coined as the Lit Silicon effect. Lit Silicon describes that in a multi-GPU node, thermal imbalance across GPUs introduces node-level straggler GPUs, which in turn slow down the leader GPUs. Lit Silicon leads to node-level performance variation and inefficiency, impacting the entire datacenter from the bottom up. We propose analytical performance and power models for Lit Silicon, to understand the potential system-level gains. We further design simple detection and mitigation techniques to effectively address the Lit Silicon problem, and evaluate three different power management solutions, including power optimization under GPU thermal design power, performance optimization under node-level GPU power capping, and performance optimization under node-level CPU power sloshing. We conduct experiments on two workloads on two AMD InstinctTM MI300X GPU systems under two LLM training frameworks, and observe up to 6% performance and 4% power improvements, potentially saving hundreds of millions of dollars in datacenters. Our solution is almost free lunch and can be effortlessly adopted in datacenters as a new node-level power management layer.

Problem

• Thermal imbalance causes performance variation in multi-GPU systems
• Concurrent computation communication couples with thermal straggling effects
• Node-level GPU performance inefficiency impacts datacenter efficiency

Innovation

• Modeled GPU thermal imbalance impact on performance
• Designed detection and mitigation techniques for stragglers
• Implemented power management solutions under different constraints