CACHEFORGE: LLM-Guided End-to-End Generative Cache Replacement Policy for Performance and Hardware Efficiency

2026-10-05

AI Summary

This study addresses the limitations of traditional cache replacement policies, which rely on fixed heuristic rules and struggle to adapt to complex memory access patterns while being prone to overfitting. We propose a large language model (LLM)-guided closed-loop evolutionary framework that pioneers embedding LLMs within a controlled hardware-in-the-loop to automatically generate and optimize C++ cache replacement logic under hardware-aware constraints. The methodology integrates ChampSim simulation, reward shaping, dynamic mutation, and cross-policy evolutionary algorithms to achieve end-to-end generation of novel mechanisms that transcend fixed predictor architectures. Experimental results demonstrate that the proposed framework significantly improves cache hit rates and instructions per cycle (IPC) on the SPEC CPU2006 benchmarks, comprehensively outperforming LRU and multiple mainstream baseline policies.

Abstract

Modern cache replacement designs saturate because they operate within fixed representational structures, hand-crafted and heuristic based feature-engineered predictors, or offline imitation models that cannot generate new decision logic on their own. At the same time, replacement is shaped by the causal interaction of prefetching, thrashing, spatial locality, and access-type behavior, producing an enormous design space that is difficult to traverse manually. Prior approaches typically rely on heuristics, parameter tuning, or imitation of an offline optimal policy, capturing correlations rather than synthesizing new mechanisms. As a result, their performance gains often plateau and they overfit under dynamic workload conditions. CACHEFORGE is the first framework to evolve cache-replacement policies end-to-end by embedding a large language model inside a governed hardware-aware loop. In each iteration, the LLM proposes new C++ replacement logic, the policy is evaluated under a trace-based CRC-2 ChampSim simulator, and the framework enforces feasibility through reward shaping, structural checks, dynamic mutation, temperature scheduling, and cross-policy crossover. This closed-loop generation-evolution loop specifically designed for cache replacement policy enables the discovery of compact policies that satisfy hardware constraints while exploring algorithmic transformations beyond fixed predictor structures. Across SPEC CPU2006, CACHEFORGE outperforms all CRC-2 baselines. It improves the total hit rate by 27.36%, 19.69%, 13.72%, 13.15%, 11.83%, and 5.73% over MPPPB, ReD, Hawk-eye, SHiP++, LIME, and LRU, respectively. On memory-intensive workloads, it increases IPC by 10.15%, 7.89%, 6.34%, 3.64%, 3.12%, and 2.71% over LRU, MPPPB, LIME, ReD, SHiP++, and Hawkeye.

Problem

• Cache Replacement Policy
• Hardware Efficiency
• Design Space Exploration
• Heuristic Limitations
• Dynamic Workloads

Innovation

• Cache Replacement Policy
• Large Language Model
• End-to-End Generation
• Hardware-aware Evolution
• Closed-loop Optimization