Athena: Synergizing Data Prefetching and Off-Chip Prediction via Online Reinforcement Learning

2026-01-24

AI Summary

Existing approaches struggle to effectively coordinate data prefetching with off-chip prediction, and their control strategies often suffer from low efficiency. This work formulates the joint optimization of these two mechanisms as an online reinforcement learning problem for the first time, introducing a unified framework that enables end-to-end adaptive scheduling. By dynamically adjusting the activation policies and aggressiveness levels of individual components based on system-level state monitoring—such as accuracy and bandwidth utilization—the proposed method achieves coordinated control in real time. Experimental results demonstrate that the approach significantly outperforms state-of-the-art solutions across a range of memory-intensive workloads and system configurations, delivering substantial performance gains while maintaining low storage overhead.

Technology Category

• Machine Learning: Hardware-aware ML
• Search and Optimization: Learning to Search
• Planning, Routing, and Scheduling: Planning/Scheduling and Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Prefetching and off-chip prediction are two techniques proposed to hide long memory access latencies in high-performance processors. In this work, we demonstrate that: (1) prefetching and off-chip prediction often provide complementary performance benefits, yet (2) naively combining them often fails to realize their full performance potential, and (3) existing prefetcher control policies leave significant room for performance improvement behind. Our goal is to design a holistic framework that can autonomously learn to coordinate an off-chip predictor with multiple prefetchers employed at various cache levels. To this end, we propose a new technique called Athena, which models the coordination between prefetchers and off-chip predictor (OCP) as a reinforcement learning (RL) problem. Athena acts as the RL agent that observes multiple system-level features (e.g., prefetcher/OCP accuracy, bandwidth usage) over an epoch of program execution, and uses them as state information to select a coordination action (i.e., enabling the prefetcher and/or OCP, and adjusting prefetcher aggressiveness). At the end of every epoch, Athena receives a numerical reward that measures the change in multiple system-level metrics (e.g., number of cycles taken to execute an epoch). Athena uses this reward to autonomously and continuously learn a policy to coordinate prefetchers with OCP. Our extensive evaluation using a diverse set of memory-intensive workloads shows that Athena consistently outperforms prior state-of-the-art coordination policies across a wide range of system configurations with various combinations of underlying prefetchers, OCPs, and main memory bandwidths, while incurring only modest storage overhead. Athena is freely available at https://github.com/CMU-SAFARI/Athena.

Problem

• data prefetching
• off-chip prediction
• memory latency
• prefetcher coordination
• performance optimization

Innovation

• reinforcement learning
• data prefetching
• off-chip prediction
• memory latency hiding
• adaptive coordination