Evolution of Buffer Management in Database Systems: From Classical Algorithms to Machine Learning and Disaggregated Memory

2025-12-28

AI Summary

The widening gap between CPU and memory access latency poses a fundamental challenge to database performance. Method: This paper systematically surveys four decades of buffer management evolution, analyzing classical algorithms (e.g., LRU-K, 2Q, LIRS, ARC), ML-enhanced approaches, and memory-disaggregated architectures (NVM-aware hierarchies, RDMA-decoupled designs). It introduces the first holistic, time-spanning analytical framework for buffer management evolution and proposes a novel cross-layer adaptive paradigm integrating ML-driven policy learning with eBPF-based kernel extensibility. Contribution/Results: Grounded in empirical analysis of 50+ top-tier conference papers and industrial systems (Linux, PostgreSQL, Oracle), the work distills core design trade-offs and identifies key challenges—including cache coherence across heterogeneous memory tiers, low-overhead decision-making, and OS–DBMS co-design—along with concrete research directions toward scalable, adaptive buffer management.

Technology Category

• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems
• Machine Learning: Hardware-aware ML
• Search and Optimization: Distributed Search

Application Category

• Security and Privacy: Large-scale security measurements
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications

Abstract

Buffer management remains a critical component of database and operating system performance, serving as the primary mechanism for bridging the persistent latency gap between CPU processing speeds and storage access times. This paper provides a comprehensive survey of buffer management evolution spanning four decades of research. We systematically analyze the progression from foundational algorithms like LRU-K, 2Q, LIRS, and ARC to contemporary machine learning-augmented policies and disaggregated memory architectures. Our survey examines the historical OS-DBMS architectural divergence, production system implementations in PostgreSQL, Oracle, and Linux, and emerging trends including eBPF-based kernel extensibility, NVM-aware tiering strategies, and RDMA-enabled memory disaggregation. Through analysis of over 50 seminal papers from leading conferences (SIGMOD, VLDB, OSDI, FAST), we identify key architectural patterns, performance trade-offs, and open research challenges. We conclude by outlining a research direction that integrates machine learning with kernel extensibility mechanisms to enable adaptive, cross-layer buffer management for heterogeneous memory hierarchies in modern database systems.

Problem

• Surveying evolution of buffer management algorithms over four decades
• Analyzing progression from classical algorithms to machine learning policies
• Identifying research challenges in adaptive buffer management for modern systems

Innovation

• Machine learning-augmented buffer management policies
• Disaggregated memory architectures with RDMA integration
• Kernel extensibility via eBPF for adaptive control