LIMAO: A Framework for Lifelong Modular Learned Query Optimization

2025-06-30

AI Summary

To address catastrophic forgetting and degraded generalization in learned query optimizers (LQOs) under dynamic query workloads—caused by frequent retraining—this paper proposes LIMAO, an engine-agnostic continual learning framework for cost prediction. LIMAO integrates a modular lifelong learning mechanism with an attention-enhanced neural network, jointly optimizing knowledge retention and workload adaptability. Crucially, it operates as a drop-in replacement for existing cost models, requiring no modifications to the underlying query execution engine. Experimental evaluation demonstrates that, under dynamic workloads, LIMAO reduces average query execution time by 40% and execution time variance by 60% compared to baseline LQOs; on certain benchmarks, it achieves up to 4× speedup over PostgreSQL’s native optimizer. These improvements significantly enhance long-term stability and optimization robustness without compromising deployment flexibility.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Life-Long and Continual Learning
• Intelligent Robots: Learning & Optimization for ROB

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Query optimizers are crucial for the performance of database systems. Recently, many learned query optimizers (LQOs) have demonstrated significant performance improvements over traditional optimizers. However, most of them operate under a limited assumption: a static query environment. This limitation prevents them from effectively handling complex, dynamic query environments in real-world scenarios. Extensive retraining can lead to the well-known catastrophic forgetting problem, which reduces the LQO generalizability over time. In this paper, we address this limitation and introduce LIMAO (Lifelong Modular Learned Query Optimizer), a framework for lifelong learning of plan cost prediction that can be seamlessly integrated into existing LQOs. LIMAO leverages a modular lifelong learning technique, an attention-based neural network composition architecture, and an efficient training paradigm designed to retain prior knowledge while continuously adapting to new environments. We implement LIMAO in two LQOs, showing that our approach is agnostic to underlying engines. Experimental results show that LIMAO significantly enhances the performance of LQOs, achieving up to a 40% improvement in query execution time and reducing the variance of execution time by up to 60% under dynamic workloads. By leveraging a precise and self-consistent design, LIMAO effectively mitigates catastrophic forgetting, ensuring stable and reliable plan quality over time. Compared to Postgres, LIMAO achieves up to a 4x speedup on selected benchmarks, highlighting its practical advantages in real-world query optimization.

Problem

• Handling dynamic query environments in learned optimizers
• Preventing catastrophic forgetting in lifelong learning systems
• Improving query execution time and stability

Innovation

• Modular lifelong learning technique for query optimization
• Attention-based neural network composition architecture
• Efficient training paradigm to prevent catastrophic forgetting