Query Optimization and Evaluation via Information Theory: A Tutorial

2026-04-06

AI Summary

This work addresses the problem of automatically selecting optimal execution plans for general conjunctive queries (CQs) based on database statistics. It proposes the PANDA framework, which, for the first time, directly integrates information-theoretically derived tight upper bounds on intermediate relation cardinalities into the query plan generation and optimization process. The approach unifies treatment across diverse query scenarios and matches or surpasses the performance of specialized algorithms on several classical problems—including those relying on fast matrix multiplication—demonstrating both generality and efficiency. The key contribution lies in establishing a novel connection among information theory, constraint satisfaction problems, and database query optimization, thereby achieving a synergistic improvement between theoretical guarantees and practical performance.

Abstract

Database theory is exciting because it studies highly general and practically useful abstractions. Conjunctive query (CQ) evaluation is a prime example: it simultaneously generalizes graph pattern matching, constraint satisfaction, and statistical inference, among others. This generality is both the strength and the central challenge of the field. The query optimization and evaluation problem is fundamentally a "meta-algorithm" problem: given a query $Q$ and statistics $\cal S$ about the input database, how should one best answer $Q$? Because the problem is so general, it is often impossible for such a meta-algorithm to match the runtimes of specialized algorithms designed for a fixed query -- or so it seemed. The past fifteen years have witnessed an exciting development in database theory: a general framework, called PANDA, that emerged from advances in database theory, constraint satisfaction problems (CSP), and graph algorithms, for evaluating conjunctive queries given input data statistics. The key idea is to derive information-theoretically tight upper bounds on the cardinalities of intermediate relations produced during query evaluation. These bounds determine the costs of query plans, and crucially, the query plans themselves are derived directly from the mathematical proof of the upper bound. This tight coupling of proof and algorithm is what makes PANDA both principled and powerful. Remarkably, this generic algorithm matches -- and in some cases subsumes -- the runtimes of specialized algorithms for the same problems, including algorithms that exploit fast matrix multiplication. This paper is a tutorial on the PANDA framework. We illustrate the key ideas through concrete examples, conveying the main intuitions behind the theory.

Problem

• conjunctive query
• query optimization
• query evaluation
• database theory
• information theory

Innovation

• information theory
• conjunctive query evaluation
• PANDA framework
• query optimization
• tight cardinality bounds