Ranked Enumeration for Database Queries

2024-09-12

SIGMOD record

AI Summary

This paper addresses the inefficiency of ranked enumeration—i.e., progressively returning the top-k results in order of importance—in database queries. We propose the any-k algorithmic framework, which pushes ranking depth into multi-way joins and integrates priority-queue-driven incremental join evaluation, ranking-aware physical plan rewriting, lazy materialization, and dynamic pruning to avoid full computation and intermediate result materialization. Our approach achieves, for the first time, theoretically optimal delay guarantees—O(log k)—and progressive output optimality. Experiments demonstrate that, compared to conventional post-join sorting, any-k accelerates the delivery of the first k results by one to two orders of magnitude, significantly reducing both response latency and memory overhead.

Technology Category

• Search and Optimization: Distributed Search
• Data Mining & Knowledge Management: Intelligent Query Processing
• Machine Learning: Learning Preferences or Rankings

Application Category

• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Ranked enumeration is a query-answering paradigm where the query answers are returned incrementally in order of importance (instead of returning all answers at once). Importance is defined by a ranking function that can be specific to the application, but typically involves either a lexicographic order (e.g., "ORDER BY R.A, S.B" in SQL) or a weighted sum of attributes (e.g., "ORDER BY 3*R.A + 2*S.B"). Recent work has introduced any-k algorithms for (multi-way) join queries, which push ranking into joins and avoid materializing intermediate results until necessary. The top-ranked answers are returned asymptotically faster than the common join-then-rank approach of database systems, resulting in orders-of-magnitude speedup in practice.

Problem

• Efficient ranked enumeration of query results
• Optimizing database join queries with ranking
• Reducing computational complexity in ranked enumeration

Innovation

• Ranked enumeration for queries
• Any-k algorithms introduced
• Simplified version presented