Evaluating LLMs in Database Scenarios: A Lifecycle Benchmark for Assessing Their Potential in Core Database Tasks

2026-08-04

AI Summary

This work addresses the limitation of existing Text-to-SQL evaluation benchmarks, which focus narrowly on a single task and overlook critical aspects of the full database lifecycle, including design, operation, and debugging. To bridge this gap, the authors propose DBLifeBench—the first comprehensive evaluation framework encompassing five key phases: design, implementation, execution, debugging, and maintenance. Central to this framework is Progressive-Text2SQL, a novel task grounded in structured reasoning graphs that emulates human iterative problem-solving to narrow the cognitive gap between natural language and complex SQL queries. Experimental results reveal that general-purpose large language models exhibit balanced performance across phases, whereas specialized Text-to-SQL models suffer from catastrophic forgetting outside coding-centric stages. This study establishes a systematic foundation for evaluating full-stack database intelligence.

Abstract

Large Language Models (LLMs) are transforming database interaction paradigms, evolving from simple query translators to autonomous database administrators (DBAs). However, current evaluation benchmarks remain disproportionately fixated on Text-to-SQL tasks, neglecting the holistic Database Lifecycle-from initial schema design to post-deployment maintenance. This narrow focus fails to capture the diverse capabilities required for real-world database management. To bridge this gap, we introduce DBLifeBench, the first benchmark to evaluate LLMs across five critical lifecycle phases: Design, Implementation, Operation, Debugging, and Maintenance. Furthermore, addressing the cognitive mismatch between ambiguous natural language and complex SQL logic, we propose Progressive-Text2SQL, a novel task utilizing structured reasoning graphs to mimic human iterative problem-solving. Our extensive evaluation reveals a critical insight: while general-purpose models demonstrate balanced performance, specialized Text-to-SQL models suffer from ``catastrophic forgetting'' in non-coding phases like design and maintenance. DBLifeBench serves as a foundational step toward evaluating and building true full-stack database intelligence.

Problem

• Large Language Models
• Database Lifecycle
• Text-to-SQL
• Benchmarking
• Database Management

Innovation

• DBLifeBench
• Database Lifecycle
• Progressive-Text2SQL
• Large Language Models
• Full-stack Database Intelligence