🚀 LiveSQLBench-Base-Lite

A dynamic, contamination‑free benchmark for evaluating LLMs on complex, real‑world text‑to‑SQL tasks.

🌐 Website • 📄 Paper (coming soon) • 💻 GitHub • 🗄️ LiveSQLBench-Base-Full-v1 • 🗄️ LiveSQLBench-Large-v1 • 🗄️ Bird-Interact (ICLR 2026 Oral)

Maintained by the 🦜 BIRD Team @ HKU & ☁️ Google Cloud

📊 LiveSQLBench Overview

LiveSQLBench (BIRD-SQL Pro v0.5) is a contamination-free, continuously evolving benchmark designed to evaluate LLMs on complex, real-world text-to-SQL tasks, featuring diverse real-world user queries, including Business Intelligence (BI), CRUD operations, and more. Each release will include 50 new, fully open-source DBs curated by the BIRD team through expert collaboration and continuous improvement. It will cover a wide range of database sizes, from end-user level (around 127 columns) to industrial level (~1000 columns). LiveSQLBench has been the infrastructure for the BIRD-Interact (ICLR 2026 Oral) , a dynamic human-centric interactive Text-to-SQL benchmark. Here are the features of the LiveSQLBench benchmark:

1. 🗄️ Live Databases: Constructed dynamically from extensive and regularly updated CSV datasets, with both base (user-end level) and large (industrial level) versions (1340+ columns each DB) to test scalability.

2. 💬 Live User Queries and SQL: Each task pairs unambiguous user queries with annotated, gold-standard SQL statements. The user queries are grounded in an external knowledge base, with medium to hard complexity solution SQL statements.

3. 🧠 Contextual Reasoning (HKB): Every DB includes a hierarchical knowledge base (HKB) where each knowledge may have dependencies to others, which requires the multi-hop reasoning ability. Two HKB formats are provided: (1) structured JSON format, and (2) unstructured Document format.

4. 🔍 The First Full SQL Spectrum: Supports not just SELECT (Business Intelligence) queries, but also CRUD (e.g., UPDATE, CREATE, and other database management operations) queries.

5. ⚡ Automated Evaluation: Support fast evaluation via PostgreSQL template & docker. Each question includes verifiable test cases for accurate, reproducible scoring. Soft EX metric is used to evaluate SELECT-ONLY tasks; customized test cases are designed for DBA tasks, such as CRUD (CREATE, READ, UPDATE, DELETE).

6. 🔄 Truly Live & Hidden Test: New databases and tasks are added over time. Each release features both open development and hidden test phases. The hidden test set from each release becomes the open development set for the next release, ensuring continuous evolution and fair evaluation.

🎯 Current Release: LiveSQLBench-Base-Lite

Currently, we are pleased to release a LiveSQLBench-Base-Lite, featuring 18 end-user level databases with 270 tasks (180 SELECT-only, 90 Management tasks), HKB-JSON and the JSON operation in SQL for trial.

💻 How to Use the Dataset

Download the dataset containing data file livesqlbench_data.jsonl and DB metafiles (including schema, HKB, column meaning files) by:

git clone https://huggingface.co/datasets/birdsql/livesqlbench-base-lite

To prevent data leakage through automated crawling, please request access to the ground truth and test cases by emailing 📧 [email protected] with the subject line [livesqlbench-base-lite GT&Test Cases]. An automated response will provide these data fields.

DDL scripts for building all DBs are in Google Drive.

And please refer to our Github repo for details of usage and evaluation based on this dataset.

📁 Directory Structure

Each database has its own directory:

.
├── README.md
├── alien
│   ├── alien_column_meaning_base.json
│   ├── alien_kb.jsonl
│   ├── alien_schema.txt
...
├── livesqlbench_data.jsonl

📂 Directory Contents:

• *_schema.txt: Database schema.
• *_kb.jsonl: Hierarchical knowledge base entries required to solve the user task.
  • id: The unique identifier for the knowledge.
  • knowledge: The name of the knowledge.
  • description: The description of the knowledge.
  • definition: The clear definition of the knowledge.
  • type: The type of the knowledge.
  • children_knowledge: A list of knowledge IDs that the current knowledge is dependent on. -1 means no children.
• *_column_meaning_base.json: Explanation of database columns.

📋 Dataset Fields (livesqlbench_data.jsonl):

• instance_id: Unique task identifier.
• selected_database: Associated database name.
• query: Unambiguous user query.
• sol_sql 🔒: Ground truth SQL solution.
• external_knowledge 🔒: IDs of required external knowledge to solve the user task.
• preprocess_sql: SQL setup queries.
• clean_up_sql: SQL queries to reset database state.
• test_cases 🔒: Test cases to validate the predicted corrected SQL.
• category: "Query" (SELECT-only) or "Management" (CRUD).
• high_level: Boolean indicating whether the user query contains high-level description.
• conditions: Indicates decimal/distinct conditions in the user query.
• difficulty_tier: Task difficulty (Simple, Moderate, Challenging).

🔒 Accessing Complete Data

To avoid data leakage by auto-crawling, certain fields (e.g., sol_sql, test_cases, external_knowledge) are excluded from the public dataset. For the full dataset, please email: 📧 [email protected] with subject tag [livesqlbench-base-lite GT&Test Cases], which will be sent automatically within 30 mins.

🏆 Model Performance on LiveSQLBench-Base-Lite (2025-05-28)

🔄 Stay Tuned!

Upcoming releases:

• 🔄 LiveSQLBench-Base-Lite-Sqlite: SQLite version of LiveSQLBench-Base-Lite. No need to setup the docker.
• 🔄 LiveSQLBench-Base-Full: 600 tasks with more natural user queries and real DBs.
• 🔄 LiveSQLBench-Large-v1: Industrial-scale databases with ~1000 columns.
• 🔄 Keep it Live!: New DB envs and tasks will be added periodically with hidden test sets, keeping the benchmark truly live and contamination-free.
• 💻 LiveSQLBench-CLI: CLI-interactive variant adapted for coding agents (e.g., Claude Code, OpenHands), where agents explore databases and iteratively solve SQL tasks through terminal workflows.

Want new dialects? Vote for new SQL dialects 🗳️ here!

📄 License:

cc-by-sa-4.0