Update 2025-06-08

We release the full version of BIRD-Critic-PG, a dataset containing 530 high-quality user issues focused on real-world PostgreSQL database applications. The schema file is include in the code repository https://github.com/bird-bench/BIRD-CRITIC-1/blob/main/baseline/data/post_schema.jsonl

BIRD-CRITIC-1.0-PG

BIRD-Critic is the first SQL debugging benchmark designed to answer a critical question: Can large language models (LLMs) fix user issues in real-world database applications?
Each task in BIRD-CRITIC has been verified by human experts on the following dimensions:

1. Reproduction of errors on BIRD env to prevent data leakage.
2. Carefully curate test case functions for each task specifically.
   • Soft EX: This metric can evaluate SELECT-ONLY tasks.
   • Soft EX + Parsing: This metric can evaluate tasks with user specific requirements or refinements.
   • Test Case: For DBA tasks, such as CRUD (CREAT, READ, UPDATE, DELET), test cases should be promised to evaluate the correct logic. This is also effective for user issues requiring multiple sequential SQLs to resolve.
   • Query Execution Plan: For user tasks involving efficiency improvement or runtime errors, QEP can be introduced to evaluate solution SQLs on algorithm level.
3. Fast Eval Sandbox via differnt dialects (PostgreSQL, MySQL, SQL Server, Oracle) template & docker.
4. Created new RDBs in different scale and professional domains.

We are pleased to announce the release of the preview version of BIRD-Critic-PG (530) shortly, bird-critic-1.0-postgresql, which includes high-quality user issues focused on PostgreSQL when developing real-world applications. We curate tasks by:

• Collecting and understanding realistic user issues.
• Distilling problem definitions and SQL knowledge.
• Reproducing bugs and solutions in the BIRD environment.
• Designing test cases for evaluation.

Model Performance Results (By Category)

Tier Classification (By Ranking):

• 🏆 Leading: The Best!
• 🌟 Elite: Top 15%
• 💎 Superior: Top 30%
• 🔸 Advanced: Top 45%
• 💫 Standard: Top 70%
• ⚪ Basic: Bottom 30%

Instance Categories:

• Query: Instances that involve classic retrieval operations (i.e., SELECT).
• Management: Instances that perform database management (e.g, CREATE, UPDATE, INSERT).
• Personalization: Instances requiring a custom approach to achieve.

Represent as category in each data instance.

Dataset Details

Dataset Description

• Curated by: BIRD Team & Google Cloud
• License: cc-by-sa-4.0

Uses

To avoid data leakage by auto-crawling, certain fields (e.g., sol_sql, test_cases) are excluded from the public dataset. For the full dataset, please email: 📧 [email protected] with subject tag [bird-critic-1 GT&Test Cases], which will be sent automatically within 30 mins.

Code Sources

• Repository: https://github.com/bird-bench/BIRD-CRITIC-1/tree/main.

Dataset Structure

Below is a description of the dataset fields and additional information about the structure:

• db_id: The name of the database.
• query: The user query is rewritten in the BIRD environment.
• error_sql: The buggy SQL query written by the user.
• sol_sql: The ground truth SQL solution.
• preprocess_sql: SQL queries to run before executing the solution or prediction.
• clean_up_sql: SQL queries to run after the test cases to revert any changes made to the database.
• test_cases: A set of test cases to validate the predicted corrected SQL.

📄 Paper

If you find our work helpful, please cite as:

@article{li2025swe,
  title={SWE-SQL: Illuminating LLM Pathways to Solve User SQL Issues in Real-World Applications},
  author={Li, Jinyang and Li, Xiaolong and Qu, Ge and Jacobsson, Per and Qin, Bowen and Hui, Binyuan and Si, Shuzheng and Huo, Nan and Xu, Xiaohan and Zhang, Yue and others},
  journal={arXiv preprint arXiv:2506.18951},
  year={2025}
}

Todo Lists

• Release lite version, bird-critic-1.0-flash (200).
• Open source code, leaderboard page.
• Release Full bird-critic-1.0-open (570 w/ 5 dialects).
• Release Full bird-critic-1.0-postgresql (530 pg tasks).
• BIRD-Nest, a Gym-like training set for bird-critic-1.0
• LiveSQLBench Base
• BIRD-CRITIC 1.5 / 2.0 on track!

Post By:

BIRD Team & Google Cloud