Update 2025-05-22

The previous issue regarding mismatched MySQL instances has been resolved. The updated version of BIRD-CRITIC-Open is now available. Thank you for your patience and understanding.

Update 2025-04-25

We’ve identified a mismatch issue in some uploaded MySQL instances. Our team is actively working to resolve this, and we’ll release the updated version promptly. Please refrain from using MySQL until the fix is deployed. Apologies for any inconvenience caused.

BIRD-CRITIC-1.0-Open

Project Page

BIRD-Critic is the first SQL debugging benchmark designed to answer a critical question: Can large language models (LLMs) fix user issues in real-world database applications?
Each task in BIRD-CRITIC has been verified by human experts on the following dimensions:

1. Reproduction of errors on BIRD env to prevent data leakage.
2. Carefully curate test case functions for each task specifically.
   • Soft EX: This metric can evaluate SELECT-ONLY tasks.
   • Soft EX + Parsing: This metric can evaluate tasks with user specific requirements or refinements.
   • Test Case: For DBA tasks, such as CRUD (CREAT, READ, UPDATE, DELET), test cases should be promised to evaluate the correct logic. This is also effective for user issues requiring multiple sequential SQLs to resolve.
   • Query Execution Plan: For user tasks involving efficiency improvement or runtime errors, QEP can be introduced to evaluate solution SQLs on algorithm level.
3. Fast Eval Sandbox via differnt dialects (PostgreSQL, MySQL, SQL Server, Oracle) template & docker.
4. Created new RDBs in different scale and professional domains.

We are pleased to announce the release of the complete version of BIRD-Critic (600) shortly, bird-critic-1.0-open, which includes high-quality user issues across multiple database dialects (PostgreSQL, MySQL, SQL Server, Oracle) when developing real-world applications. We curate tasks by:

• Collecting and understanding realistic user issues.
• Distilling problem definitions and SQL knowledge.
• Reproducing bugs and solutions in the BIRD environment.
• Designing test cases for evaluation.

Model Performance Results (By Category)

Model Performance Results (By Dialect)

Paper: SWE-SQL: Illuminating LLM Pathways to Solve User SQL Issues in Real-World Applications

Tier Classification (By Ranking):

• 🏆 Leading: The Best!
• 🌟 Elite: Top 15%
• 💎 Superior: Top 30%
• 🔸 Advanced: Top 45%
• 💫 Standard: Top 70%
• ⚪ Basic: Bottom 30%

Dialect Distribution:

• PostgreSQL: 300
• MySQL: 100
• SQL Server: 100
• Oracle 100

Instance Categories:

• Query: Instances that involve classic retrieval operations (i.e., SELECT).
• Management: Instances that perform database management (e.g, CREATE, UPDATE, INSERT).
• Personalization: Instances requiring a custom approach to achieve.\
• Efficiency: Instances focused on query optimization.

Represent as issue_type in each data instance.

Dataset Details

Dataset Description

• Curated by: BIRD Team & Google Cloud
• License: cc-by-sa-4.0

Uses

To avoid data leakage by auto-crawling, certain fields (e.g., sol_sql, test_cases) are excluded from the public dataset. For the full dataset, please email: 📧 [email protected] with subject tag [bird-critic-1 GT&Test Cases], which will be sent automatically within 30 mins.

Code Sources

• Repository: https://github.com/bird-bench/BIRD-CRITIC-1/tree/main.

Dataset Structure

Below is a description of the dataset fields and additional information about the structure:

• db_id: The name of the database.
• query: The user query is rewritten in the BIRD environment.
• error_sql: The buggy SQL query written by the user.
• sol_sql: The ground truth SQL solution.
• preprocess_sql: SQL queries to run before executing the solution or prediction.
• clean_up_sql: SQL queries to run after the test cases to revert any changes made to the database.
• test_cases: A set of test cases to validate the predicted corrected SQL.
• efficiency: True if this question needs optimization, measure the cost by Query Execution Plan (QEP)
• external_data: For the external JSON data if present

📄 Paper

If you find our work helpful, please cite as:

@article{li2025swe,
  title={SWE-SQL: Illuminating LLM Pathways to Solve User SQL Issues in Real-World Applications},
  author={Li, Jinyang and Li, Xiaolong and Qu, Ge and Jacobsson, Per and Qin, Bowen and Hui, Binyuan and Si, Shuzheng and Huo, Nan and Xu, Xiaohan and Zhang, Yue and others},
  journal={arXiv preprint arXiv:2506.18951},
  year={2025}
}

Todo Lists

• Release lite version, bird-critic-1.0-flash (200).
• Open source code, leaderboard page.
• Release Full bird-critic-1.0-open (600 w/ 5 dialects).
• Release Full bird-critic-1.0-postgresql (600 pg tasks).
• BIRD-Nest, a Gym-like training set for bird-critic-1.0
• LiveSQLBench Base
• BIRD-CRITIC 1.5 / 2.0 on track!

Post By:

BIRD Team & Google Cloud