GradeSQL Training Dataset for BIRD (Unbalanced Variant)

Dataset Description

This dataset was created to train the GradeSQL Outcome Reward Model (ORM) for BIRD dataset, for verifying semantic correctness of SQL queries generated from natural language questions. It contains multiple SQL candidate queries per question, each annotated with a correctness label indicating whether it produces the same result as the reference (gold) SQL query.

The dataset helps models learn to distinguish semantically correct queries from incorrect ones, improving alignment with user intent in Text-to-SQL tasks.

Construction Methodology

• For a Text-to-SQL benchmark, the training set of questions is denoted as Qtrain.
• For each question in Qtrain, we generate N SQL candidate queries by prompting a large language model (LLM) to produce multiple chain-of-thought (CoT) reasoning paths. Each path ends with a SQL query.
• This results in a candidate set of queries for each question: Cq = {c1, c2, ..., cN}.
• Each candidate set of queries is filtered to keep only syntactically correct queries.
• To assign a correctness label, both the candidate query ci and the gold reference query gq are executed on the associated database, producing result sets R(ci) and R(gq).
• The labeling rules are:
  • correct if R(ci) equals R(gq)
  • incorrect if R(ci) differs from R(gq)
  • discarded if executing ci results in an error

If you use GradeSQL in your research, please cite the following paper:

@misc{gradesqloutcomerewardmodels2025,
      title={GradeSQL: Outcome Reward Models for Ranking SQL Queries from Large Language Models}, 
      author={Mattia Tritto and Giuseppe Farano and Dario Di Palma and Gaetano Rossiello and Fedelucio Narducci and Dharmashankar Subramanian and Tommaso Di Noia},
      year={2025},
      eprint={2509.01308},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2509.01308}, 
}