Dataset Card for SciQA

Table of Contents

• Dataset Description
  • Dataset Summary
  • Supported Tasks and Leaderboards
  • Languages
• Dataset Structure
  • Data Instances
  • Data Fields
  • Data Splits
• Dataset Creation
  • Curation Rationale
  • Source Data
  • Annotations
  • Personal and Sensitive Information
• Considerations for Using the Data
  • Social Impact of Dataset
  • Discussion of Biases
  • Other Known Limitations
• Additional Information
  • Dataset Curators
  • Licensing Information
  • Citation Information
  • Contributions

Dataset Description

• Homepage: SciQA Homepage
• Repository: SciQA Repository
• Paper: The SciQA Scientific Question Answering Benchmark for Scholarly Knowledge
• Point of Contact: Yaser Jaradeh

Dataset Summary

SciQA contains 2,565 SPARQL query - question pairs along with answers fetched from the open research knowledge graph (ORKG) via a Virtuoso SPARQL endpoint, it is a collection of both handcrafted and autogenerated questions and queries. The dataset is split into 70% training, 10% validation and 20% test examples.

Dataset Structure

Data Instances

An example of a question is given below:

{
    "id": "AQ2251",
    "query_type": "Factoid",
    "question": {
        "string": "Provide a list of papers that have utilized the Depth DDPPO model and include the links to their code?"
    },
    "paraphrased_question": [],
    "query": {
        "sparql": "SELECT DISTINCT ?code\nWHERE {\n  ?model    a  orkgc:Model;\n            rdfs:label    ?model_lbl.\n  FILTER (str(?model_lbl) = \"Depth DDPPO\")\n  ?benchmark      orkgp:HAS_DATASET        ?dataset.\n  ?cont           orkgp:HAS_BENCHMARK      ?benchmark.\n  ?cont           orkgp:HAS_MODEL          ?model;\n                  orkgp:HAS_SOURCE_CODE    ?code.\n}"
    },
    "template_id": "T07",
    "auto_generated": true,
    "query_shape": "Tree",
    "query_class": "WHICH-WHAT",
    "number_of_patterns": 4,
}

Data Fields

• id: the id of the question
• question: a string containing the question
• paraphrased_question: a set of paraphrased versions of the question
• query: a SPARQL query that answers the question
• query_type: the type of the query
• query_template: an optional template of the query
• query_shape: a string indicating the shape of the query
• query_class: a string indicating the class of the query
• auto_generated: a boolean indicating whether the question is auto-generated or not
• number_of_patterns: an integer number indicating the number of gtaph patterns in the query

Data Splits

The dataset is split into 70% training, 10% validation and 20% test questions.

Additional Information

Licensing Information

SciQA is licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0).

Citation Information

@Article{SciQA2023,
  author={Auer, S{\"o}ren
  and Barone, Dante A. C.
  and Bartz, Cassiano
  and Cortes, Eduardo G.
  and Jaradeh, Mohamad Yaser
  and Karras, Oliver
  and Koubarakis, Manolis
  and Mouromtsev, Dmitry
  and Pliukhin, Dmitrii
  and Radyush, Daniil
  and Shilin, Ivan
  and Stocker, Markus
  and Tsalapati, Eleni},
  title={The SciQA Scientific Question Answering Benchmark for Scholarly Knowledge},
  journal={Scientific Reports},
  year={2023},
  month={May},
  day={04},
  volume={13},
  number={1},
  pages={7240},
  abstract={Knowledge graphs have gained increasing popularity in the last decade in science and technology. However, knowledge graphs are currently relatively simple to moderate semantic structures that are mainly a collection of factual statements. Question answering (QA) benchmarks and systems were so far mainly geared towards encyclopedic knowledge graphs such as DBpedia and Wikidata. We present SciQA a scientific QA benchmark for scholarly knowledge. The benchmark leverages the Open Research Knowledge Graph (ORKG) which includes almost 170,000 resources describing research contributions of almost 15,000 scholarly articles from 709 research fields. Following a bottom-up methodology, we first manually developed a set of 100 complex questions that can be answered using this knowledge graph. Furthermore, we devised eight question templates with which we automatically generated further 2465 questions, that can also be answered with the ORKG. The questions cover a range of research fields and question types and are translated into corresponding SPARQL queries over the ORKG. Based on two preliminary evaluations, we show that the resulting SciQA benchmark represents a challenging task for next-generation QA systems. This task is part of the open competitions at the 22nd International Semantic Web Conference 2023 as the Scholarly Question Answering over Linked Data (QALD) Challenge.},
  issn={2045-2322},
  doi={10.1038/s41598-023-33607-z},
  url={https://doi.org/10.1038/s41598-023-33607-z}
}

Contributions

Thanks to @YaserJaradeh for adding this dataset.