AstroMLab/Astrobench_MCQ_v1_Public

Astrobench is a specialized benchmarking dataset for evaluating the performance of Large Language Models in astronomy and astrophysics knowledge recall. This dataset consists of multiple-choice questions derived from the Annual Review of Astronomy and Astrophysics, designed to test models' comprehension and knowledge of astronomical research.

Example Usage

Here's how to load and use the dataset:

from datasets import load_dataset

# Load the dataset
dataset = load_dataset("AstroMLab/Astrobench_MCQ_v1_Public")

# Print the first question and its MCQ options
index = 0
example = dataset['train'][index]
print("Question:", example['question'])
print("\nOptions:")
print("A:", example['A'])
print("B:", example['B'])
print("C:", example['C'])
print("D:", example['D'])

Important Note on Dataset Structure

This public release intentionally omits the correct answers (keys) to the multiple-choice questions. This design choice is deliberate to:

• Prevent future models from being trained directly on the answers
• Maintain the dataset's validity as a genuine benchmarking tool
• Ensure fair and meaningful evaluation of model performance
• Allow for continuous use of this benchmark without compromising its effectiveness

Model Evaluation

We encourage researchers and developers to evaluate their models using this benchmark. To receive an evaluation score:

• Run your model on the dataset to generate answers
• Send your predicted answers to: astromachinelearninglab@gmail.com
• We will evaluate your submissions and return the performance metrics

Dataset Details

• Domain: Astronomy and Astrophysics
• Data Format: Multiple-choice questions
• Source: 885 articles from Annual Review of Astronomy and Astrophysics (1963-2023)
• Format: Multiple choice questions with four options (A, B, C, D) without answer keys
• Language: English
• License: MIT

Generation Guidelines

Questions were generated following specific criteria:

1. Specific enough to the article's content but general enough to be asked independently
2. Each question accompanied by four options with only one correct answer
3. A fraction of the questions have been validated by domain experts, the general quality should be robust, to ensure accuracy

Usage and Limitations

Intended Use

• Benchmarking LLMs for astronomical knowledge and reasoning
• Evaluating models' understanding of specialized scientific content in astronomy
• Assessing LLMs' potential for deployment in astronomical research

Citation

If you use this dataset in your research, please cite:

@ARTICLE{2024arXiv240711194T,
       author = {{Ting}, Yuan-Sen and {Dung Nguyen}, Tuan and {Ghosal}, Tirthankar and {Pan}, Rui and {Arora}, Hardik and {Sun}, Zechang and {de Haan}, Tijmen and {Ramachandra}, Nesar and {Wells}, Azton and {Madireddy}, Sandeep and {Accomazzi}, Alberto},
        title = "{AstroMLab 1: Who Wins Astronomy Jeopardy!?}",
      journal = {arXiv e-prints},
     keywords = {Astrophysics - Instrumentation and Methods for Astrophysics, Astrophysics - Earth and Planetary Astrophysics, Astrophysics - Astrophysics of Galaxies, Astrophysics - Solar and Stellar Astrophysics, Computer Science - Artificial Intelligence, Computer Science - Computation and Language},
         year = 2024,
        month = jul,
          eid = {arXiv:2407.11194},
        pages = {arXiv:2407.11194},
          doi = {10.48550/arXiv.2407.11194},
archivePrefix = {arXiv},
       eprint = {2407.11194},
 primaryClass = {astro-ph.IM},
       adsurl = {https://ui.adsabs.harvard.edu/abs/2024arXiv240711194T},
      adsnote = {Provided by the SAO/NASA Astrophysics Data System}
}

Data and Model Access

• Dataset: https://huggingface.co/datasets/AstroMLab/Astrobench_MCQ_v1_Public
• Documentation: https://astromlab.org