Dataset Card for "CSAT-QA"

Dataset Summary

The field of Korean Language Processing is experiencing a surge in interest, illustrated by the introduction of open-source models such as Polyglot-Ko and proprietary models like HyperClova. Yet, as the development of larger and superior language models accelerates, evaluation methods aren't keeping pace. Recognizing this gap, we at HAE-RAE are dedicated to creating tailored benchmarks for the rigorous evaluation of these models.

CSAT-QA is a comprehensive collection of 936 multiple choice question answering (MCQA) questions, manually collected the College Scholastic Ability Test (CSAT), a rigorous Korean University entrance exam. The CSAT-QA is divided into two subsets: a complete version encompassing all 936 questions, and a smaller, specialized version used for targeted evaluations.

The smaller subset further diversifies into six distinct categories: Writing (WR), Grammar (GR), Reading Comprehension: Science (RCS), Reading Comprehension: Social Science (RCSS), Reading Comprehension: Humanities (RCH), and Literature (LI). Moreover, the smaller subset includes the recorded accuracy of South Korean students, providing a valuable real-world performance benchmark.

For a detailed explanation of how the CSAT-QA was created please check out the accompanying blog post, and for evaluation check out LM-Eval-Harness on github.

Evaluation Results

How to Use

The CSAT-QA includes two subsets. The full version with 936 questions can be downloaded using the following code:

from datasets import load_dataset
dataset = load_dataset("EleutherAI/CSAT-QA", "full")

A more condensed version, which includes human accuracy data, can be downloaded using the following code:

from datasets import load_dataset
import pandas as pd

dataset = load_dataset("EleutherAI/CSAT-QA", "GR") # Choose from either WR, GR, LI, RCH, RCS, RCSS, 

Evaluate using LM-Eval-Harness

To evaluate your model simply by using the LM-Eval-Harness by EleutherAI follow the steps below.

1. To install lm-eval from the github repository main branch, run:

git clone https://github.com/EleutherAI/lm-evaluation-harness
cd lm-evaluation-harness
pip install -e .

2. To install additional multilingual tokenization and text segmentation packages, you must install the package with the multilingual extra:

pip install -e ".[multilingual]"

3. Run the evaluation by:

python main.py \
    --model hf-causal \
    --model_args pretrained=EleutherAI/polyglot-ko-1.3b \
    --tasks csatqa_wr,csatqa_gr,csatqa_rcs,csatqa_rcss,csatqa_rch,csatqa_li \
    --device cuda:0

License

The copyright of this material belongs to the Korea Institute for Curriculum and Evaluation(한국교육과정평가원) and may be used for research purposes only.

More Information needed