KorMedMCQA : Multi-Choice Question Answering Benchmark for Korean Healthcare Professional Licensing Examinations

We present KorMedMCQA, the first Korean Medical Multiple-Choice Question Answering benchmark, derived from professional healthcare licensing examinations conducted in Korea between 2012 and 2024. The dataset contains 7,469 questions from examinations for doctor, nurse, pharmacist, and dentist, covering a wide range of medical disciplines. We evaluate the performance of 59 large language models, spanning proprietary and open-source models, multilingual and Korean-specialized models, and those fine-tuned for clinical applications. Our results show that applying Chain of Thought (CoT) reasoning can enhance the model performance by up to 4.5% compared to direct answering approaches. We also investigate whether MedQA, one of the most widely used medical benchmarks derived from the U.S. Medical Licensing Examination, can serve as a reliable proxy for evaluating model performance in other regions-in this case, Korea. Our correlation analysis between model scores on KorMedMCQA and MedQA reveals that these two benchmarks align no better than benchmarks from entirely different domains (e.g., MedQA and MMLU-Pro). This finding underscores the substantial linguistic and clinical differences between Korean and U.S. medical contexts, reinforcing the need for region-specific medical QA benchmarks.

Paper : https://arxiv.org/abs/2403.01469

Notice

We have made the following updates to the KorMedMCQA dataset:

1. Dentist Exam: Incorporated exam questions from 2021 to 2024.
2. Updated Test Sets: Added the 2024 exam questions for the doctor, nurse, and pharmacist test sets.
3. Few-Shot Split: Introduced a fewshot split, containing 5 shots from each validation set.
4. Chain-of-Thought(CoT): In each exam's few-shot split (cot column), there is an answer with reasoning annotated by professionals

Dataset Details

Languages

Korean

Subtask

from datasets import load_dataset
doctor = load_dataset(path = "sean0042/KorMedMCQA",name = "doctor")
nurse = load_dataset(path = "sean0042/KorMedMCQA",name = "nurse")
pharmacist = load_dataset(path = "sean0042/KorMedMCQA",name = "pharm")
dentist = load_dataset(path = "sean0042/KorMedMCQA",name = "dentist")

Statistics

Data Fields

• subject: doctor, nurse, or pharm
• year: year of the examination
• period: period of the examination
• q_number: question number of the examination
• question: question
• A: First answer choice
• B: Second answer choice
• C: Third answer choice
• D: Fourth answer choice
• E: Fifth answer choice
• cot : Answer with reasoning annotated by professionals (only available in fewshot split)
• answer : Answer (1 to 5). 1 denotes answer A, and 5 denotes answer E

Contact

sean0042@kaist.ac.kr