KMMLU (Korean-MMLU)

We propose KMMLU, a new Korean benchmark with 35,030 expert-level multiple-choice questions across 45 subjects ranging from humanities to STEM. Unlike previous Korean benchmarks that are translated from existing English benchmarks, KMMLU is collected from original Korean exams, capturing linguistic and cultural aspects of the Korean language. We test 26 publically available and proprietary LLMs, identifying significant room for improvement. The best publicly available model achieves 50.54% on KMMLU, far below the average human performance of 62.6%. This model was primarily trained for English and Chinese, not Korean. Current LLMs tailored to Korean, such as Polyglot-Ko, perform far worse. Surprisingly, even the most capable proprietary LLMs, e.g., GPT-4 and HyperCLOVA X, achieve 59.95% and 53.40%, respectively. This suggests that further work is needed to improve Korean LLMs, and KMMLU offers the right tool to track this progress. We make our dataset publicly available on the Hugging Face Hub and integrate the benchmark into EleutherAI's Language Model Evaluation Harness.

Link to Paper: KMMLU: Measuring Massive Multitask Language Understanding in Korean

KMMLU Statistics

Categories

To reimplement the categories in the paper, refer to the following:

supercategories = {
        "accounting": "HUMSS",
        "agricultural_sciences": "Other",
        "aviation_engineering_and_maintenance": "Applied Science",
        "biology": "STEM",
        "chemical_engineering": "STEM",
        "chemistry": "STEM",
        "civil_engineering": "STEM",
        "computer_science": "STEM",
        "construction": "Other",
        "criminal_law": "HUMSS",
        "ecology": "STEM",
        "economics": "HUMSS",
        "education": "HUMSS",
        "electrical_engineering": "STEM",
        "electronics_engineering": "Applied Science",
        "energy_management": "Applied Science",
        "environmental_science": "Applied Science",
        "fashion": "Other",
        "food_processing": "Other",
        "gas_technology_and_engineering": "Applied Science",
        "geomatics": "Applied Science",
        "health": "Other",
        "industrial_engineer": "Applied Science",
        "information_technology": "STEM",
        "interior_architecture_and_design": "Other",
        "law": "HUMSS",
        "machine_design_and_manufacturing": "Applied Science",
        "management": "HUMSS",
        "maritime_engineering": "Applied Science",
        "marketing": "Other",
        "materials_engineering": "STEM",
        "mechanical_engineering": "STEM",
        "nondestructive_testing": "Applied Science",
        "patent": "Other",
        "political_science_and_sociology": "HUMSS",
        "psychology": "HUMSS",
        "public_safety": "Other",
        "railway_and_automotive_engineering": "Applied Science",
        "real_estate": "Other",
        "refrigerating_machinery": "Other",
        "social_welfare": "HUMSS",
        "taxation": "HUMSS",
        "telecommunications_and_wireless_technology": "Applied Science",
        "korean_history": "HUMSS",
        "math": "STEM"
    }

Point of Contact

For any questions contact us via the following email:)

spthsrbwls123@yonsei.ac.kr