TMMLU+ : Large scale traditional chinese massive multitask language understanding

iKala presents TMMLU+, a large-scale benchmark for evaluating LLM capabilities in Traditional Chinese, with content primarily reflecting Taiwan's linguistic, educational, and professional contexts. It covers 66 subjects, from elementary to professional domains, and is approximately six times larger than TMMLU with broader, more balanced coverage.

TMMLU+ v1.1 improves benchmark quality through systematic review: outdated legal and regulatory content was updated, incomplete or invalid questions were corrected or removed, and ambiguous items were reviewed by domain experts. Questions without a single defensible answer were excluded.

from datasets import load_dataset
task_list = [
             'engineering_math', 'dentistry', 'traditional_chinese_medicine_clinical_medicine', 'clinical_psychology', 'technical', 'culinary_skills', 'mechanical', 'logic_reasoning', 'real_estate',
             'general_principles_of_law', 'finance_banking', 'anti_money_laundering', 'ttqav2', 'marketing_management', 'business_management', 'organic_chemistry', 'advance_chemistry',
             'physics', 'secondary_physics', 'human_behavior', 'national_protection', 'jce_humanities', 'politic_science', 'agriculture', 'official_document_management',
             'financial_analysis', 'pharmacy', 'educational_psychology', 'statistics_and_machine_learning', 'management_accounting', 'introduction_to_law', 'computer_science', 'veterinary_pathology',
             'accounting', 'fire_science', 'optometry', 'insurance_studies', 'pharmacology', 'taxation', 'trust_practice', 'geography_of_taiwan', 'physical_education', 'auditing', 'administrative_law',
             'education_(profession_level)', 'economics', 'veterinary_pharmacology', 'nautical_science', 'occupational_therapy_for_psychological_disorders',
             'basic_medical_science', 'macroeconomics', 'trade', 'chinese_language_and_literature', 'tve_design', 'junior_science_exam', 'junior_math_exam', 'junior_chinese_exam',
             'junior_social_studies', 'tve_mathematics', 'tve_chinese_language', 'tve_natural_sciences', 'junior_chemistry', 'music', 'education', 'three_principles_of_people',
             'taiwanese_hokkien'
            ]
for task in task_list:
  val = load_dataset('ikala/tmmluplus', task)['validation']
  dev = load_dataset('ikala/tmmluplus', task)['train']
  test = load_dataset('ikala/tmmluplus', task)['test']

For each dataset split

for row in test:
  print(row)
  break
>> Dataset({
    features: ['question', 'A', 'B', 'C', 'D', 'answer'],
    num_rows: 11
})

Statistic on all four categories : STEM, Social Science, Humanities, Other

Dataset Versions

v1.1 was produced by individually re-verifying every question flagged as potentially problematic by four rule-based scans (6,116 candidate questions out of 22,742), followed by a human review pass on the 691 questions where the verification suggested a change.

• 197 questions had their answer corrected — 196 with the answer key changed (change_answer), plus 1 whose stem was rewritten to fix a corrupted/duplicated fragment (all 4 options and the answer were kept)
• 539 questions were removed entirely (255 change_content — defective stem/options; 233 expert_review — unresolved ambiguity even after review; 51 remove — flagged in an earlier review pass)
• 22,203 questions remain in v1.1 (out of 22,742 in v1.0)

(For v1.0 figures, see the "Statistic on all four categories" table above, or load revision="v1.0".)

To load a specific version:

from datasets import load_dataset
load_dataset('ikala/tmmluplus', 'accounting', revision='v1.0')  # original
load_dataset('ikala/tmmluplus', 'accounting', revision='v1.1')  # verified/corrected

Leaderboard

Scores below are computed against the v1.1 question set for the 21 models that have completed evaluation on all 66 subjects. Category and Total scores are the average of each category's own accuracy (STEM/Social Science/Humanities/Other weighted equally, matching the ievals methodology), not a per-question average. For v1.0 scores, load revision="v1.0" and re-run evaluation against that question set.

Note: all models were called with default API parameters (no reasoning effort or thinking mode explicitly configured). Models marked (reasoning) reported a separate reasoning/thinking token count from the API under this default before producing their final answer; other models answered directly without an exposed reasoning trace.

Licensing Information

This dataset is released under the MIT License. You are free to use, copy, modify, and redistribute it, including for commercial purposes, provided the original copyright notice is retained.

Citation

@article{ikala2023eval,
  title={An Improved Traditional Chinese Evaluation Suite for Foundation Model},
  author={Tam, Zhi-Rui and Pai, Ya-Ting and Lee, Yen-Wei and Cheng, Sega and Shuai, Hong-Han},
  journal={arXiv preprint arXiv:2403.01858},
  year={2023}
}

About iKala

iKala helps enterprises make better, faster decisions by embedding AI and data at the core of their business. We support AI transformation by helping organizations move from data to decisions, delivering full AI solutions that combine their first-party data with iKala's intelligence built on billions of global social signals.

Headquartered in Taiwan with a global footprint, iKala serves over 1,000 enterprises and 50,000 brands across more than 190 countries, including Fortune 500 companies.

iKalaOfficial Website: ikala.ai
KolrOfficial Website: kolr.ai
KuromaOfficial Website: kuroma.ai