Taiwan Legal Benchmark v1

A multiple-choice benchmark for evaluating large language models on Taiwan law in Traditional Chinese (繁體中文). It covers six legal domains with 209 questions drawn from Taiwan bar exam and certification-style questions.

Overview

Legal Domains Covered

Questions test statutory interpretation, procedural rules, and applied legal reasoning — representative of the difficulty level found in Taiwan's national bar and professional certification exams.

Dataset Structure

benchmark.csv
├── question   (str)  — Legal question in Traditional Chinese
├── A          (str)  — Option A
├── B          (str)  — Option B
├── C          (str)  — Option C
├── D          (str)  — Option D
├── choices    (str)  — JSON list of [A, B, C, D] option texts
└── answer     (str)  — Correct answer letter: "A", "B", "C", or "D"

Sample

{
  "question": "下列關於專利申請之敘述，何者正確？",
  "A": "專利申請人申請專利及辦理有關專利事項，不得由專利申請人自行辦理之",
  "B": "若在中華民國境內，無住所或營業所者，申請專利及辦理專利有關事項，應由專利申請人自行辦理之",
  "C": "專利代理人，除法令另有規定外，以專利師為限",
  "D": "專利師之資格及管理，由律師法定之",
  "choices": "[\"專利申請人...\", \"若在中華民國...\", \"專利代理人...\", \"專利師之資格...\"]",
  "answer": "C"
}

Usage

Load with 🤗 Datasets

from datasets import load_dataset

ds = load_dataset("lianghsun/tw-legal-benchmark-v1")
print(ds["train"][0])

Quick accuracy evaluation

from datasets import load_dataset

ds = load_dataset("lianghsun/tw-legal-benchmark-v1", split="train")

def evaluate(predict_fn):
    correct = 0
    for row in ds:
        pred = predict_fn(row["question"], row["A"], row["B"], row["C"], row["D"])
        if pred.strip().upper() == row["answer"]:
            correct += 1
    return correct / len(ds)

Benchmark Registration

This dataset is registered as a HuggingFace benchmark via eval.yaml using the inspect-ai evaluation framework:

• Task ID: tw_legal
• Input field: question
• Target field: answer
• Choices field: choices
• Solver: multiple_choice
• Scorer: choice

To submit evaluation results for a model, add a .eval_results/*.yaml file to the model repo referencing lianghsun/tw-legal-benchmark-v1.

Citation

If you use this dataset in your research, please cite:

@dataset{tw_legal_benchmark_v1,
  author    = {Huang Liang Hsun},
  title     = {Taiwan Legal Benchmark v1},
  year      = {2025},
  publisher = {Hugging Face},
  url       = {https://huggingface.co/datasets/lianghsun/tw-legal-benchmark-v1}
}