Rakuda - Questions for Japanese models

Repository: https://github.com/yuzu-ai/japanese-llm-ranking

This is a set of 40 questions in Japanese about Japanese-specific topics designed to evaluate the capabilities of AI Assistants in Japanese.

The questions are evenly distributed between four categories: history, society, government, and geography. Questions in the first three categories are open-ended, while the geography questions are more specific.

Answers to these questions can be used to rank the Japanese abilities of models, in the same way the vicuna-eval questions are frequently used to measure the usefulness of assistants.

Usage

from datasets import load_dataset

dataset = load_dataset("yuzuai/rakuda-questions")

print(dataset)
# => DatasetDict({
#       train: Dataset({
#           features: ['category', 'question_id', 'text'],
#           num_rows: 40
#       })
#    })