Massive Multitask Language Understanding (MMLU) in German

This dataset is to be used for the evaluation of LLM German language understanding. It is based on the hendrycksTest dataset (here and here) and was created by using the GPT-3.5 API to translate the entire test set and a few examples of the validation set. To make sure the answer options follow the intended sentence structure and are always of the correct format, GPT was prompted to output in a JSON format. This came with some complications that were later manually fixed.

The prompt used to translate a single example was the following:

insert prompt here @TODO

This translation cost a total of ~13€ including iterating on the prompt and fixing broken examples.