M2QA: Multi-domain Multilingual Question Answering

M2QA (Multi-domain Multilingual Question Answering) is an extractive question answering benchmark for evaluating joint language and domain transfer. M2QA includes 13,500 SQuAD 2.0-style question-answer instances in German, Turkish, and Chinese for the domains of product reviews, news, and creative writing.

This Hugging Face datasets repo accompanies our paper "M2QA: Multi-domain Multilingual Question Answering". If you want an explanation and code to reproduce all our results or want to use our custom-built annotation platform, have a look at our GitHub repository: https://github.com/UKPLab/m2qa

Loading & Decrypting the Dataset

Following Jacovi et al. (2023), we encrypt the validation data to prevent leakage of the dataset into LLM training datasets. But loading the dataset is still easy:

To load the dataset, you can use the following code:

from datasets import load_dataset
from cryptography.fernet import Fernet

# Load the dataset
subset = "m2qa.german.news" # Change to the subset that you want to use
dataset = load_dataset("UKPLab/m2qa", subset)

# Decrypt it
fernet = Fernet(b"aRY0LZZb_rPnXWDSiSJn9krCYezQMOBbGII2eGkN5jo=")

def decrypt(example):
    example["question"] = fernet.decrypt(example["question"].encode()).decode()
    example["context"] = fernet.decrypt(example["context"].encode()).decode()
    example["answers"]["text"] = [fernet.decrypt(answer.encode()).decode() for answer in example["answers"]["text"]]
    return example

dataset["validation"] = dataset["validation"].map(decrypt)

The M2QA dataset is licensed under a "no derivative" agreement. To prevent contamination of LLM training datasets and thus preserve the dataset's usefulness to our research community, please upload the dataset only in encrypted form. Additionally, please use only APIs that do not utilize the data for training.

Overview / Data Splits

All used text passages stem from sources with open licenses. We list the licenses here: https://github.com/UKPLab/m2qa/tree/main/m2qa_dataset

We have validation data for the following domains and languages:

Additional Training Data

We also provide training data for five domain-language pairs, consisting of 1500 question-answer instances each, totalling 7500 training examples. These are the subsets that contain training data:

• m2qa.chinese.news
• m2qa.chinese.product_reviews
• m2qa.german.news
• m2qa.german.product_reviews
• m2qa.turkish.news

The training data is not encrypted.

Citation

If you use this dataset, please cite our paper:

@inproceedings{englander-etal-2024-m2qa,
    title = "M2QA: Multi-domain Multilingual Question Answering",
    author = {Engl{\"a}nder, Leon  and
      Sterz, Hannah  and
      Poth, Clifton A  and
      Pfeiffer, Jonas  and
      Kuznetsov, Ilia  and
      Gurevych, Iryna},
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2024",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.findings-emnlp.365",
    pages = "6283--6305",
}

License

This dataset is distributed under the CC-BY-ND 4.0 license.

Following Jacovi et al. (2023), we decided to publish with a "No Derivatives" license to mitigate the risk of data contamination of crawled training datasets.