Dataset Card for xP3x

Table of Contents

• Table of Contents
• Dataset Description
  • Dataset Summary
  • Supported Tasks and Leaderboards
  • Languages
• Dataset Structure
  • Data Instances
  • Data Fields
  • Data Splits
• Dataset Creation
  • Curation Rationale
  • Source Data
  • Annotations
• Additional Information
  • Licensing Information
  • Citation Information
  • Contributions

Dataset Description

• Repository: https://github.com/bigscience-workshop/xmtf
• Paper: Crosslingual Generalization through Multitask Finetuning
• Point of Contact: Niklas Muennighoff

Dataset Summary

xP3x (Crosslingual Public Pool of Prompts eXtended) is a collection of prompts & datasets across 277 languages & 16 NLP tasks. It contains all of xP3 + much more! It is used for training future contenders of mT0 & BLOOMZ at project Aya @Cohere Labs 🧡

• Creation: The dataset can be recreated using instructions available here together with the file in this repository named xp3x_create.py. We provide this version to save processing time.
• Languages: 277
• xP3 Dataset Family:

Dataset Structure

Data Instances

An example looks as follows:

{
  'inputs': '11月、遂にクロームはファイヤーフォックスを引き離し始めた。_はインターネットユーザーの評価が高まったのだ。\nReplace the _ in the above sentence with the correct option: \n- ファイヤーフォックス\n- クローム',
  'targets': 'クローム',
  'language': 'jpn_Jpan',
  'split': 'test',
  'template': 'Replace',
  'dataset': 'Muennighoff/xwinograd',
  'config': 'jp'
}

Data Fields

The data fields are the same among all splits:

• inputs: the natural language input fed to the model
• targets: the natural language target that the model has to generate
• language: The language code. The codes are an extension of the FLORES-200 codes, where the first part is the language code and the second part the script code.
• template: The name of the prompt used.
• dataset: The Hugging Face dataset identifier of where the data stems from.
• config: The config of the Hugging Face dataset.

Usage

The dataset has 680 gigabytes and 530 million samples. You may want to filter it and then deduplicate depending on your needs.

Loading by language:

# pip install -q datasets
from datasets import load_dataset
ds = load_dataset("Muennighoff/xP3x", "zho_Hans", streaming=True) # Use streaming to not download all at once
for x in ds["train"]:
    print(x)
    break

You can then filter down by the data fields to e.g. only get certain configs or datasets. As every dataset-config-template is its own jsonl file, you can also decide on the datasets, configs and templates you want and only download them. For example, to download all Japanese xwinograd samples, you could do:

# pip install -q datasets
from datasets import load_dataset
import multiprocessing
# pip install --upgrade huggingface-hub
from huggingface_hub import HfFileSystem, hf_hub_url

fs = HfFileSystem()
fps = fs.glob(f"datasets/CohereLabs/xP3x/data/jpn_Jpan/*xwinograd*")
resolved_paths = [fs.resolve_path(file) for file in fps]
data_files = [hf_hub_url(resolved_path.repo_id, resolved_path.path_in_repo, repo_type=resolved_path.repo_type) for resolved_path in resolved_paths]

ds = load_dataset("json", data_files=data_files, num_proc=8)["train"]

Sometimes it may be faster to clone the entire repo. To download all English files, you could do e.g.

GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/datasets/CohereLabs/xP3x
cd xP3x
git lfs pull --include="data/eng_Latn/*"

Data Splits

Language specifics

• Japanese: Data in jpn_Hira, jpn_Kana, jpn_Hani is guaranteed to have Hiragana, Katakana or Kanji, respectively in each sample. However, they may still include other styles. So while all samples in jpn_Kana are guaranteed to have Katakana, there may still be Hiragana or Kanji.

Dataset Creation

Source Data

Training datasets

• Code Miscellaneous
  • CodeComplex
  • Docstring Corpus
  • GreatCode
  • State Changes
• Closed-book QA
  • Hotpot QA
  • Trivia QA
  • Web Questions
  • Wiki QA
• Extractive QA
  • Adversarial QA
  • CMRC2018
  • DRCD
  • DuoRC
  • MLQA
  • Quoref
  • ReCoRD
  • ROPES
  • SQuAD v2
  • xQuAD
  • TyDI QA
    • Primary
    • Goldp
• Multiple-Choice QA
  • ARC
  • C3
  • CoS-E
  • Cosmos
  • DREAM
  • MultiRC
  • OpenBookQA
  • PiQA
  • QUAIL
  • QuaRel
  • QuaRTz
  • QASC
  • RACE
  • SciQ
  • Social IQA
  • Wiki Hop
  • WiQA
• Paraphrase Identification
  • MRPC
  • PAWS
  • PAWS-X
  • QQP
• Program Synthesis
  • APPS
  • CodeContests
  • JupyterCodePairs
  • MBPP
  • NeuralCodeSearch
  • XLCoST
• Structure-to-text
  • Common Gen
  • Wiki Bio
• Sentiment
  • Amazon
  • App Reviews
  • IMDB
  • Rotten Tomatoes
  • Yelp
• Simplification
  • BiSECT
• Summarization
  • CNN Daily Mail
  • Gigaword
  • MultiNews
  • SamSum
  • Wiki-Lingua
  • XLSum
  • XSum
• Topic Classification
  • AG News
  • DBPedia
  • TNEWS
  • TREC
  • CSL
• Translation
  • Flores-200
  • Tatoeba
  • MultiEURLEX
• Word Sense disambiguation
  • WiC
  • XL-WiC
• Natural Language Inference (NLI)
  • ANLI
  • CB
  • RTE
  • XNLI
• Coreference Resolution
  • Winogrande
  • XWinograd
• Sentence Completion
  • COPA
  • Story Cloze
  • XCOPA
  • XStoryCloze

Dataset specifics

• Flores-200: There are three prompts for Flores: continuation, question, command, which represent three commonly used prompting styles, i.e. making a prompt seem like a natural continuation, turning it into a question or commanding the model to do something.
• tatoeba_mt: Contains duplicates. For example, it has data that is both classified as jpn_Kana and jpn_Jpan, so you may want to deduplicate.

Additional Information

Licensing Information

The dataset collection is released under Apache 2.0. Note that individual datasets may have different licenses.

Citation Information

@article{muennighoff2022crosslingual,
  title={Crosslingual generalization through multitask finetuning},
  author={Muennighoff, Niklas and Wang, Thomas and Sutawika, Lintang and Roberts, Adam and Biderman, Stella and Scao, Teven Le and Bari, M Saiful and Shen, Sheng and Yong, Zheng-Xin and Schoelkopf, Hailey and others},
  journal={arXiv preprint arXiv:2211.01786},
  year={2022}
}

Contributions

Thanks to the contributors of promptsource for adding many prompts used in this dataset. Thanks to the Aya team @Cohere Labs 🧡