Dataset Card for Wikipedia

Table of Contents

• Dataset Description
  • Dataset Summary
  • Supported Tasks and Leaderboards
  • Languages
• Dataset Structure
  • Data Instances
  • Data Fields
  • Data Splits
• Dataset Creation
  • Curation Rationale
  • Source Data
  • Annotations
  • Personal and Sensitive Information
• Considerations for Using the Data
  • Social Impact of Dataset
  • Discussion of Biases
  • Other Known Limitations
• Additional Information
  • Dataset Curators
  • Licensing Information
  • Citation Information
  • Contributions

Dataset Description

• Homepage: https://dumps.wikimedia.org
• Repository: More Information Needed
• Paper: More Information Needed
• Point of Contact: More Information Needed

Dataset Summary

Wikipedia dataset containing cleaned articles of all languages. The datasets are built from the Wikipedia dump (https://dumps.wikimedia.org/) with one split per language. Each example contains the content of one full Wikipedia article with cleaning to strip markdown and unwanted sections (references, etc.).

The articles are parsed using the mwparserfromhell tool, which can be installed with:

pip install mwparserfromhell

Then, you can load any subset of Wikipedia per language and per date this way:

from datasets import load_dataset

load_dataset("wikipedia", language="sw", date="20220120")

[!TIP] You can specify num_proc= in load_dataset to generate the dataset in parallel.

You can find the full list of languages and dates here.

Some subsets of Wikipedia have already been processed by HuggingFace, and you can load them just with:

from datasets import load_dataset

load_dataset("wikipedia", "20220301.en")

The list of pre-processed subsets is:

• "20220301.de"
• "20220301.en"
• "20220301.fr"
• "20220301.frr"
• "20220301.it"
• "20220301.simple"

Supported Tasks and Leaderboards

The dataset is generally used for Language Modeling.

Languages

You can find the list of languages here.

Dataset Structure

Data Instances

An example looks as follows:

{'id': '1',
 'url': 'https://simple.wikipedia.org/wiki/April',
 'title': 'April',
 'text': 'April is the fourth month...'
}

Some subsets of Wikipedia have already been processed by HuggingFace, as you can see below:

20220301.de

• Size of downloaded dataset files: 5.34 GB
• Size of the generated dataset: 8.91 GB
• Total amount of disk used: 14.25 GB

20220301.en

• Size of downloaded dataset files: 11.69 GB
• Size of the generated dataset: 20.28 GB
• Total amount of disk used: 31.96 GB

20220301.fr

• Size of downloaded dataset files: 4.22 GB
• Size of the generated dataset: 7.38 GB
• Total amount of disk used: 11.60 GB

20220301.frr

• Size of downloaded dataset files: 4.53 MB
• Size of the generated dataset: 9.13 MB
• Total amount of disk used: 13.66 MB

20220301.it

• Size of downloaded dataset files: 2.71 GB
• Size of the generated dataset: 4.54 GB
• Total amount of disk used: 7.25 GB

20220301.simple

• Size of downloaded dataset files: 133.89 MB
• Size of the generated dataset: 235.07 MB
• Total amount of disk used: 368.96 MB

Data Fields

The data fields are the same among all configurations:

• id (str): ID of the article.
• url (str): URL of the article.
• title (str): Title of the article.
• text (str): Text content of the article.

Data Splits

Here are the number of examples for several configurations:

Dataset Creation

Curation Rationale

More Information Needed

Source Data

Initial Data Collection and Normalization

More Information Needed

Who are the source language producers?

More Information Needed

Annotations

Annotation process

More Information Needed

Who are the annotators?

More Information Needed

Personal and Sensitive Information

More Information Needed

Considerations for Using the Data

Social Impact of Dataset

More Information Needed

Discussion of Biases

More Information Needed

Other Known Limitations

More Information Needed

Additional Information

Dataset Curators

More Information Needed

Licensing Information

Most of Wikipedia's text and many of its images are co-licensed under the Creative Commons Attribution-ShareAlike 3.0 Unported License (CC BY-SA) and the GNU Free Documentation License (GFDL) (unversioned, with no invariant sections, front-cover texts, or back-cover texts).

Some text has been imported only under CC BY-SA and CC BY-SA-compatible license and cannot be reused under GFDL; such text will be identified on the page footer, in the page history, or on the discussion page of the article that utilizes the text.

Citation Information

@ONLINE{wikidump,
    author = "Wikimedia Foundation",
    title  = "Wikimedia Downloads",
    url    = "https://dumps.wikimedia.org"
}

Contributions

Thanks to @lewtun, @mariamabarham, @thomwolf, @lhoestq, @patrickvonplaten for adding this dataset.