Indonesian Wikipedia Data Repository

---

license: cc-by-sa-3.0

Welcome to Indonesian Wikipedia Data Repository. The datasets are extracted from Wikipedia HF and processed using the scripts available in this repository for reproducibility purpose.

FAQS

What are the available languages provided in dataset?

Please check the following table.

How do I extract new Wikipedia Dataset of Indonesian languages?

You may check to the script extract_raw_wiki_data.py to understand its implementations, or you can adjust the bash provided in extract_raw_wiki_data_indo.sh to extract it on your own. Please note that this dataset is extensible to any languages of your choice.

How do I extract new Wikipedia Dataset of Indonesian languages?

You may visit this Wikipedia Dump Index to check any latest available data and this link Wikipedia Language Coverage to map into any languages that you're wanting to extract.

How does the data being preprocessed? What makes it different from loading it directly from Wikipedia HF?

The data available in here are processed with following flows:

1. Raw data is being deduplicated on title and text (text-content from a given article), to remove articles containing boilerplate text (template text that are used usually for no-available informations or asking for contributions of content in that article), which usually deemed noisy for NLP data.
2. Furthermore, the title and text data are being checked for string-matching duplication (duplication of text that are being pre-processed, i.e symbols removed, HTML tags striped, or ASCII-chars validated). You may check this cleanse_wiki_data.py script to understand its implementation.

Getting Started

To read the datasets directly

Use one of the following code chunks to load it from HuggingFace Hub: You can refer to the 2nd args of config name using the following script

dataset = load_dataset(
  "sabilmakbar/indo_wiki",
  "indo_wiki_dedup_data" # a config name, can be "indo_wiki_raw_data" or "indowiki_dedup_id_only", defaults to "indo_wiki_dedup_data"
)

Or you can provide both lang and date_stamp (providing only one will thrown an error)

dataset = load_dataset(
  "sabilmakbar/indo_wiki",
  lang = "id", # see the splits for complete lang choices
  date_stamp="20230901"
)

To replicate the whole dataset generation process

1. Set-up a new Python/Conda Environment (recommended Python version: 3.9.6 to 3.9.18 or 3.10.0 to 3.10.13) and install the requirements on requirements.txt use this codebase via pip install -r requirements.txt.
2. Activate the chosen Python/Conda environment which the requirements are being installed.
3. Run this sh script for extractions from Wikimedia Dump: sh extract_raw_wiki_data_indo.sh.
4. Run this sh script of deduplication: sh dedup_raw_wiki_data_indo.sh.

Citation Info:

@ONLINE{wikidump,
    author = "Wikimedia Foundation",
    title  = "Wikimedia Downloads",
    url    = "https://dumps.wikimedia.org"}
@ONLINE{wikipedia-hf,
    title  = "Huggingface Wikipedia Dataset",
    url    = "https://huggingface.co/datasets/wikipedia"}