Dataset Description

• Repository: https://github.com/cisnlp/GlotCC
• Paper: https://arxiv.org/abs/2410.23825
• Point of Contact: amir@cis.lmu.de

Dataset Summary

GlotCC-V1.0 is a document-level, general domain dataset derived from CommonCrawl, covering more than 1000 languages.
It is built using the GlotLID language identification and Ungoliant pipeline from CommonCrawl.
We release our pipeline as open-source at https://github.com/cisnlp/GlotCC.

List of Languages: See https://datasets-server.huggingface.co/splits?dataset=cis-lmu/GlotCC-V1 to get the list of splits available.

Usage (Huggingface Hub -- Recommended)

Replace bal-Arab with your specific language.

from huggingface_hub import snapshot_download

folder = snapshot_download(
    "cis-lmu/glotcc-v1", 
    repo_type="dataset",
    local_dir="./path/to/glotcc-v1/",
    # Replace "v1.0/bal-Arab/*" with the path for any other language available in the dataset
    allow_patterns="v1.0/bal-Arab/*"
)

For faster downloads, make sure to pip install huggingface_hub[hf_transfer] and set the environment variable HF_HUB_ENABLE_HF_TRANSFER=1.

Then you can load it with any library that supports Parquet files, such as Pandas:

import pandas as pd

# Load the dataset from a Parquet file
# Replace the file path with the path to the desired language's Parquet file
dataset = pd.read_parquet('./path/to/glotcc-v1/v1.0/bal-Arab/bal-Arab_0.parquet')

# Print the first 5 rows of the dataset
print(dataset.head())

Usage (Huggingface datasets)

from datasets import load_dataset

# Replace "bal-Arab" with the name of any other language available in the dataset
dataset = load_dataset("cis-lmu/glotcc-v1", name="bal-Arab", split="train")

# Print the first row of data
print(dataset[0])

Usage (Huggingface datasets -- streaming=True)

from datasets import load_dataset

# Replace "bal-Arab" with the name of any other language available in the dataset
fw = load_dataset("cis-lmu/glotcc-v1", name="bal-Arab", split="train", streaming=True)

# Create an iterator from the streaming dataset
iterator = iter(fw)

# Print the next item from the iterator
print(next(iterator))

Usage (direct download)

If you prefer not to use the Hugging Face datasets or hub you can download it directly. For example, to download the first file of bal-Arab:

!wget https://huggingface.co/datasets/cis-lmu/GlotCC-V1/resolve/main/v1.0/bal-Arab/bal-Arab_0.parquet

Additional Information

The dataset is currently heavily under audit and changes accordingly.

Licensing Information

GlotCC data is released under the following licensing scheme: We do not own any of the text from which this data has been extracted. The data is licensed under the terms of the CommonCrawl Terms of Use. We license the actual packaging, metadata, and annotations of this data under the Creative Commons CC0 license.

Citation Information

If you find our data useful for your research, please cite:

@article{kargaran2024glotcc,
  title     = {Glot{CC}: An Open Broad-Coverage CommonCrawl Corpus and Pipeline for Minority Languages},
  author    = {Kargaran, Amir Hossein and Yvon, Fran{\c{c}}ois and Sch{\"u}tze, Hinrich},
  journal   = {Advances in Neural Information Processing Systems},
  year      = {2024},
  url       = {https://arxiv.org/abs/2410.23825}
}