Dataset Card for "Mixed Arabic Datasets (MAD) Corpus"

The Mixed Arabic Datasets Corpus : A Community-Driven Collection of Diverse Arabic Texts

Dataset Description

The Mixed Arabic Datasets (MAD) presents a dynamic compilation of diverse Arabic texts sourced from various online platforms and datasets. It addresses a critical challenge faced by researchers, linguists, and language enthusiasts: the fragmentation of Arabic language datasets across the Internet. With MAD, we are trying to centralize these dispersed resources into a single, comprehensive repository.

Encompassing a wide spectrum of content, ranging from social media conversations to literary masterpieces, MAD captures the rich tapestry of Arabic communication, including both standard Arabic and regional dialects.

This corpus offers comprehensive insights into the linguistic diversity and cultural nuances of Arabic expression.

Usage

If you want to use this dataset you pick one among the available configs:

Ara--MBZUAI--Bactrian-X | Ara--OpenAssistant--oasst1 | Ary--AbderrahmanSkiredj1--Darija-Wikipedia

Ara--Wikipedia | Ary--Wikipedia | Arz--Wikipedia

Ary--Ali-C137--Darija-Stories-Dataset | Ara--Ali-C137--Hindawi-Books-dataset | ``

Example of usage:

dataset = load_dataset('M-A-D/Mixed-Arabic-Datasets-Repo', 'Ara--MBZUAI--Bactrian-X')

If you loaded multiple datasets and wanted to merge them together then you can simply laverage concatenate_datasets() from datasets

dataset3 = concatenate_datasets([dataset1['train'], dataset2['train']])

Note : proccess the datasets before merging in order to make sure you have a new dataset that is consistent

Dataset Size

The Mixed Arabic Datasets (MAD) is a dynamic and evolving collection, with its size fluctuating as new datasets are added or removed. As MAD continuously expands, it becomes a living resource that adapts to the ever-changing landscape of Arabic language datasets.

Dataset List

MAD draws from a diverse array of sources, each contributing to its richness and breadth. While the collection is constantly evolving, some of the datasets that are poised to join MAD in the near future include:

• [✔] OpenAssistant/oasst1 (ar portion) : Dataset Link
• [✔] MBZUAI/Bactrian-X (ar portion) : Dataset Link
• [✔] AbderrahmanSkiredj1/Darija-Wikipedia : Dataset Link
• [✔] Arabic Wikipedia : Dataset Link
• [✔] Moroccan Arabic Wikipedia : Dataset Link
• [✔] Egyptian Arabic Wikipedia : Dataset Link
• [✔] Darija Stories Dataset : Dataset Link
• [✔] Hindawi Books Dataset : Dataset Link
• [] uonlp/CulturaX - ar : Dataset Link
• [✔] Pain/ArabicTweets : Dataset Link
• [] Abu-El-Khair Corpus : Dataset Link
• [✔] QuranExe : Dataset Link
• [✔] MNAD : Dataset Link
• [✔] IADD : Dataset Link
• [] OSIAN : Dataset Link
• [✔] MAC corpus : Dataset Link
• [✔] Goud.ma-Sum : Dataset Link
• [✔] SaudiNewsNet : Dataset Link
• [✔] Miracl : Dataset Link
• [✔] CardiffNLP/TweetSentimentMulti : Dataset Link
• [] OSCAR-2301 : Dataset Link
• [] mc4 : Dataset Link
• [✔] bigscience/xP3 : Dataset Link
• [] Muennighoff/xP3x : Dataset Link
• [] Ai_Society : Dataset Link

Potential Use Cases

The Mixed Arabic Datasets (MAD) holds the potential to catalyze a multitude of groundbreaking applications:

• Linguistic Analysis: Employ MAD to conduct in-depth linguistic studies, exploring dialectal variances, language evolution, and grammatical structures.
• Topic Modeling: Dive into diverse themes and subjects through the extensive collection, revealing insights into emerging trends and prevalent topics.
• Sentiment Understanding: Decode sentiments spanning Arabic dialects, revealing cultural nuances and emotional dynamics.
• Sociocultural Research: Embark on a sociolinguistic journey, unraveling the intricate connection between language, culture, and societal shifts.

Dataset Access

MAD's access mechanism is unique: while it doesn't carry a general license itself, each constituent dataset within the corpus retains its individual license. By accessing the dataset details through the provided links in the "Dataset List" section above, users can understand the specific licensing terms for each dataset.

Join Us on Discord

For discussions, contributions, and community interactions, join us on Discord!

How to Contribute

Want to contribute to the Mixed Arabic Datasets project? Follow our comprehensive guide on Google Colab for step-by-step instructions: Contribution Guide.

Note: If you'd like to test a contribution before submitting it, feel free to do so on the MAD Test Dataset.

Citation

@dataset{ 
title = {Mixed Arabic Datasets (MAD)},
author = {MAD Community},
howpublished = {Dataset},
url = {https://huggingface.co/datasets/M-A-D/Mixed-Arabic-Datasets-Repo},
year = {2023},
}