Arabic Billion Words Dataset 🌕

The Abu El-Khair Arabic News Corpus (arabic-billion-words) is a comprehensive collection of Arabic text, encompassing over five million newspaper articles. The corpus is rich in linguistic diversity, containing more than a billion and a half words, with approximately three million unique words. The text is encoded in two formats: UTF-8 and Windows CP-1256, and marked up using two markup languages: SGML and XML.

Data Example

An example from the "Almasryalyoum" configuration subset:

{
  "url": "http://today.almasryalyoum.com/printerfriendly.aspx?ArticleID=61300",
  "head_line": "رئيس وزراء المجر: عنصرية جماهير أوجبيست جلبت العار للبلاد",
  "date": "19/5/2007",
  "text": """قال متحدث باسم الحكومة المجرية: إن رئيس الوزراء فيرنك جيوركساني رحب بقرار اتحاد كرة القدم المجري بخصم ثلاث نقاط من نادي أوجبيست بسبب السلوك العنصري الذي صدر من جماهيره.
  ...
  يذكر أن المجر بها مجموعة من مشجعي كرة القدم المشاغبين «الهوليجانز»، وشارك الكثير منهم في أعمال شغب معادية للحكومة في العام الماضي.""",
}

Data Fields

The dataset comprises the following fields:

• "url": string, original URL of the article.
• "head_line": string, headline of the article.
• "date": string, date of the article.
• "text": string, textual content of the article.

Data Splits

The dataset is divided into a single "training" split, with the following distribution of examples across different configuration subsets:

Dataset Usage

This dataset is valuable for natural language processing (NLP) tasks, including but not limited to sentiment analysis, topic modeling, and language modeling in the context of Arabic news articles. Researchers and practitioners in the field of NLP can benefit from the diverse linguistic content and the substantial size of the corpus.