ArabicRAGB: Arabic Retrieval-Augmented Generation Benchmark
13
14 commits
1 linked in READMEs
updated Dec 15, 2025
ArabicRAGB is a benchmark dataset for evaluating Retrieval-Augmented Generation (RAG) systems on Arabic language tasks. Each record contains a query-passage pair where the query is grounded in the passage content.
| Component | Count |
|---|---|
| Training Records | 10,530 |
| Test Records | 2,633 |
| Total Records | 13,163 |
| Dialect | Count |
|---|---|
| MSA | 4,012 |
| Egyptian | 3,999 |
| Gulf | 3,783 |
| Levantine | 697 |
| Maghrebi | 672 |
| Level | Count |
|---|---|
| Simple | 4,092 |
| Moderate | 4,030 |
| Complex | 4,215 |
| Multi-hop | 826 |
from datasets import load_dataset
# Load the dataset
dataset = load_dataset("HeshamHaroon/ArabicRAGB")
train_data = dataset["train"]
test_data = dataset["test"]
print(f"Train records: {len(train_data)}")
print(f"Test records: {len(test_data)}")
# Example record
example = train_data[0]
print(f"Query: {example['query']}")
print(f"Dialect: {example['query_dialect']}")
print(f"Passage: {example['passage_text'][:200]}...")
from datasets import load_dataset
from sentence_transformers import SentenceTransformer
import numpy as np
# Load dataset
dataset = load_dataset("HeshamHaroon/ArabicRAGB")
test_data = dataset["test"]
# Load multilingual model
model = SentenceTransformer('sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2')
# Build passage index from unique passages
passages = list({r["passage_id"]: r["passage_text"] for r in test_data}.values())
passage_embeddings = model.encode(passages, show_progress_bar=True)
# Evaluate retrieval
def retrieve(query, top_k=5):
query_emb = model.encode(query)
scores = np.dot(passage_embeddings, query_emb)
top_indices = np.argsort(scores)[-top_k:][::-1]
return [passages[i] for i in top_indices]
# Test with a query
results = retrieve(test_data[0]["query"])
print(f"Query: {test_data[0]['query']}")
print(f"Top result: {results[0][:200]}...")
Each record contains a query paired with its source passage:
{
"id": "r_abc123def456",
"query": "ما هي شروط الحصول على تأشيرة دخول السعودية؟",
"query_dialect": "msa",
"query_complexity": "simple",
"passage_id": "p_xyz789",
"passage_title": "تأشيرة السعودية",
"passage_text": "للحصول على تأشيرة دخول المملكة العربية السعودية...",
"source_url": "https://example.com/visa",
"source_category": "government"
}
| Field | Description |
|---|---|
id | Unique record identifier |
query | Arabic question answerable from the passage |
query_dialect | Dialect: msa, egyptian, gulf, levantine, maghrebi |
query_complexity | Complexity: simple, moderate, complex, multi_hop |
passage_id | Unique passage identifier |
passage_title | Title of the source document |
passage_text | Full passage text (query answer is within) |
source_url | Original source URL |
source_category | Category: wikipedia, government, healthcare, etc. |
The corpus covers diverse topics relevant to Arabic speakers:
| Category | Topics |
|---|---|
| Geography | Arab countries, major cities |
| History | Islamic history, regional history |
| Culture | Arabic literature, poetry |
| Science | AI, computing, technology |
| Health | Medical conditions, healthcare |
| Law | Legal systems, rights |
| Economy | Finance, investments |
Recommended metrics for benchmarking:
Retrieval:
Generation:
@dataset{arabicragb2025,
title={ArabicRAGB: Arabic Retrieval-Augmented Generation Benchmark},
author={Hesham Haroun},
year={2025},
publisher={Hugging Face},
url={https://huggingface.co/datasets/HeshamHaroon/ArabicRAGB}
}
This dataset is released under CC BY-SA 4.0.
ArabicRAGB: Arabic Retrieval-Augmented Generation Benchmark
13
14 commits
1 linked in READMEs
updated Dec 15, 2025
ArabicRAGB is a benchmark dataset for evaluating Retrieval-Augmented Generation (RAG) systems on Arabic language tasks. Each record contains a query-passage pair where the query is grounded in the passage content.
| Component | Count |
|---|---|
| Training Records | 10,530 |
| Test Records | 2,633 |
| Total Records | 13,163 |
| Dialect | Count |
|---|---|
| MSA | 4,012 |
| Egyptian | 3,999 |
| Gulf | 3,783 |
| Levantine | 697 |
| Maghrebi | 672 |
| Level | Count |
|---|---|
| Simple | 4,092 |
| Moderate | 4,030 |
| Complex | 4,215 |
| Multi-hop | 826 |
from datasets import load_dataset
# Load the dataset
dataset = load_dataset("HeshamHaroon/ArabicRAGB")
train_data = dataset["train"]
test_data = dataset["test"]
print(f"Train records: {len(train_data)}")
print(f"Test records: {len(test_data)}")
# Example record
example = train_data[0]
print(f"Query: {example['query']}")
print(f"Dialect: {example['query_dialect']}")
print(f"Passage: {example['passage_text'][:200]}...")
from datasets import load_dataset
from sentence_transformers import SentenceTransformer
import numpy as np
# Load dataset
dataset = load_dataset("HeshamHaroon/ArabicRAGB")
test_data = dataset["test"]
# Load multilingual model
model = SentenceTransformer('sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2')
# Build passage index from unique passages
passages = list({r["passage_id"]: r["passage_text"] for r in test_data}.values())
passage_embeddings = model.encode(passages, show_progress_bar=True)
# Evaluate retrieval
def retrieve(query, top_k=5):
query_emb = model.encode(query)
scores = np.dot(passage_embeddings, query_emb)
top_indices = np.argsort(scores)[-top_k:][::-1]
return [passages[i] for i in top_indices]
# Test with a query
results = retrieve(test_data[0]["query"])
print(f"Query: {test_data[0]['query']}")
print(f"Top result: {results[0][:200]}...")
Each record contains a query paired with its source passage:
{
"id": "r_abc123def456",
"query": "ما هي شروط الحصول على تأشيرة دخول السعودية؟",
"query_dialect": "msa",
"query_complexity": "simple",
"passage_id": "p_xyz789",
"passage_title": "تأشيرة السعودية",
"passage_text": "للحصول على تأشيرة دخول المملكة العربية السعودية...",
"source_url": "https://example.com/visa",
"source_category": "government"
}
| Field | Description |
|---|---|
id | Unique record identifier |
query | Arabic question answerable from the passage |
query_dialect | Dialect: msa, egyptian, gulf, levantine, maghrebi |
query_complexity | Complexity: simple, moderate, complex, multi_hop |
passage_id | Unique passage identifier |
passage_title | Title of the source document |
passage_text | Full passage text (query answer is within) |
source_url | Original source URL |
source_category | Category: wikipedia, government, healthcare, etc. |
The corpus covers diverse topics relevant to Arabic speakers:
| Category | Topics |
|---|---|
| Geography | Arab countries, major cities |
| History | Islamic history, regional history |
| Culture | Arabic literature, poetry |
| Science | AI, computing, technology |
| Health | Medical conditions, healthcare |
| Law | Legal systems, rights |
| Economy | Finance, investments |
Recommended metrics for benchmarking:
Retrieval:
Generation:
@dataset{arabicragb2025,
title={ArabicRAGB: Arabic Retrieval-Augmented Generation Benchmark},
author={Hesham Haroun},
year={2025},
publisher={Hugging Face},
url={https://huggingface.co/datasets/HeshamHaroon/ArabicRAGB}
}
This dataset is released under CC BY-SA 4.0.