Kaz-RAG-Search-Benchmark

Evidence-based benchmark for Kazakh information retrieval — the independent proof base for the Kazakh Stemmer.

Corpus: 8,370 passages from Kazakh Wikipedia Queries: 300 queries × 3 categories (natural / inflected / vocabulary-gap) Format: BEIR-compatible — three subsets: corpus, queries, qrels

Browse the data: use the subset switcher at the top of the Data Studio viewer to move between corpus (Kazakh passages), queries (the 300 questions), and qrels (relevance judgements). The default view is the corpus.

Key result

A Kazakh morphological stemmer significantly improves lexical search: +16% nDCG@10 on inflected queries (p=0.0017) and +9% overall (p=0.0001), on 300 queries with paired-bootstrap significance (10k resamples). BM25+stemmer is also the most balanced retriever overall and beats Dense LaBSE on every category (nDCG@10 0.754 vs 0.481).

Full comparison (n=300, nDCG@10)

• E5 is the strongest dense model; Granite-278m collapses on vocabulary-gap (0.303) — worse than a plain BM25 baseline — exposing a synonym blind spot in dense retrieval on Kazakh.
• BM25 + stemmer never drops below 0.727 on any slice: the most balanced single system.

End-to-end RAG (Qwen2.5-7B, n=300) is reported honestly: the stemmer raises retrieval hit@3 (0.737 → 0.803), but the end-to-end accuracy gain is not statistically significant (McNemar p=0.63) — the bottleneck is the Kazakh-language generator, not the retriever. The stemmer's value is proven at the retrieval level.

See github.com/Tim2190/Kaz-RAG-search-benchmark for full results, code, and reproduction instructions.

Broader benchmark: 14 models, two domains

This dataset is the in-domain (Wikipedia) half of a larger evaluation. The full study now covers 14 embedding systems across two domains — this Wikipedia set (in-domain) and an out-of-domain set of Akorda formal speeches — with a second preprint on hybrid retrieval and out-of-domain robustness.

(top 5 shown — full 14-system table with hybrids and significance in the repo)

Headline findings:

• Good tokenization is a necessary floor, not a guarantee: byte-level fallback (Cohere embed-v4, Qwen3) and [UNK] collapse (Nomic) wreck Kazakh retrieval regardless of model size.
• Strong Wikipedia scores do not guarantee robustness on formal Kazakh — Cohere embed-v4 ranks #3 in-domain but drops to 10th out-of-domain (largest cross-domain fall in the benchmark).
• Naive Kazakh fine-tunes of e5 (kazakh-e5, KazEmbed-V5) underperform the base model on both domains.

➡️ Full leaderboard (14 systems, hybrids, significance): github.com/Tim2190/Kaz-RAG-search-benchmark

📄 Second preprint (hybrid retrieval & OOD robustness): https://doi.org/10.5281/zenodo.20781386

Source & license

The corpus is derived from Kazakh Wikipedia (the wikimedia/wikipedia dump on Hugging Face), licensed under CC BY-SA 4.0. This dataset inherits the same license: attribute the source and share derivatives alike.

Usage

from datasets import load_dataset

corpus  = load_dataset("Tim2190/kaz-rag-search-benchmark", "corpus",  split="corpus")
queries = load_dataset("Tim2190/kaz-rag-search-benchmark", "queries", split="queries")
qrels   = load_dataset("Tim2190/kaz-rag-search-benchmark", "qrels",   split="test")

Citation

This dataset accompanies a preprint archived on Zenodo with a permanent DOI. If you use the benchmark, please cite:

Seidalin, T. (2026). Morphology Beats Multilingual Embeddings for Kazakh Retrieval: A 300-Query Benchmark with Honest Negative Results. Zenodo. https://doi.org/10.5281/zenodo.20605663

@misc{seidalin2026kazakh,
  author = {Seidalin, Timur},
  title  = {Morphology Beats Multilingual Embeddings for Kazakh Retrieval: A 300-Query Benchmark with Honest Negative Results},
  year   = {2026},
  publisher = {Zenodo},
  doi    = {10.5281/zenodo.20605663},
  url    = {https://doi.org/10.5281/zenodo.20605663}
}

Links

• Preprint (Zenodo, DOI): https://doi.org/10.5281/zenodo.20605663
• Second preprint (hybrid & OOD, Zenodo): https://doi.org/10.5281/zenodo.20781386
• Code & full results (GitHub): https://github.com/Tim2190/Kaz-RAG-search-benchmark
• Kazakh Stemmer: https://qaz-api.vercel.app/