AZ-Eval

An open parallel Azerbaijani–English benchmark of 356 human-verified short-answer items. Every item is stated in both languages, so a model's Azerbaijani performance can be compared against its own English performance on the same fact — separating what a model knows from what it can express in Azerbaijani.

To our knowledge, no open parallel evaluation set for Azerbaijani existed before this one.

• 📄 Code & full write-up: https://github.com/nihatgaribli/AZ-Eval
• 🔖 Cite this exact version (v0.1.0, n = 356): 10.5281/zenodo.22050777
• 🔖 Latest version (concept DOI): 10.5281/zenodo.22050776

What it was built to measure

Kazakh is Azerbaijani's closest well-resourced Turkic relative. Does adapting a model to Kazakh carry over to Azerbaijani? On this benchmark it does not — it actively harms.

Exact match, strict normalization, Holm-corrected p = 0.0024.

On a control stratum of universally known facts (n = 96) the two models are statistically indistinguishable in English (59.4% vs 59.4%, p = 1.0000), while the Azerbaijani gap reaches 35.4 points — ruling out a general capability difference.

The mechanism is largely orthographic: 326 of 356 Azerbaijani answers come back in Cyrillic (1 of 356 for the base model). Under transliteration the control-stratum gap loses significance entirely, though a 5.9-point residual remains across the full set.

Fields

Composition

356 items, single test split — this is an evaluation set, not training data.

255 items were written by hand by a native speaker; 101 were generated from templated Wikidata queries. All 356 passed human verification.

Why the alias lists matter

Azerbaijani is agglutinative: the same answer appears as Paris, Parisin, Parisə, Parisdə, Parisdən and so on depending on the case the question induces. Scoring exact match against a single surface form would measure morphology, not knowledge. Each item therefore carries a rule-generated alias set.

The accompanying pipeline scores under four nested normalizations that separately isolate morphology, diacritics and writing system, which is what makes the orthographic mechanism above visible rather than merely suspected.

Usage

from datasets import load_dataset

ds = load_dataset("nihatgaribli/az-eval", split="test")
print(ds[0]["question_az"], "->", ds[0]["answer"])

The evaluation harness, the four normalizations, the statistical machinery (paired bootstrap CIs, sign-flipping permutation tests, Holm correction) and every result table live in the GitHub repository.

Limitations

• Factual short-answer QA only. No reasoning, no generation, no long-form tasks.
• 356 items; the cleanest control stratum is 96. Effects reported are large, but the set is small.
• Facts derive from Wikidata and Azerbaijani Wikipedia, so coverage inherits their biases.
• medium dominates the difficulty distribution (344 of 356); difficulty is a coarse label.

Licence

CC BY 4.0. Source facts derive from Wikidata, released under CC0 1.0. The accompanying code is MIT.

Citation

@dataset{garibli_2026_azeval,
  author    = {Garibli, Nihat},
  title     = {AZ-Eval: a parallel Azerbaijani--English benchmark},
  year      = {2026},
  publisher = {Zenodo},
  version   = {v0.1.0},
  doi       = {10.5281/zenodo.22050777},
  url       = {https://doi.org/10.5281/zenodo.22050777}
}