One reproducible agreement number between ISNAD's weakest-link chain grading and a rule-based grading convention derived from Ibn Hajar's 12 narrator tiers.
This is derived output, not a re-host of the source data. The underlying
dataset is emadjumaah/hadith-kg
(CC-BY-4.0). This dataset contains, per chain, the dataset's chain-verdict field (hukum), ISNAD's
predicted chain grade, and the principled disagreement bucket — computed by the
exact functions that produce the benchmark's κ.
ISNAD's weakest-link chain grading conforms to the dataset's rule-based chain-verdict convention (derived from Ibn Hajar's 12 narrator tiers) at Cohen's κ = 0.871 (strict default), across 575,060 chains graded by a rule-based convention (Ibn Hajar's 12 narrator tiers), with a shuffled-rank control at κ = −0.0066.
ISNAD conforms to the rule-based convention; it is not "better than the scholars". The narrator-grade agreement — how contested the underlying tiers are — is κ = 0.331 (critic-vs-critic, context). ISNAD's 0.871 means the pre-committed tier→grade mapping conforms to the dataset's own convention; the 0.331 is narrator-grade context, not a ceiling.
| Model / method | κ (strict) | Narrator-grade agreement (context) | Source |
|---|---|---|---|
| ISNAD weakest-link (strict) | 0.871 | 0.331 | this dataset · bench/docs/RESULTS.md |
| ISNAD weakest-link (lenient) | 0.761 | 0.331 | opt-in lenient_unknown=True |
| shuffled-rank control | -0.0066 | — | negative control |
| majority-class control | 0.000 | — | negative control |
This measures chain-grade (isnād) agreement only — not matn (content) verdicts, and not hadith authenticity. "Mawḍūʿ" here is keyed on COMPROMISED integrity (proven liar/fabricator, rank 12); "ḍaʿīf jiddan" (very weak) on SUSPECT (accused/abandoned, ranks 10–11). Both still quarantine.
| Field | Value |
|---|---|
| Derived from | emadjumaah/hadith-kg (CC-BY-4.0) |
| Source SHA-256 | d528084321e715006712e0e2461809a3afc9408065a1d1af90238c8b723815a6 |
| Mapping | bench/docs/mapping.md (pre-committed; mapping v2 is post-hoc after the expert review) — mapping v2 (integrity-keyed mawḍūʿ / ḍaʿīf jiddan split, 3.0.0) |
| Reproduction | uv run python -m bench.run --seed 0 |
| Software | pip install isnad (Apache-2.0) |
| Paper | arXiv:2607.24117 · DOI 10.48550/arXiv.2607.24117 |
The 12-tier → ISNAD grade mapping is the author's best-effort reading of Ibn Ḥajar's Taqrīb al-Tahdhīb tiers. The project has no external domain reviewer yet; ranks 6–7 and 10–12 are the rows most likely to need a scholar's correction. Any correction is a new mapping version with a new, separately reported number — never a silent edit.
One JSON object per line, with no leading comment/header line (JSONL has no comment syntax). Schema:
sanad_id, hukum, true_grade, predicted_grade, disagreement_bucket, is_complete, has_gap, has_taliq, narrator_rank_nos, mode
One reproducible agreement number between ISNAD's weakest-link chain grading and a rule-based grading convention derived from Ibn Hajar's 12 narrator tiers.
This is derived output, not a re-host of the source data. The underlying
dataset is emadjumaah/hadith-kg
(CC-BY-4.0). This dataset contains, per chain, the dataset's chain-verdict field (hukum), ISNAD's
predicted chain grade, and the principled disagreement bucket — computed by the
exact functions that produce the benchmark's κ.
ISNAD's weakest-link chain grading conforms to the dataset's rule-based chain-verdict convention (derived from Ibn Hajar's 12 narrator tiers) at Cohen's κ = 0.871 (strict default), across 575,060 chains graded by a rule-based convention (Ibn Hajar's 12 narrator tiers), with a shuffled-rank control at κ = −0.0066.
ISNAD conforms to the rule-based convention; it is not "better than the scholars". The narrator-grade agreement — how contested the underlying tiers are — is κ = 0.331 (critic-vs-critic, context). ISNAD's 0.871 means the pre-committed tier→grade mapping conforms to the dataset's own convention; the 0.331 is narrator-grade context, not a ceiling.
| Model / method | κ (strict) | Narrator-grade agreement (context) | Source |
|---|---|---|---|
| ISNAD weakest-link (strict) | 0.871 | 0.331 | this dataset · bench/docs/RESULTS.md |
| ISNAD weakest-link (lenient) | 0.761 | 0.331 | opt-in lenient_unknown=True |
| shuffled-rank control | -0.0066 | — | negative control |
| majority-class control | 0.000 | — | negative control |
This measures chain-grade (isnād) agreement only — not matn (content) verdicts, and not hadith authenticity. "Mawḍūʿ" here is keyed on COMPROMISED integrity (proven liar/fabricator, rank 12); "ḍaʿīf jiddan" (very weak) on SUSPECT (accused/abandoned, ranks 10–11). Both still quarantine.
| Field | Value |
|---|---|
| Derived from | emadjumaah/hadith-kg (CC-BY-4.0) |
| Source SHA-256 | d528084321e715006712e0e2461809a3afc9408065a1d1af90238c8b723815a6 |
| Mapping | bench/docs/mapping.md (pre-committed; mapping v2 is post-hoc after the expert review) — mapping v2 (integrity-keyed mawḍūʿ / ḍaʿīf jiddan split, 3.0.0) |
| Reproduction | uv run python -m bench.run --seed 0 |
| Software | pip install isnad (Apache-2.0) |
| Paper | arXiv:2607.24117 · DOI 10.48550/arXiv.2607.24117 |
The 12-tier → ISNAD grade mapping is the author's best-effort reading of Ibn Ḥajar's Taqrīb al-Tahdhīb tiers. The project has no external domain reviewer yet; ranks 6–7 and 10–12 are the rows most likely to need a scholar's correction. Any correction is a new mapping version with a new, separately reported number — never a silent edit.
One JSON object per line, with no leading comment/header line (JSONL has no comment syntax). Schema:
sanad_id, hukum, true_grade, predicted_grade, disagreement_bucket, is_complete, has_gap, has_taliq, narrator_rank_nos, mode
