RAG Claim Verification — Jev 130K

Does the answer actually follow from its evidence?

130,000 typed RAG decisions connecting source-derived reference labels with actual jev-1.13.0 predictions. Inspect individual claims, whole-response grounding, hallucination types and response-format defects without losing the context behind a decision.

The useful part is not just agreement: 21,181 teacher/reference disagreements are preserved for inspection, including 1,265 categorical decisions with provider-reported confidence of at least 0.90.

General-domain RAG research, not a clinical corpus or a production truth oracle. Jev predictions and source references are separate columns. Neither is silently substituted for the other. This is not a context-held-out benchmark.

Documentation reviewed: 17 October 2026. This refresh adds runnable workflows, label-balance information and a context-overlap audit. The Parquet data and model answers are unchanged; Jev responses were collected on 21 September 2026.

At a glance

These are 130,000 derived decisions, not 130,000 newly human-annotated answers. Several questions and sentence-level claims can come from one upstream response; several upstream responses can share the same query and evidence context.

What can you do with it?

• Inspect grounding failures: compare a candidate claim with the supplied context.
• Prototype evidence-checking classifiers: use explicit categorical labels and bounded true/false scores rather than open-ended judge prose, subject to usage rights.
• Study teacher/reference disagreement: find annotation-derived tasks an automated verifier agrees with, misses, or flags confidently.
• Explore confidence and review thresholds: assess behavior after appropriate regrouping; do not assume the supplied scores are calibrated.

This is an inspectable research and error-analysis corpus, not evidence that a downstream model is safe or production-ready. Training uses remain subject to the applicable source and provider terms.

Quick start: inspect one claim

Install datasets. While this repository is private, authenticate with an account that has access using hf auth login. Once public, this example needs no login. Never paste an access token into a shared notebook.

import json
from datasets import load_dataset

dataset = load_dataset(
    "MaziyarPanahi/Jev-RAG-Claim-Verification-130K",
    streaming=True,
)
claims = dataset["train"].filter(
    lambda row: row["task_family"] == "claim_span_grounding"
)
row = next(iter(claims))
state = json.loads(row["state_json"])

print("Claim:", state["candidate_claim"])
print("Context:", state["authoritative_context"])
print("Reference:", row["gold_label"])
print("Jev:", row["jev_prediction"])
print("Distribution:", json.loads(row["jev_probabilities_json"]))

state_preview is shortened. Use state_json for the full retained state and request_json for the exact model-visible request. Streaming avoids eagerly materializing roughly 940 MB of repeated strings in Arrow memory.

Find disagreements across both decision types

def disagrees(row):
    if row["primitive"] == "choice":
        return row["jev_prediction"] != row["gold_label"]
    # Reproduce the descriptive audit's 0.5 rule for bounded true/false scores.
    return (float(row["jev_prediction"]) >= 0.5) != (row["gold_score"] >= 0.5)

disagreements = dataset["train"].filter(disagrees)
example = next(iter(disagreements))
print(example["task_family"], example["question_text"])
print(example["gold_label"], example["jev_prediction"])

Inspect the context and annotation provenance before calling a disagreement a model error. Keep gold_*, label_source, jev_* and other answer/provenance fields out of a detector's input features to avoid accidental label leakage.

Four tasks, with different meanings

choice contributes 93,646 categorical decisions with a declared label set. noul contributes 36,354 true/false scores in [0, 1]; it is not an abstention label or demonstrated calibrated probability. The audit thresholds it at 0.5.

For claims, no_annotated_hallucination means no marked span overlaps the sentence, not independently proven truth. For response quality, good means no truncation/incorrect-refusal defect was marked, not that the answer is fully grounded. Read the task-specific criteria_json.

Coverage

Reference-label balance

These tasks are not class-balanced. Response quality is overwhelmingly good; an impressive-looking aggregate agreement number can hide that imbalance.

Splits: what is—and is not—kept apart

These are derived release splits, not the canonical upstream RAGTruth train/test evaluation protocol.

No group_id or upstream response-record identity crosses splits. However, 349 of the 2,019 query/context pairs occur in more than one split, and 67 exact serialized-request identities cross splits. The existing split repair grouped response records, not every shared underlying document/request.

Do not describe these splits as leakage-free, document-held-out or an independent benchmark. For a new train/evaluation study, rebuild splits using connected components linking shared query/context pairs and identical requests, keeping all decisions from a response together. Do not randomly split individual decision rows.

Exact-request duplicates are retained: 749 duplicate groups account for 944 rows beyond the first occurrence. Keep them for repeated-inference/annotation analysis, or deduplicate carefully for your study; inspect conflicting references rather than arbitrarily treating one as authoritative.

Source lineage and what was generated

All 130,000 rows derive from wandb/RAGTruth-processed, pinned to revision eb4f4b9d1b68eb7092d3e1a61c0cd82d9808737b. Credit the original RAGTruth project and its authors for the corpus/annotations, and the processed-data maintainers for the conversion.

• Source-derived gold_* fields preserve or transform upstream annotations: sentence/span overlap, hallucination presence, or response quality.
• Jev answered the constructed decision requests. Its outputs live in jev_*.
• Jev did not author the original reference annotations or all source responses. This release adds typed decision structure and a separate model-behavior layer, not a new human annotation campaign.
• The builder allows context up to 24,000 characters, responses up to 18,000 and the first ten eligible sentence spans per response. The 17 October review found no retained state field carrying the builder's truncation marker in this slice.

This is general-domain material, not curated clinical data. Source text may still mention health, politics, personal information or other sensitive topics.

Descriptive teacher/reference audit

Jev agrees with the transformed source references on 108,819 / 130,000 decisions (83.71%) and disagrees on 21,181. This is construction-level agreement, not externally validated model accuracy or production performance.

The last column describes the majority label over this complete slice, not a fitted or held-out baseline. It shows why agreement alone is insufficient. The 1,265 high-confidence disagreements use jev_confidence on categorical rows only. Bounded-score rows do not have that confidence field, and provider confidence is not necessarily the selected-label probability.

Recorded usage: 153,317,045 input tokens and 5,630,336 output tokens. Historical estimated input cost: $6.439315890 at the collector's rate, not a current pricing quote or complete billing statement. No new model calls were made for this refresh.

Columns and provenance

The provider request-ID field is empty on all 130,000 rows because the collector did not receive that metadata. Request/response bodies remain available; do not describe them as provider-ID-traceable billing receipts.

The review rechecked all row IDs, local shard hashes, task/split counts, raw versus normalized answers, response grouping, request reuse and context overlap. Zero terminal jev_error rows were found. Structural integrity does not establish that every source annotation or teacher answer is correct.

Limitations and rights

• Source annotations may be incomplete/noisy. No marked hallucination does not establish open-world truth, completeness or absence of subtle unsupported claims.
• Shared contexts, repeated requests, derived decisions and class imbalance limit naive training/evaluation interpretations.
• Scores and confidence are uncalibrated. Source/public data may already occur in model pretraining; this is not a contamination-free benchmark.
• This does not replace retrieval evaluation, end-to-end RAG evaluation, independent human review or production guardrails.
• The aggregate remains license: other. Rows record mit-with-upstream-review-required for the seed. The original RAGTruth project publishes an MIT licence (Copyright 2023 Particle Media), but third-party text/model outputs and applicable Jev/provider conditions require separate consideration. This card grants no additional redistribution or model-training rights.

If you use this release, credit the upstream authors/processed-data maintainers and this derived decision dataset, and retain applicable notices.