LettuceDetect Grounded Hallucination Dataset

Token-level hallucination annotations on LLM responses grounded in structured context across five sources — source code, developer-tool output, academic papers, GitHub READMEs, and Wikipedia. Part of the LettuceDetect data collection.

Every sample pairs a grounded context with an LLM answer that is either correct or contains a minimally perturbed, character-span-annotated hallucination. All spans use one unified taxonomy, so the sources share a single label space and can be trained jointly or filtered apart via the dataset / context_modality fields.

Sources

Statistics

Per source:

Hallucination Taxonomy

Spans are labeled with the LettuceDetect unified taxonomy — three injectable top-level categories, each with subtypes. Every source maps into the same space.

Each span carries label (native source label, e.g. wrong_implementation, fabricated_api, or NUMERICAL), plus the unified category and subcategory.

Data Format

Each line in the JSONL files is one sample:

{
  "prompt": "User request: ...\n\n...grounded context...",
  "context": "...grounded context (passages / source files / tool output)...",
  "question": "the user request",
  "answer": "the LLM answer (hallucinated or clean)",
  "labels": [
    {"start": 18, "end": 25, "label": "NUMERICAL",
     "category": "contradiction", "subcategory": "numerical"}
  ],
  "split": "train",
  "task_type": "qa",
  "dataset": "lettucedetect-wikipedia",
  "language": "en",
  "context_modality": "markdown",
  "category": "contradiction",
  "subcategory": "numerical",
  "metadata": "{\"doc_id\": \"...\", \"is_hallucinated\": true, \"injector_model\": \"Qwen/Qwen3.6-35B-A3B\"}"
}

Fields:

• prompt — the full model input. The request is placed first (User request: {question}\n\n{context}) so it is never lost when a long context is truncated.
• context — the grounding evidence alone (passages / source files / tool output)
• question — the user request alone (null for summarization-style tasks)
• answer — the LLM response (hallucinated or clean)
• labels — annotated hallucination spans (empty for clean samples)
• category / subcategory — unified taxonomy fields
• dataset / context_modality — source discriminators
• metadata — a JSON-encoded string of source-specific provenance (keys vary by source: instance_id/hallucination_mode/answer_style for code, tool_type for tool output, paper_id for ACL, doc_id for readme/wiki). Parse with json.loads.

Per-source notes

• Code — a coding agent is given a developer request and repository context; the answer is the project's real fix, rendered as a patched function, a hunk, or an edit. Request-grounded mistakes are injected — wrong implementation, unrequested changes, and fabricated API references. References the answer makes but the truncated context lacks are grounded back in: repository definitions are resolved through the import graph at the base commit (a "Referenced definitions" block), and third-party APIs via Context7 (a "Library signatures" block), so a clean reference is never mistaken for a fabrication. Splits are repository-disjoint.
• Tool output — a short answer grounded in a tool's output; the hallucination misreports what the output shows.
• ACL — RAG setup: the context is the top-k retrieved paper chunks; injections are paper-specific (numerical/entity/relational/methodological/citational) and detectable against the excerpts. Split by paper (no leakage).
• README / Wikipedia — a document is chunked by heading; a question is generated per chunk, answered, and corrupted with a generic factual injection. Split by repo / article.

Usage

import json
from datasets import load_dataset

ds = load_dataset("KRLabsOrg/lettucedetect-code-hallucination")

# Everything, or filter by source / modality
code = ds["train"].filter(lambda r: r["dataset"] == "lettucedetect-code-agent")
markdown = ds["train"].filter(lambda r: r["context_modality"] == "markdown")

for sample in ds["train"]:
    if sample["labels"]:
        meta = json.loads(sample["metadata"])
        print(sample["category"], meta.get("is_hallucinated"))

With LettuceDetect

from lettucedetect.models.inference import HallucinationDetector

detector = HallucinationDetector(
    method="transformer",
    model_path="KRLabsOrg/lettucedetect-large-modernbert-en-v1",
)
spans = detector.predict(
    context=[sample["prompt"]], question="", answer=sample["answer"],
    output_format="spans",
)

Generation

Every source pairs a correct, grounded answer with an injected, localized hallucination, producing exact character-level spans (no diff alignment). For the code source the correct answer is the project's real fix; document sources (README, Wikipedia) additionally generate a question from the document; ACL uses retrieved chunks as context. Injection models, served via vLLM: google/gemma-4-31B-it for the code source and Qwen/Qwen3.6-35B-A3B for the others; the exact model is recorded per sample in metadata.

Limitations

README and Wikipedia are a first pass: on canonical topics a strong model could occasionally flag a hallucination from world knowledge rather than the context, and a minority of injected edits are detectable without grounding. The intended quality gate is the answer-only vs. answer+context detection gap; treat the markdown sources as lower-precision than the code/tool-output/ACL sources.

Test-set verification (code source)

Every sample of the lettucedetect-code-agent test split was individually reviewed before release (2,038 reviewed → 2,015 retained; 1,014 hallucinated / 1,001 clean). The protocol had three tiers:

1. Full first-pass review of every test sample against a written rubric (span validity, category, boundary tightness, explanation quality; fix-plausibility and mismatch checks for clean samples) — 92.9% of hallucinated samples accepted as labeled.
2. Blind second-pass adjudication of all flagged cases, re-derived from the raw data with no access to first-pass verdicts.
3. Evidence arbitration of disagreements against the true pre-fix sources: span text absent from the original repository ⇒ the answer introduced it ⇒ genuine hallucination.

Applied repairs: 235 span boundaries tightened to the minimal hallucinated substring, 23 invalid spans dropped, 2 categories corrected, 5 samples reclassified clean, 23 samples removed (question/answer mismatch, no-op edits, or incoherent renders). No rebalancing was applied after review — the 50.3% test ratio is the natural outcome of verification. The train and validation splits are machine-generated with automated quality gates but are not individually reviewed. This is model-assisted expert annotation, not independent human annotation.

Verification artifacts

The full test-set review is auditable. The per-sample verdicts, the contested cases with their evidence-based resolutions, and the annotator rubric are released under annotations/: REPORT.md, RUBRIC.md, verdicts/verdicts_{clean,hall}_*.jsonl, and contested_for_human_review.json / contested_source_check.json.

Citation

@misc{kovács2026documentgroundingspanlevelhallucination,
      title={Beyond Document Grounding: Span-Level Hallucination Detection over Code, Tool Output, and Documents}, 
      author={Ádám Kovács and Bowei He and Xue Liu and István Boros and Szilveszter Tóth and Gábor Recski},
      year={2026},
      eprint={2607.00895},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2607.00895}, 
}

License

CC-BY-4.0