Verbatim Spans

A multi-domain training dataset for query-conditioned extractive evidence selection. Given a question and a passage, the task is to highlight the verbatim substrings of the passage that support the answer.

Combines three sources covering distinct domains and annotation conventions:

The dataset is designed for training a generic span-highlighter encoder — the intended model is a ModernBERT token classifier. For the specialized ACL-only benchmark see KRLabsOrg/acl-verbatim-spans.

Configs

canonical

One row per (question, chunk) pair, with raw text. Use this config if you want to train with your own tokenizer or inspect rows.

Fields:

encoder

Pretokenized, ready for direct training with answerdotai/ModernBERT-base or a compatible ModernBERT checkpoint.

Fields: input_ids, attention_mask, labels.

Pretokenization settings:

If you want to train with a different tokenizer, rebuild from canonical:

python acl_verbatim/span_training/prepare_token_cls_dataset.py \
  --input-file <canonical_train.jsonl> \
  --output-file train.my_tokenizer.binary.jsonl \
  --tokenizer <your-tokenizer> \
  --label-scheme binary \
  --drop-unlabeled-positives

Composition details

RAGBench cap: 15,000 rows per config for train, 2,000 per config for validation, random seed 1337. This balances the 12 RAGBench configs so that high-volume configs (tatqa, pubmedqa, finqa) do not dominate the mix. Without capping, tatqa + pubmedqa alone would be ~70% of the training data.

No RAGBench test split included. The RAGBench test split is reserved for downstream evaluation.

Squeez: all train and validation rows are included. The negative/positive split (~2:1) is preserved as-is; negatives are important signal for teaching the model when not to fire.

ACL silver: taken from the caption-preserving split released in KRLabsOrg/acl-verbatim-spans.

Intended use

Training a generic query-conditioned token classifier for evidence highlighting across diverse RAG / extractive-QA use cases. The associated model is released separately once trained.

Evaluation against the human-annotated ACL gold benchmark is handled by the sibling dataset KRLabsOrg/acl-verbatim-spans (canonical/test split).

Limitations

• All labels are LLM-produced (Qwen for ACL, GPT-4o for RAGBench, silver-style GLiNER output for Squeez) — not strictly human-annotated.
• Evidence conventions vary across sources: a model trained on this mix will learn an average of sentence / paragraph / code-block scales rather than a single consistent convention.
• RAGBench domains are English only; Squeez is code + tool output; ACL is scientific prose. Transfer to other languages or domains (e.g. bilingual use-cases or spoken dialogue) is not evaluated.
• The GLiNER-style Squeez format uses a single entity type RELEVANT; span boundaries were produced by a GLiNER-trained model rather than human annotators.

Licensing and attribution

Apache 2.0. All three source datasets are Apache 2.0:

• ACL silver: KRLabs Org — derived from the ACL Anthology corpus (KRLabsOrg/acl-anthology-md)
• RAGBench: Galileo Labs — galileo-ai/ragbench
• Squeez: KRLabs Org — KRLabsOrg/tool-output-extraction-swebench-gliner

Reproducibility

All preparation scripts live in KRLabsOrg/acl-verbatim:

• scripts/prepare_ragbench_spans.py — RAGBench → canonical spans
• scripts/prepare_squeez_spans.py — Squeez GLiNER format → canonical spans
• scripts/build_generic_spans_dataset.py — caps + merges + pushes to HF (random seed 1337 for the RAGBench cap)

Citation

@misc{Recski:2026,
      title={ACL-Verbatim: hallucination-free question answering for research}, 
      author={Gábor Recski and Szilveszter Tóth and Nadia Verdha and István Boros and Ádám Kovács},
      year={2026},
      eprint={2605.21102},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2605.21102}, 
}