ACL-Verbatim Span Dataset

KRLabsOrg/acl-verbatim-spans is a dataset for query-conditioned extractive evidence selection over papers from the ACL Anthology.

The release combines:

• a gold test benchmark with manual span annotations
• a larger silver training set produced from synthetic questions, retrieval, and LLM-based span annotation
• an encoder-ready config for training token-classification models directly

The underlying document collection is KRLabsOrg/acl-anthology-md.

What This Dataset Is For

This dataset is intended for systems that, given a question and a retrieved paper chunk, must identify the supporting evidence verbatim in the chunk.

Typical uses include:

• training token classifiers for semantic highlighting
• evaluating span extractors and evidence selectors
• comparing LLM teachers, token-level students, and sentence-selection baselines
• studying paragraph-scale evidence extraction in scientific text

Configs

canonical

One row per (question, chunk) pair.

• train and validation are silver supervision
• test is the manual gold benchmark

This is the main config for analysis, evaluation, and downstream reuse.

Current split sizes:

Expected fields:

encoder

Token-classification-ready rows derived from the silver canonical data by tokenization and windowing.

This config is intended for direct encoder training with Hugging Face transformers.

Current split sizes:

Expected fields:

How the encoder split was built

The encoder split was pretokenized with:

If you want to train with a different tokenizer or label scheme, rebuild from the canonical config:

python acl_verbatim/span_training/prepare_token_cls_dataset.py \
  --input-file <canonical_train.jsonl> \
  --output-file train.my_tokenizer.binary.jsonl \
  --tokenizer <your-tokenizer> \
  --label-scheme binary \
  --drop-unlabeled-positives

Evaluation always uses canonical/test (raw text); the encoder config intentionally does not include a test split, because span-level scoring needs to map token predictions back to character offsets in the original chunk, which requires retokenizing at inference time.

Annotation Convention

The benchmark uses paragraph-oriented evidence annotation rather than minimal SQuAD-style answer spans.

Important consequences:

• broader supporting passages are often preferred over minimal snippets
• tables, figure captions, and other structured evidence are considered valid positives
• bibliography/reference sections are generally out of scope as positive evidence

Gold Benchmark Summary

The canonical/test split is the manually annotated benchmark used for extractor evaluation.

Silver Training Data

The silver training data was produced by:

1. sampling papers from the ACL Anthology corpus
2. generating synthetic questions from paper chunks
3. rewriting those questions into retrieval-style queries
4. retrieving top-ranked chunks
5. annotating answer-bearing chunks with an LLM span extractor
6. filtering noisy positives while preserving table/caption evidence

Intended Uses

• Train query-conditioned token classifiers for semantic highlighting
• Compare LLM teachers, token models, sentence-level compressors, and semantic highlighters under one span-scoring harness
• Study boundary conventions for evidence extraction in scientific text

Limitations

• The gold benchmark is small and single-annotator
• Silver labels inherit retrieval noise and teacher noise
• Tables and captions are represented through markdown/caption text, not full table structure
• The benchmark reflects a paragraph-oriented annotation convention rather than a strict minimal answer-span convention

How To Load The Dataset

Load the canonical config:

from datasets import load_dataset

ds = load_dataset("KRLabsOrg/acl-verbatim-spans", "canonical")
train = ds["train"]
dev = ds["validation"]
test = ds["test"]

Load encoder-ready training rows:

from datasets import load_dataset

encoder = load_dataset("KRLabsOrg/acl-verbatim-spans", "encoder")
train = encoder["train"]
dev = encoder["validation"]

Example: inspect gold vs silver splits

from datasets import load_dataset

canonical = load_dataset("KRLabsOrg/acl-verbatim-spans", "canonical")

silver_train = canonical["train"]
silver_dev = canonical["validation"]
gold_test = canonical["test"]

Related Resources

• Raw ACL Anthology markdown corpus: KRLabsOrg/acl-anthology-md
• Codebase: KRLabsOrg/acl-verbatim

Citation

@misc{Recski:2026,
      title={ACL-Verbatim: hallucination-free question answering for research}, 
      author={Gábor Recski and Szilveszter Tóth and Nadia Verdha and István Boros and Ádám Kovács},
      year={2026},
      eprint={2605.21102},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2605.21102}, 
}