SHELF: Synthetic Harness for Evaluating LLM Fitness

SHELF is a synthetic benchmark for evaluating language model fitness on bibliographic classification, retrieval, and clustering tasks using Library of Congress taxonomies.

Dataset Description

• Homepage: https://github.com/mjbommar/shelf-benchmark
• Repository: https://github.com/mjbommar/shelf-benchmark
• Paper: arXiv:2609.03047
• License: cc-by-4.0
• Release: 62,899-document aggregate corpus

Dataset Summary

SHELF contains 62,899 synthetic documents annotated with Library of Congress taxonomies. The primary all configuration contains the complete corpus. Separate Project Gutenberg and LCSHBench configurations are natural-text transfer controls and are never pooled into SHELF.

• LCC (Library of Congress Classification): 21 subject classes (A-Z)
• LCGFT (Library of Congress Genre/Form Terms): 14 categories, 133 specific forms
• Topics: 112 subject headings (multi-label)
• Geographic: 44 locations mapped to 8 regions (multi-label)
• Audience: 25 target audience types
• Register: 8 writing styles (academic, professional, casual, etc.)

The dataset is designed for:

1. Document Classification - Predicting LCC codes, LCGFT forms, topics, audiences
2. Document Retrieval - Finding similar documents by subject, genre, or topic
3. Document Clustering - Grouping documents by subject, genre, or geographic region
4. Pair Classification - Determining if document pairs share categories

Supported Tasks

Languages

English only.

Dataset Structure

Data Instances

{
  "id": "20251211_123456_abcd1234",
  "title": "Introduction to Machine Learning",
  "body": "This comprehensive guide covers the fundamentals of machine learning...",
  "word_count": 450,
  "lcc_code": "Q",
  "lcc_name": "Science",
  "lcc_uri": "http://id.loc.gov/authorities/classification/Q",
  "lcgft_category": "Instructional and educational works",
  "lcgft_form": "Textbooks",
  "topics": ["Computer science", "Artificial intelligence"],
  "geographic": ["United States"],
  "audience": "Students",
  "register": "academic",
  "register_description": "academic and scholarly, suitable for research contexts",
  "target_length": "medium",
  "target_word_range": [300, 500]
}

Data Fields

Data Splits

The all configuration preserves each document's source split. Its original 42,532-document component uses stratified document-level splits. The later components carry recorded specifications and use stratified splits grouped by spec_id, so all realizations of one specification stay together. The specification-level leakage guarantee therefore applies to the later components, not to all as a whole.

Dataset Creation

Source Data

Documents were generated by 25 writing models. A generator-balanced factorial component contains 15 models from Anthropic, OpenAI, Google, Alibaba, DeepSeek, Zhipu, Moonshot, MiniMax, Meta, Mistral, and xAI. The aggregate corpus is not generator-balanced: its largest writing model supplies 47.7% of documents. In the factorial component, the largest share is 9.24%.

The generation process ensures:

• Balanced distribution across all 21 LCC codes
• Coverage of all 133 LCGFT forms
• Diverse topics, audiences, and registers
• Varied document lengths (12 to 6,000+ words)
• Multi-model diversity to reduce single-model biases

Quality Filtering

The following quality filters were applied:

• Empty document removal
• Non-English content detection and removal
• Length validation against target ranges

Annotations

All annotations are generated alongside the documents using structured prompting. Labels represent the intended classification as specified in the generation prompt.

Usage

Loading the Dataset

from datasets import load_dataset

# Load the complete 62,899-document corpus
dataset = load_dataset("mjbommar/SHELF", "all")

# Load specific split
train = load_dataset("mjbommar/SHELF", "all", split="train")
test = load_dataset("mjbommar/SHELF", "all", split="test")

# Access examples
print(train[0])

Dataset Configurations

The dataset has multiple configurations:

# Load pair classification data
lcc_pairs = load_dataset("mjbommar/SHELF", name="same_lcc_pairs")
form_pairs = load_dataset("mjbommar/SHELF", name="same_form_pairs")
audience_pairs = load_dataset("mjbommar/SHELF", name="same_audience_pairs")
register_pairs = load_dataset("mjbommar/SHELF", name="same_register_pairs")

# Load topic overlap pairs
topic_binary = load_dataset("mjbommar/SHELF", name="same_topic_pairs")
topic_graded = load_dataset("mjbommar/SHELF", name="topic_overlap_pairs")

# Pair format (categorical pairs)
print(lcc_pairs["train"][0])
# {'id': 'pair_000001', 'doc_a_id': '...', 'doc_a_title': '...',
#   'doc_a_body': '...', 'doc_b_id': '...', 'doc_b_title': '...',
#   'doc_b_body': '...', 'label': 1, 'label_field': 'lcc_code'}

# Topic overlap format (multi-label pairs)
print(topic_graded["train"][0])
# {'id': 'pair_000001', 'doc_a_id': '...', 'doc_a_title': '...',
#   'doc_a_body': '...', 'doc_b_id': '...', 'doc_b_title': '...',
#   'doc_b_body': '...', 'label': 2, 'overlap_count': 2,
#   'shared_topics': ['Ethics', 'Philosophy']}

Classification Example

from datasets import load_dataset
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report

# Load data
dataset = load_dataset("mjbommar/SHELF")

# Prepare for LCC classification
X_train = dataset["train"]["body"]
y_train = dataset["train"]["lcc_code"]
X_test = dataset["test"]["body"]
y_test = dataset["test"]["lcc_code"]

# Train simple baseline
vectorizer = TfidfVectorizer(max_features=10000)
X_train_vec = vectorizer.fit_transform(X_train)
X_test_vec = vectorizer.transform(X_test)

clf = LogisticRegression(max_iter=1000)
clf.fit(X_train_vec, y_train)

# Evaluate
y_pred = clf.predict(X_test_vec)
print(classification_report(y_test, y_pred))

Retrieval Example

from datasets import load_dataset
from sentence_transformers import SentenceTransformer
import numpy as np

# Load data
dataset = load_dataset("mjbommar/SHELF")
corpus = dataset["train"]["body"]

# Encode with sentence transformer
model = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = model.encode(corpus, show_progress_bar=True)

# Query
query = "Introduction to constitutional law"
query_emb = model.encode(query)

# Find similar documents
similarities = np.dot(embeddings, query_emb)
top_k = np.argsort(similarities)[-5:][::-1]

for idx in top_k:
    print(f"{similarities[idx]:.3f}: {dataset['train']['title'][idx]}")

Pair Classification Example

from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

# Load pair data
pairs = load_dataset("mjbommar/SHELF", name="same_lcc_pairs")

# Prepare inputs (concatenate doc_a and doc_b)
def format_pair(example):
    text_a = f"{example['doc_a_title']} {example['doc_a_body']}"
    text_b = f"{example['doc_b_title']} {example['doc_b_body']}"
    return {"text": f"{text_a} [SEP] {text_b}", "label": example["label"]}

train_data = pairs["train"].map(format_pair)

# Fine-tune a model (simplified)
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased", num_labels=2)

# ... training code ...

Considerations for Using the Data

Social Impact

This dataset is intended for research and development of document classification systems. It may help improve:

• Library cataloging automation
• Document organization systems
• Research paper classification
• Content recommendation systems

Limitations

• Synthetic Data: Documents are AI-generated and may not perfectly reflect real-world document distributions
• English Only: Currently limited to English language documents
• Generator Imbalance: The aggregate corpus contains 25 writing models but is not balanced; use v0_4_core for generator comparisons
• Citation Artifacts: Some documents may contain fabricated citations that should not be treated as real references

Bias Considerations

• Topic distribution reflects Library of Congress classification priorities
• Geographic coverage may be skewed toward certain regions
• Register distribution may not match real-world document frequencies

Citation

@article{bommarito2026shelf,
  title = {SHELF: A Synthetic Harness for Multi-Task Bibliographic Benchmarking},
  author = {Bommarito, Michael J.},
  year = {2026},
  journal = {arXiv preprint arXiv:2609.03047},
  eprint = {2609.03047},
  archivePrefix = {arXiv},
  primaryClass = {cs.CL},
  url = {https://arxiv.org/abs/2609.03047}
}

Additional Information

Dataset Curators

Michael J. Bommarito II

Licensing Information

This dataset is released under the Creative Commons Attribution 4.0 International (CC BY 4.0) license.

Version History

• v0.3.1 (2025-12-14): Multi-model generation with 42,532 documents (9 LLMs)
• v0.2.0 (2025-12-12): Initial pre-release with 40,100 documents (GPT-5.1/5.2)

Contact

For questions or issues, please open an issue on the GitHub repository.

Component configurations and provenance

The aggregate corpus is the default object of study. Component configurations remain available for provenance, generator-balanced analysis, and comparison with earlier results.

Why the balanced component remains separate

The default corpus is 94.1% GPT-5.x, with two models supplying that share and five of its nine generators contributing about 100 documents each. v0_4_core spreads 15 generators with no single one above 9.2%.

Pooling the two would return the largest generator to roughly half the combined corpus, which would undo the balance v0_4_core exists to provide. Any experiment that needs generator balance -- cross-generator generalization, generator attribution, train-on-family-A / test-on-family-B -- should use v0_4_core alone.

Known limitations

No subclass tier in this release. An LCC subclass tier was planned and is not shipped. The specification blocks assigned all 80 subclasses, but the generation path passed the parent class description to the model, so the documents were conditioned on 16 parent classes rather than 80 subclasses and carry no subclass label. Those documents are published as v0_4_supplement, described for what they are. The tier will return when the conditioning is fixed.

Empty-document rate. v0_4_supplement and v0_4_minimal_pairs were generated before a fix for a reasoning-budget defect: on short length targets, reasoning tokens consumed the whole output cap, producing a title truncated mid-word and no body. 13-15% of raw generations were affected. QC removes them, so the published slices are clean but roughly 14% smaller than their nominal spec count.

Generator confound on register and length. In v0_4_core, generator is independent of the labels that matter -- LCC class (Cramer's V 0.018) and LCGFT category (0.028) -- but correlates weakly with register (0.062), target_length (0.066) and prompt_variant_id (0.085). The cause is non-uniform QC removal: the empty-body defect hit short documents hardest and two generators failed for part of the run. Effect sizes are small; p-values are not marginal. Analyses conditioned on register or length carry this confound.

transfer_gutenberg is contaminated and must not be pooled. It is in the pretraining data of essentially every model that would be evaluated on it. SHELF is the clean-synthetic condition, Gutenberg is the contaminated-natural one, and the gap between them is the measurement. A lexical baseline trained on SHELF scores 0.893 macro-F1 in-domain and 0.301 on Gutenberg; trained on Gutenberg it reaches 0.526 in-domain. Transfer fails symmetrically, which is domain shift rather than memorisation.

No human ceiling yet. No human annotation round has been run, so model scores on these slices have no interpretable upper bound.

Prompt variants differ from default. v0.4 documents use four new system-prompt variants and form-conditional output formatting; default used a single prompt. prompt_variant_id records which. A controlled A/B measured spurious markdown on non-markdown forms falling from 26.7% to ~1.3% (Fisher exact p < 0.00001).

The all config

A single aggregate corpus of every synthetic SHELF document.

Splits: train 37,795 / validation 12,600 / test 12,504. Each document keeps the split it was assigned in its source config.

This config is not generator balanced, and that is the trade. Pooling returns the largest generator to 47.7% of the corpus, against 9.2% in v0_4_core. Use all when sample count matters more than balance, and v0_4_core when it does not. Reporting a generator-sensitive result on all without saying so would be misleading.

Every row carries source_config and source_version, so any component slice can be recovered exactly:

from datasets import load_dataset
ds = load_dataset("mjbommar/SHELF", "all")
core = ds["train"].filter(lambda r: r["source_config"] == "v0_4_core")

Schema is the union of both generations (44 columns), so no column is dropped; columns absent from a source are null. text is always populated. Provider routing prefixes are normalised, so one model is one id. Titles carrying a leading markdown heading or Title: label were cleaned (169 rows). Deduplicated on normalised body text: zero duplicates were found across the two corpora, as expected from disjoint spec blocks.

The Gutenberg transfer control is deliberately excluded. It is natural text used to measure whether SHELF scores transfer, and pooling it into the corpus would destroy that measurement. It remains a separate config.

Measured properties

Two properties matter for anyone deciding whether to use SHELF, and they have different answers. Both were measured with TF-IDF plus logistic regression on 21-class LCC (no pretraining, so contamination cannot explain either), and with 25 distinct embedding models for the ranking result.

Absolute scores do not transfer. A classifier scoring 0.880 on SHELF scores 0.319 on Project Gutenberg passages. The failure is symmetric.

This is not peculiar to generated text. The two human-catalogued corpora, Gutenberg and LCSHBench, reach only 0.2135 and 0.2800 on each other -- the worst pairing in the matrix. No bibliographic corpus stands in for the task in general.

Model rankings do transfer. Ranking 25 distinct embedding models on each corpus:

So: use SHELF to choose between models, not to predict a production score. Note that the natural corpora still agree with each other most closely (0.963); the intervals overlap, so SHELF ranks models about as well as natural bibliographic data, not better.

How much label signal sits on the surface. Verbatim lcc_name in its own document, length-controlled to 200 words: Gutenberg 6.1%, SHELF 19.1%. Real documents do contain their own descriptive terms -- a zero baseline would be strange -- but SHELF carries about 3x the natural rate, which partly explains its lexical ceiling. QC reduced this measurably between generations: topics fell from 76.6% to 44.5%, form from 7.2% to 1.5%.

transfer_lcshbench

English records from LCSHBench (CC0), real catalogue records from Harvard, Columbia, and Princeton carrying real LCC classes. 4,924 rows across 21 classes, used here as a second natural control.

Do not pool it with transfer_gutenberg. Gutenberg is running prose; LCSHBench is catalogue metadata with a median of 596 characters. Report them separately.