PaperSearchQA Dataset

A large-scale biomedical question-answering dataset for training and evaluating search agents that reason over scientific literature.

Dataset Description

PaperSearchQA contains 60,000 question-answer pairs generated from PubMed abstracts, designed for training retrieval-augmented language models on biomedical question answering tasks.

Dataset Statistics

• Training set: 54,907 Q&A pairs
• Test set: 5,000 Q&A pairs
• Retrieval corpus: 16 million PubMed abstracts
• Question categories: 10 biomedical domains

Key Features

• Golden Answers: Each question includes multiple acceptable answer variations (synonyms, abbreviations, alternate names) to facilitate exact match (EM) evaluation. This allows for robust automatic evaluation without requiring semantic similarity models.

• Question Paraphrasing: 50% of questions are paraphrased versions to increase diversity and reduce reliance on surface-form matching during retrieval.

• Source Attribution: Each Q&A pair is linked to its source PubMed abstract via PMID for verification and further exploration.

• Category Labels: Questions span 10 biomedical categories including genetics, therapeutics, protein function, disease mechanisms, and more.

Dataset Construction

The dataset was generated using a scalable pipeline:

1. Source Corpus: 16 million PubMed abstracts from allMeSH (BioASQ 2022)
2. Question Generation: Used GPT-4.1 to generate 3 questions per sampled abstract, with category-aware prompts
3. Filtering: Removed questions containing "this study" or "the study" to ensure questions are answerable without the source abstract
4. Golden Answers: Generated comprehensive synonym lists using GPT-4.1 to capture answer variations (e.g., "dystrophin", "DMD protein")
5. Paraphrasing: Applied GPT-4.1 to paraphrase 50% of questions while preserving meaning
6. Validation: All questions are answerable by retrieving and reasoning over the PubMed corpus

LLM Models Used: GPT-4.1 (via OpenRouter) for all generation steps

Dataset Structure

Data Fields

Question Categories

1. Genetic mutations - Gene variants, inheritance patterns
2. Therapeutics & clinical evidence - Drugs, treatments, clinical trials
3. Protein function & signalling - Molecular mechanisms, pathways
4. Methods & resources - Experimental techniques, databases
5. Disease causation & pathogens - Etiology, infectious agents
6. Biomarkers & diagnostics - Disease markers, diagnostic tests
7. Bioinformatics databases - Computational resources
8. Clinical scales & classifications - Medical scoring systems
9. Anatomy & cellular localisation - Structures, subcellular locations
10. Psychology & behavioral health - Mental health, behavior

Usage

Loading the Dataset

from datasets import load_dataset

# Load full dataset
dataset = load_dataset("PaperSearchQA/PaperSearchQA")

# Access splits
train_data = dataset["train"]
test_data = dataset["test"]

# Example
example = train_data[0]
print(f"Question: {example['question']}")
print(f"Answer: {example['answer']}")
print(f"Golden Answers: {example['golden_answers']}")

Evaluation with Exact Match

The golden_answers field enables exact match evaluation with multiple acceptable answers:

def compute_exact_match(prediction: str, golden_answers: List[str]) -> float:
    """Returns 1.0 if prediction matches any golden answer (case-insensitive)"""
    prediction_normalized = prediction.strip().lower()
    for golden in golden_answers:
        if prediction_normalized == golden.strip().lower():
            return 1.0
    return 0.0

# Example
prediction = "dystrophin"
golden_answers = ["dystrophin", "DMD protein", "dystrophin protein"]
score = compute_exact_match(prediction, golden_answers)  # Returns 1.0

Example Data

{
  "question": "What anatomical structure, when atretic congenitally, can lead to unilateral hydrocephalus?",
  "answer": "Foramen of Monro",
  "golden_answers": [
    "Foramen of Monro",
    "interventricular foramen",
    "Monro's foramen",
    "interventricular foramina",
    "foramina of Monro"
  ],
  "question_original": "What anatomical structure, when atretic congenitally, can lead to unilateral hydrocephalus?",
  "is_paraphrased": false,
  "cat_num": "9",
  "cat": "Anatomy & cellular localisation",
  "pmid": "2666938",
  "paper_title": "Perinatal unilateral hydrocephalus. Atresia of the foramen of Monro."
}

Retrieval Corpus

The PubMed retrieval corpus is available separately:

• HuggingFace: jmhb/pubmed_bioasq_2022
• Size: 16 million abstracts (23GB JSONL, 13GB Parquet)
• Fields: title, abstract, PMID, MeSH terms, publication year

Additional Evaluation Data

For additional evaluation, use the BioASQ factoid test set:

• HuggingFace: jmhb/BioASQ
• Size: 1,609 factoid questions from BioASQ challenge
• Note: Requires registration at https://bioasq.org/

Citation

If you use PaperSearchQA in your research, please cite:

@misc{burgess2026papersearchqalearningsearchreason,
      title={PaperSearchQA: Learning to Search and Reason over Scientific Papers with RLVR},
      author={James Burgess and Jan N. Hansen and Duo Peng and Yuhui Zhang and Alejandro Lozano and Min Woo Sun and Emma Lundberg and Serena Yeung-Levy},
      year={2026},
      eprint={2601.18207},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2601.18207},
}

License

MIT License - see the GitHub repository for details.

Project Links

• Paper: https://arxiv.org/abs/2601.18207
• Website: https://jmhb0.github.io/PaperSearchQA/
• Code: https://github.com/jmhb0/PaperSearchQA
• PubMed Corpus: https://huggingface.co/datasets/jmhb/pubmed_bioasq_2022
• BioASQ Test Set: https://huggingface.co/datasets/jmhb/BioASQ

Acknowledgments

The PubMed corpus is derived from allMeSH (BioASQ 2022). BioASQ evaluation data requires registration at http://bioasq.org/.