pdfQA: Diverse, Challenging, and Realistic Question Answering over PDFs

pdfQA is a structured benchmark collection for document-level question answering and PDF understanding research.

The dataset is organized to support:

• Raw document processing research
• Structured extraction pipelines
• Retrieval-augmented QA
• End-to-end document reasoning systems

It preserves original documents alongside structured derivatives to enable reproducible evaluation across preprocessing strategies.

---

Dataset Structure

⚠️ The QA annotations are released as a separate Hugging Face dataset:
https://huggingface.co/datasets/pdfqa/pdfQA-Annotations
(Dataset ID: pdfqa/pdfQA-Annotations)

This repository follows a strict hierarchical layout:

<category>/<type>/<dataset>/...

Categories

• real-pdfQA/ — Real-world benchmark datasets
• syn-pdfQA/ — Synthetic benchmark datasets

Types

Each dataset contains three file-type folders:

• 01.1_Input_Files_Non_PDF/ — Original source formats (e.g., xlsx, epub, htm, tex, txt)
• 01.2_Input_Files_PDF/ — Original PDF files
• 01.3_Input_Files_CSV/ — Structured tabular representations

Datasets

Each type folder contains subfolders for individual datasets. Supported datasets include:

Real-world Datasets

• ClimateFinanceBench/
• ClimRetrieve/
• FeTaQA/
• FinanceBench/
• FinQA/
• NaturalQuestions/
• PaperTab/
• PaperText/
• Tat-QA/

Synthetic Datasets

• books/
• financial_reports/
• sustainability_disclosures/
• research_articles/

Example

syn-pdfQA/
  01.2_Input_Files_PDF/
    books/
      file1.pdf
  01.3_Input_Files_CSV/
    books/
      file1.csv
  01.1_Input_Files_Non_PDF/
    books/
      file1.xlsx

This design allows:

• Access to original PDFs
• Access to structured evaluation data
• Access to original source formats for preprocessing research

---

Intended Use

This dataset is intended for:

• PDF parsing and layout understanding
• Financial and sustainability document QA
• Retrieval-augmented generation (RAG)
• Multi-modal document pipelines
• Table extraction and structured reasoning
• Robustness evaluation across preprocessing pipelines

It is particularly useful for comparing:

• Direct PDF-based reasoning
• OCR pipelines
• Structured table extraction
• Raw-source ingestion approaches

---

Access Patterns

The dataset supports multiple access patterns depending on research needs.

All official download scripts are available in the GitHub repository:

👉 https://github.com/tobischimanski/pdfQA

Scripts are provided in both:

• Bash (git + Git LFS) --- recommended for large-scale downloads\
• Python (huggingface_hub API) --- recommended for programmatic workflows

---

1️⃣ Download Everything

Download the entire repository (all categories, types, and datasets).

Bash (git + LFS)

./tools/download_using_bash/download_all.sh

Bash script

Python (HF API)

python tools/download_using_python/download_all.py

Python script

---

2️⃣ Download by Category

Download only:

• real-pdfQA/
• or syn-pdfQA/

Example

./tools/download_using_bash/download_category.sh syn-pdfQA

Bash script

Python script

---

3️⃣ Download by Dataset (All Types)

Download a single dataset across all three file-type folders:

• 01.1_Input_Files_Non_PDF/
• 01.2_Input_Files_PDF/
• 01.3_Input_Files_CSV/

Example

./tools/download_using_bash/download_dataset.sh syn-pdfQA books

Bash script

Python script

---

4️⃣ Download Arbitrary Folders

Download one or multiple arbitrary folder paths.

Example

./tools/download_using_bash/download_folders.sh \
  "syn-pdfQA/01.2_Input_Files_PDF/books" \
  "syn-pdfQA/01.3_Input_Files_CSV/books"

Bash script

Python script

---

5️⃣ Download Specific Files

Download one or more individual files.

Example (Bash)

./tools/download_using_bash/download_files.sh \
  "syn-pdfQA/01.2_Input_Files_PDF/books/file1.pdf"

Bash script

Python script

---

6️⃣ Direct API Access (Single File)

Files can also be downloaded directly using the Hugging Face API. Example:

from huggingface_hub import hf_hub_download

hf_hub_download(
    repo_id="pdfqa/pdfQA-Benchmark",
    repo_type="dataset",
    filename="syn-pdfQA/01.2_Input_Files_PDF/books/978-3-030-10752-9.pdf"
)

---

Recommended Usage

• For large-scale research experiments → use Bash + git LFS (fully resumable).
• For automated pipelines → use Python scripts.
• For fine-grained subset control → use folder or file-based scripts.

---

Data Modalities

Depending on the dataset:

• Financial reports
• Sustainability disclosures
• Structured financial QA corpora
• Table-heavy documents
• Mixed structured/unstructured content

Formats may include: PDF, CSV, XLS/XLSX, EPUB, HTML/HTM, TEX, TXT

---

Research Motivation

Many document QA benchmarks release only structured data or only PDFs. pdfQA preserves all representations:

• Original document
• Structured derivative
• Raw source format (if available)

This enables:

• Studying preprocessing impact
• Comparing parsing strategies
• Evaluating robustness to format variation
• End-to-end pipeline benchmarking

---

Citation

If you use pdfQA, please cite:

@misc{schimanski2026pdfqa,
      title={pdfQA: Diverse, Challenging, and Realistic Question Answering over PDFs}, 
      author={Tobias Schimanski and Imene Kolli and Yu Fan and Ario Saeid Vaghefi and Jingwei Ni and Elliott Ash and Markus Leippold},
      year={2026},
      eprint={2601.02285},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2601.02285}, 
}

---

Contact

Visit https://github.com/tobischimanski/pdfQA for access and updates.