UNIDOC-BENCH Dataset

A unified benchmark for document-centric multimodal retrieval-augmented generation (MM-RAG).

Table of Contents

• Dataset Description
  • Key Features
  • Dataset Structure
• Domain Descriptions
• QA Dataset Format
  • Field Descriptions
  • Query Types
• Data Access
  • Download Options
  • Compressed File Sizes
• Usage
  • Loading the Dataset
  • Working with PDF Files from Hugging Face
  • Building RAG Database from PDFs
  • Working with PDF Documents
  • Working with QA Pairs
  • Visual Question Answering (VQA)
• Evaluation Paradigms
• Benchmark Results
• Data Quality
• Citation
• License
  • Important Usage Notice
  • License Terms
• Paper
• Contact
• Acknowledgments

Dataset Description

UNIDOC-BENCH is the first large-scale, realistic benchmark for multimodal retrieval-augmented generation (MM-RAG) and Visual Question Answering (VQA) built from 70,000 real-world PDF pages across eight domains. The dataset extracts and links evidence from text, tables, and figures, then generates 1,700+ multimodal QA pairs spanning factual retrieval, comparison, summarization, and logical reasoning queries. Each QA pair includes image paths (mage_paths) for the entire pdf for direct VQA evaluation, making it suitable for both MM-RAG and VQA research.

Key Features

• 70,000 real-world PDF pages across 8 diverse domains
• 1,700+ multimodal QA pairs with expert validation
• Four evaluation paradigms: text-only, image-only, multimodal text-image fusion, and multimodal joint retrieval
• Visual Question Answering (VQA) support: Image paths (longdoc_image_paths and gt_image_paths) for direct VQA evaluation
• 20% expert validation ensuring high-quality annotations
• Comprehensive query types: factual retrieval, comparison, summarization, and logical reasoning

Dataset Structure

The dataset is organized as a single split named test containing all 1,742 QA pairs. Each QA pair includes a domain field that indicates which of the 8 domains it belongs to. To access domain-specific data, filter the test split by the domain field.

Dataset splits:

• test: 1,742 QA pairs across all 8 domains

Available domains:

• commerce_manufacturing
• construction
• crm
• education
• energy
• finance
• healthcare
• legal

Domain Descriptions

QA Dataset Format

The dataset contains 1,742 question-answer pairs in a single test split. Each QA pair has the following structure:

{
  "question": "Which procedures are part of Hunt Design's corrective measures process as defined in QMSP-1011?",
  "answer": "The procedures part of Hunt Design's corrective measures process as defined in QMSP-1011 include the analysis of data, internal audits, corrective actions, control of non-conforming products, and management review.",
  "gt_image_paths": [
    "/path/to/images/commerce_manufacturing/0355944/0355944_page_0004.png"
  ],
  "question_type": "factual_retrieval",
  "answer_type": "image_only",
  "domain": "commerce_manufacturing",
  "longdoc_image_paths": [
    "/path/to/images/commerce_manufacturing/0355944/0355944_page_0001.png",
    "/path/to/images/commerce_manufacturing/0355944/0355944_page_0002.png",
    "/path/to/images/commerce_manufacturing/0355944/0355944_page_0003.png",
    "/path/to/images/commerce_manufacturing/0355944/0355944_page_0004.png",
    "/path/to/images/commerce_manufacturing/0355944/0355944_page_0005.png"
  ]
}

Field Descriptions

• question: The question text (obscured version for evaluation)
• answer: The complete answer to the question
• gt_image_paths: Ground-truth image paths - List of image file paths that contain the answer. These are the exact pages/images needed to answer the question correctly. Use this for Retrieval evaluation.
• longdoc_image_paths: All image paths in this doc - List of all image file paths from the document(s). These include all pages from the source document(s). Use this for VQA evaluation.
• question_type: Type of question (factual_retrieval, comparison, summarization, logical_reasoning)
• answer_type: Answer modality (text_only, image_only, table_required, multimodal)
• domain: Domain category (commerce_manufacturing, construction, crm, education, energy, finance, healthcare, legal)

Query Types

• Factual Retrieval: Direct questions about specific information
• Comparison: Questions requiring comparison between multiple sources
• Summarization: Questions asking for summaries or overviews
• Logical Reasoning: Questions requiring inference and reasoning

Data Access

📊 Compressed File Sizes

🚀 Usage

📚 Loading the Dataset

from datasets import load_dataset

# Load the dataset from Hugging Face
# The dataset has 8 domain-based splits, each containing 200 QA pairs
dataset = load_dataset("Salesforce/UniDoc-Bench")

# Available splits: commerce_manufacturing, construction, crm, education, energy, finance, healthcare, legal
print(dataset.keys())

# Access QA pairs by domain
healthcare_qa = dataset["healthcare"]
construction_qa = dataset["construction"]
crm_qa = dataset["crm"]
education_qa = dataset["education"]
energy_qa = dataset["energy"]
finance_qa = dataset["finance"]
commerce_manufacturing_qa = dataset["commerce_manufacturing"]
legal_qa = dataset["legal"]

# Or load a specific domain directly
healthcare_only = load_dataset("Salesforce/UniDoc-Bench", split="healthcare")

📦 Working with PDF Files from Hugging Face

The dataset includes compressed PDF files for each domain that can be used for RAG database construction:

• commerce_manufacturing_pdfs.tar.gz (529 MB)
• construction_pdfs.tar.gz (885 MB)
• crm_pdfs.tar.gz (704 MB)
• education_pdfs.tar.gz (816 MB)
• energy_pdfs.tar.gz (708 MB)
• finance_pdfs.tar.gz (360 MB)
• healthcare_pdfs.tar.gz (1.19 GB)
• legal_pdfs.tar.gz (541 MB)

from datasets import load_dataset
import tarfile
import os
from huggingface_hub import hf_hub_download

# Method 1: Download specific PDF archives
def download_pdf_archive(domain_name, save_path="."):
    """Download PDF archive for a specific domain"""
    filename = f"{domain_name}_pdfs.tar.gz"
    file_path = hf_hub_download(
        repo_id="Salesforce/UniDoc-Bench",
        filename=filename,
        repo_type="dataset",
        local_dir=save_path
    )
    return file_path

# Method 2: Extract PDFs for RAG database construction
def extract_pdfs_for_rag(domain_name, extract_to="pdf_database"):
    """Download and extract PDFs for RAG database"""
    # Download the archive
    archive_path = download_pdf_archive(domain_name)
    
    # Create extraction directory
    domain_dir = os.path.join(extract_to, domain_name)
    os.makedirs(domain_dir, exist_ok=True)
    
    # Extract PDFs
    with tarfile.open(archive_path, 'r:gz') as tar:
        tar.extractall(domain_dir)
    
    print(f"Extracted {domain_name} PDFs to {domain_dir}")
    return domain_dir

# Example: Extract healthcare PDFs for RAG
healthcare_pdfs = extract_pdfs_for_rag("healthcare")

🗄️ Building RAG Database from PDFs

import fitz  # PyMuPDF
from pathlib import Path
import json
from huggingface_hub import hf_hub_download
import tarfile

def build_rag_database(domain_name, output_dir="rag_database"):
    """Build RAG database from domain PDFs"""
    
    # Download and extract PDFs
    archive_path = hf_hub_download(
        repo_id="Salesforce/UniDoc-Bench",
        filename=f"{domain_name}_pdfs.tar.gz",
        repo_type="dataset"
    )
    
    # Extract to temporary directory
    temp_dir = f"temp_{domain_name}_pdfs"
    with tarfile.open(archive_path, 'r:gz') as tar:
        tar.extractall(temp_dir)
    
    # Process PDFs for RAG
    rag_data = []
    pdf_dir = Path(temp_dir)
    
    for pdf_file in pdf_dir.glob("**/*.pdf"):
        doc = fitz.open(pdf_file)
        
        for page_num in range(len(doc)):
            page = doc[page_num]
            text = page.get_text()
            images = page.get_images()
            
            # Create RAG entry
            entry = {
                "source": str(pdf_file),
                "page": page_num,
                "text": text,
                "images": len(images),
                "domain": domain_name
            }
            rag_data.append(entry)
        
        doc.close()
    
    # Save RAG database
    os.makedirs(output_dir, exist_ok=True)
    with open(f"{output_dir}/{domain_name}_rag.json", "w") as f:
        json.dump(rag_data, f, indent=2)
    
    print(f"Built RAG database for {domain_name}: {len(rag_data)} entries")
    return rag_data

# Example: Build RAG database for healthcare
healthcare_rag = build_rag_database("healthcare")

📄 Working with PDF Documents

import fitz  # PyMuPDF
from pathlib import Path

# Load a specific PDF
pdf_path = "data/final_database/healthcare/medical_doc.pdf"
doc = fitz.open(pdf_path)

# Extract text and images
for page_num in range(len(doc)):
    page = doc[page_num]
    text = page.get_text()
    images = page.get_images()
    
    # Process multimodal content
    for img_index, img in enumerate(images):
        xref = img[0]
        pix = fitz.Pixmap(doc, xref)
        # Process image...

❓ Working with QA Pairs

from datasets import load_dataset

# Load the dataset from Hugging Face
dataset = load_dataset("Salesforce/UniDoc-Bench")

# Access the dataset (contains all 1,742 QA pairs)
data = dataset["test"]

# Filter QA data for a specific domain
healthcare_qa = data.filter(lambda x: x["domain"] == "healthcare")
construction_qa = data.filter(lambda x: x["domain"] == "construction")

# Filter by query type within a domain
healthcare_factual = healthcare_qa.filter(lambda x: x["question_type"] == "factual_retrieval")

# Filter by answer type within a domain
healthcare_image_only = healthcare_qa.filter(lambda x: x["answer_type"] == "image_only")

# Filter by multiple criteria
healthcare_factual_image = data.filter(
    lambda x: x["domain"] == "healthcare" 
    and x["question_type"] == "factual_retrieval" 
    and x["answer_type"] == "image_only"
)

🖼️ Visual Question Answering (VQA)

UNIDOC-BENCH supports Visual Question Answering (VQA) evaluation, where models answer questions based on document images. The dataset provides image paths (image_paths) that contain the answer, enabling direct VQA evaluation.

VQA Dataset Structure

Each QA item includes:

• longdoc_image_paths: All relevant document pages (for retrieval-based evaluation)
• question: The question to answer
• answer: The ground-truth answer

VQA Usage Examples

from datasets import load_dataset
from PIL import Image

# Load the dataset
dataset = load_dataset("Salesforce/UniDoc-Bench", split="test")

# Get an example with all relevant images
example = dataset[0]
question = example["question"]
all_images = example["longdoc_image_paths"]  # All relevant document pages

# Simulate retrieval: select top-k images
top_k = 5
retrieved_images = all_images[:top_k]

# Load retrieved images
retrieved_image_list = []
for img_path in retrieved_images:
    image = Image.open(img_path)  # Adjust path based on your setup
    retrieved_image_list.append(image)

# Use with your VQA model
# model_answer = vqa_model(question, retrieved_image_list)

Benchmark Results

According to the UNIDOC-BENCH paper:

• Multimodal text-image fusion RAG systems consistently outperform both unimodal and jointly multimodal embedding-based retrieval
• Neither text nor images alone are sufficient for optimal document understanding
• Current multimodal embeddings remain inadequate for complex document-centric tasks

SOTA Results

We evaluate three state-of-the-art vision-language models: GPT-5, Claude-4.5-Sonnet, and Gemini-2.5-Pro across all domains. Results are reported for three retrieval settings:

• topk=10/20: Retrieval-based evaluation with top-k retrieved pages using the best retrieval method in the paper (T+I)
• GT: Ground-truth evaluation using the exact pages containing the answer

Completeness by Domain

Key Findings:

• Claude-4.5-Sonnet achieves the highest recall (completeness) across all settings, with an average of 79.93% (GT), 76.56% (topk=10), and 75.94% (topk=20)
• GPT-5 shows strong performance with an average of 74.30% (GT), 68.06% (topk=10), and 69.46% (topk=20)
• Gemini-2.5-Pro shows competitive performance with an average of 69.50% (GT), 67.67% (topk=10), and 68.99% (topk=20)

Completeness by Answer Type (GT Setting)

• Text Only: Claude-4.5-Sonnet (0.8940) > GPT-5 (0.8380) > Gemini-2.5-Pro (0.7605)
• Image Only: GPT-5 (0.6815) > Claude-4.5-Sonnet (0.6769) > Gemini-2.5-Pro (0.6332)
• Table Required: Claude-4.5-Sonnet (0.8057) > GPT-5 (0.7530) > Gemini-2.5-Pro (0.7123)
• Image Plus Text: Claude-4.5-Sonnet (0.8096) > GPT-5 (0.7002) > Gemini-2.5-Pro (0.6857)

Citation

If you use this dataset in your research, please cite:

@article{peng2025unidoc,
  title={UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAG},
  author={Peng, Xiangyu and Qin, Can and Chen, Zeyuan and Xu, Ran and Xiong, Caiming and Wu, Chien-Sheng},
  journal={arXiv preprint arXiv:2510.03663},
  year={2025}
}

License

This dataset is released under the Creative Commons Attribution-NonCommercial 4.0 International License (CC-BY-NC 4.0).

⚠️ Important Usage Notice

🚫 This dataset was generated using GPT-4.1 and should not be used to develop models that compete with OpenAI.

License Terms

• Attribution: You must give appropriate credit, provide a link to the license, and indicate if changes were made.
• Non-Commercial: You may not use the material for commercial purposes.
• No additional restrictions: You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.

For full license details, see: https://creativecommons.org/licenses/by-nc/4.0/

Paper

UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAG

Contact

For questions about this dataset, please open an issue on the GitHub repository or contact the authors.

Acknowledgments

We thank the annotators and domain experts who contributed to the validation of this dataset. This work was supported by Salesforce Research.