DocGenome: An Open Large-scale Scientific Document Benchmark for Training and Testing Multi-modal Large Language Models

We present DocGenome, a structured document dataset constructed by annotating 500K scientific documents from 153 disciplines in the arXiv open-access community, using our custom auto-labeling pipeline DocParser. DocGenome features four characteristics:

• 1. Completeness: It is the first dataset to structure data from all modalities including 13 layout attributes along with their \LaTeX\ source codes.
• 2. Logicality: It provides 6 logical relationships between different entities within each scientific document.
• 3. Diversity: It covers various document-oriented tasks, including document classification, visual grounding, document layout detection, document transformation, open-ended single-page QA and multi-page QA.
• 4. Correctness: It undergoes rigorous quality control checks conducted by a specialized team.

Release

• [2025/1/13] Release the Document parsing tools DocParser
• [2024/9/5] 🔥 Add the data quality rating for each structured document in DocGenome here
• [2024/8/27] Add the tutorials on how to use the DocGenome dataset.
• [2024/8/7] Add the detalied explanation about the different file structures in DocGenome.Dataset_Details_README
• [2024/7/23] We have supported TestSet downloads from Huggingface. If you want to evaluate your model on TestSet, please refer to Evaluation.
• [2024/7/12] We have supported dataset downloads from Huggingface.
• [2024/6/15] 🔥 Our paper entitled "DocGenome: An Open Large-scale Scientific Document Benchmark for Training and Testing Multi-modal Large Models" has been released in arXiv Link
• [2024/6/6] 🔥 We have released the DocGenome benchmark, includes 8 subsets as follows:
  • docgenome-train-000.tar.gz
  • docgenome-train-001.tar.gz
  • docgenome-train-002.tar.gz
  • docgenome-train-003.tar.gz
  • docgenome-train-004.tar.gz
  • docgenome-train-005.tar.gz
  • docgenome-train-006.tar.gz
  • docgenome-train-007.tar.gz

File Structure

Please refer to Dataset_Details_README for the detalied explanation about the different file structures in DocGenome.

DocGenome Benchmark Introduction

 

👇🏻DocGenome-train Download

We provide 8 subsets of DocGenome-train for downloading:

Data Download

• docgenome-train-000.tar.gz
• docgenome-train-001.tar.gz
• docgenome-train-002.tar.gz
• docgenome-train-003.tar.gz
• docgenome-train-004.tar.gz
• docgenome-train-005.tar.gz
• docgenome-train-006.tar.gz
• docgenome-train-007.tar.gz

Definition of relationships between component units

DocGenome contains 4 level relation types and 2 cite relation types, as shown in the following table:

Attribute of component units

DocGenome has 13 attributes of component units, which can be categorized into two classes

• 1) Fixed-form units, including Text, Title, Abstract, etc., which are characterized by sequential reading and hierarchical relationships readily discernible from the list obtained in Stage-two of the designed DocParser.
• 2) Floating-form units, including Table, Figure, etc., which establish directional references to fixed-form units through commands like \texttt{\textbackslash ref} and \texttt{\textbackslash label}.

Types of disciplines

Page distribution of DocGenome. 20% of documents are five pages or fewer, 50% are ten pages or fewer, and 80% are nineteen pages or fewer.

Page Distribution

 

Distribution of secondary disciplines in our DocGenome. The count on the x-axis represents the number of documents, and documents from the same primary discipline are marked with the same color.

Discipline Distribution

 

DocParser: A Cutting-edge Auto-labeling Pipeline

Visualizations

Visual Example One of annotations in DocGenome

Visual Example One of annotations in DocGenome

Visual examples of document-oriented tasks in DocGenome

Citation

If you find our work useful in your research, please consider citing Fox:

@article{xia2024docgenome,
  title={DocGenome: An Open Large-scale Scientific Document Benchmark for Training and Testing Multi-modal Large Language Models},
  author={Xia, Renqiu and Mao, Song and Yan, Xiangchao and Zhou, Hongbin and Zhang, Bo and Peng, Haoyang and Pi, Jiahao and Fu, Daocheng and Wu, Wenjie and Ye, Hancheng and others},
  journal={arXiv preprint arXiv:2406.11633},
  year={2024}
}