DocGenome: An Open Large-scale Scientific Document Benchmark for Training and Testing Multi-modal Large Language Models

paper link: DocGenome

We present DocGenome, a structured document dataset constructed by annotating 500K scientific documents from 153 disciplines in the arXiv open-access community, using our custom auto-labeling pipeline DocParser. DocGenome features four characteristics:

• 1. Completeness: It is the first dataset to structure data from all modalities including 13 layout attributes along with their \LaTeX\ source codes.
• 2. Logicality: It provides 6 logical relationships between different entities within each scientific document.
• 3. Diversity: It covers various document-oriented tasks, including document classification, visual grounding, document layout detection, document transformation, open-ended single-page QA and multi-page QA.
• 4. Correctness: It undergoes rigorous quality control checks conducted by a specialized team.

Highlight:

• Data quality rating: for each structured document in DocGenome shown here
• Tutorials on how to use the DocGenome dataset.
• TestSet downloads from Huggingface. If you want to evaluate your model on TestSet, please refer to Evaluation.

DocGenome Benchmark Introduction

 

Definition of relationships between component units

DocGenome contains 4 level relation types and 2 cite relation types, as shown in the following table:

Attribute of component units

DocGenome has 13 attributes of component units, which can be categorized into two classes

• 1) Fixed-form units, including Text, Title, Abstract, etc., which are characterized by sequential reading and hierarchical relationships readily discernible from the list obtained in Stage-two of the designed DocParser.
• 2) Floating-form units, including Table, Figure, etc., which establish directional references to fixed-form units through commands like \texttt{\textbackslash ref} and \texttt{\textbackslash label}.

Citation

If you find our work useful in your research, please consider citing Fox:

@article{xia2024docgenome,
  title={DocGenome: An Open Large-scale Scientific Document Benchmark for Training and Testing Multi-modal Large Language Models},
  author={Xia, Renqiu and Mao, Song and Yan, Xiangchao and Zhou, Hongbin and Zhang, Bo and Peng, Haoyang and Pi, Jiahao and Fu, Daocheng and Wu, Wenjie and Ye, Hancheng and others},
  journal={arXiv preprint arXiv:2406.11633},
  year={2024}
}