BigDocs-Bench

Benchmark data for the paper: BigDocs: An Open and Permissively-Licensed Dataset for Training Multimodal Models on Document and Code Tasks

🌐 Homepage | 📖 arXiv

🔔 News

• [2024-12-10]: Initial release of the the BigDocs-Bench data.

Introduction

We introduce BigDocs-Bench a comprehensive benchmark suite designed to evaluate downstream tasks that transform visual inputs into structured outputs, such as GUI2UserIntent (fine-grained reasoning) and Image2Flow (structured output). We are actively working on releasing additional components of BigDocs-Bench and will update this repository as they become available.

Guide on Data Loading

For a part of BigDocs-Bench (GUI2UserIntent, GUI2Summary, GUI-VQA, and GUI2BBox), we do not explicitly distribute the images as part of this repository. You will find below instructions and a script facilitating the alignment of these images with our annotations. These instructions assumes a Linux-like environment.

1. Download the SeeClick source images from the official website using wget.

   wget -O seeclick_data.zip https://box.nju.edu.cn/f/6a804cf190dd490a808f/?dl=1
   

2. Unzip it to a path data_path defined by the user.

   unzip seeclick_data.zip -d [data_path]
   

3. Use the following python script to load these 4 datasets. Please assign the value of data_path as the one above where source images were downloaded.

   import os
   import datasets
   from PIL import Image as PILImage
   
   data_path = ""   # your data path where unzipped images source can be found
   
   def recover_image_column(dataset_dict, user_local_path, num_proc=os.cpu_count()-1):
       def load_image(entry):
           img_path = os.path.join(user_local_path, entry['img_id'])
           try:
               entry['image'] = PILImage.open(img_path)
           except Exception as e:
               raise RuntimeError(f"Error loading image at {img_path}: {e}")
           entry.pop('img_id', None)
           return entry
   
       # Apply the mapping function to all splits in the DatasetDict
       return dataset_dict.map(load_image, num_proc=num_proc)
   
   repo_name = "ServiceNow/BigDocs-Bench"
   
   # GUI2UserIntent
   formal_name = "GUI2UserIntent"
   GUI2UserIntent_raw = datasets.load_dataset(repo_name, formal_name)
   GUI2UserIntent = recover_image_column(GUI2UserIntent_raw, data_path) # complete dataset 
   
   # GUI2Summary
   formal_name = "GUI2Summary"
   GUI2Summary_raw = datasets.load_dataset(repo_name, formal_name)
   GUI2Summary = recover_image_column(GUI2Summary_raw, data_path) # complete dataset 
   
   # GUI-VQA
   formal_name = "GUI-VQA"
   GUI_VQA_raw = datasets.load_dataset(repo_name, formal_name)
   GUI_VQA = recover_image_column(GUI_VQA_raw, data_path) # complete dataset 
   
   # GUI2BBox
   formal_name = "GUI2BBox"
   GUI2BBox_raw = datasets.load_dataset(repo_name, formal_name)
   GUI2BBox = recover_image_column(GUI2BBox_raw, data_path) # complete dataset
   
   # Other datasets don't need any particular processing.
   Image2Flow_GraphViz = datasets.load_dataset(repo_name, "Image2Flow_(GraphViz)")
   Image2Flow_JSON = datasets.load_dataset(repo_name, "Image2Flow_(JSON)")
   Table2LaTex = datasets.load_dataset(repo_name, "Table2LaTex")
   

Special Notes

• We notice that some QA are not very closely related to the image in GUI-VQA dataset. We thus provide an extra filter_out column in this dataset; please consider eschewing the samples for which this entry is True.

Licensing

The part of this repository generated by us is Copyright ServiceNow 2024 and licensed under the CC-BY-4.0 license.

Multiple datasets, documents, and tools were involved in the generation of BigDocs-Bench. We document these dependencies on a per-sample basis through the query_info, annotation_info and image_info fields, respectively documenting the query, annotations and image fields of our datasets. We summarize below these dependencies.

• We use documents retrieved from arXiv, providing the license and attributing the authors of each paper on a per-sample basis. Some samples were adopted from the Arxiv Figures & Tables Database (AFTdb).
• We use HTML and metadata from SeeClick. The images from SeeClick are not distributed with BigDocs-Bench and must be downloaded separately from SeeClick. ServiceNow does not claim copyright over these images. We provide a script to facilitating the alignment of these images with our annotations.
• We use the InternVL-8B model to generate summaries.
• We use the Llama 3.1 405B model to generate queries, annotations and Graphviz code. By using this dataset, you agree to comply with the LLaMA 3.1 license.
• We use Graphviz's PNG visualizer to produce images from Graphviz code.