FigureBench

The first large-scale benchmark for generating scientific illustrations from long-form scientific texts.

Paper | Code

Overview

FigureBench is curated to encompass a wide array of document types, including research papers, surveys, technical blogs, and textbooks, establishing a challenging and diverse testbed to spur research in automatic scientific illustration generation.

This dataset contains:

• Development Set (dev): 3,000 samples with conversation-format annotations for training and development
• Test Set (test): 300 samples with simple text-image pairs strictly for evaluation

HuggingFace

The dataset is available on HuggingFace: WestlakeNLP/FigureBench

from datasets import load_dataset

# Load the dataset
dataset = load_dataset("WestlakeNLP/FigureBench")

# Access development set (for training/development)
dev_sample = dataset["dev"][0]

# Access test set (for evaluation only)
test_sample = dataset["test"][0]

Dataset Statistics

Key Challenges

• Long-context reasoning: Text tokens vary by over an order of magnitude (352 for Textbooks to 12,732 for Papers)
• High text density: Average 41.2% of image area occupied by text
• Structural complexity: Average 5.3 components and 6.4 shapes per illustration
• Visual richness: Average 6.2 colors per illustration

Data Format

Development Set (dev)

Each sample in the development set uses a conversation format:

{
    "messages": [
        {
            "role": "system",
            "content": "You are an expert in analyzing scientific papers..."
        },
        {
            "role": "user",
            "content": "Please determine whether the image is the most important illustration...

[Paper text content]"
        },
        {
            "role": "assistant",
            "content": "yes"
        }
    ],
    "images": ["images/{paper_id}/{image_filename}"]
}

Test Set (test)

The test set uses a simple text-image pair format for evaluation:

{
    "text": "Original document text content...",
    "image": "test_images/{category}/{filename}.png",
    "category": "paper|survey|blog|textbook"
}

Usage

Loading Development Set

from datasets import load_dataset

dataset = load_dataset("WestlakeNLP/FigureBench")
dev = dataset["dev"]

sample = dev[0]
messages = sample["messages"]
image_path = sample["images"][0]

Loading Test Set for Evaluation

from datasets import load_dataset

dataset = load_dataset("WestlakeNLP/FigureBench")
test = dataset["test"]

sample = test[0]
text = sample["text"]
image_path = sample["image"]
category = sample["category"]

Dataset Structure

FigureBench/
├── README.md
├── data/
│   ├── dev.parquet      # Development set (3,000 samples)
│   └── test.parquet     # Test set (300 samples)
├── images/              # Development set images
│   ├── {paper_id}/
│   │   └── {paper_id}_{figure_name}.png
│   └── ...
└── test_images/         # Test set images
    ├── blog/
    ├── paper/
    ├── survey/
    └── textbook/

Data Curation

Source Data

The dataset is curated from multiple high-quality sources:

• Papers: Sampled from the Research-14K dataset, filtered using GPT to select illustrations that best represent core methodologies
• Surveys: Structural diagrams (roadmaps, taxonomies) from recent AI surveys on arXiv
• Textbooks: Sourced from open-licensed educational platforms (e.g., OpenStax) for pedagogical clarity
• Blogs: Hand-collected from technical outlets (e.g., ICLR Blog Track) for modern visual styles

Annotation Process

1. Initial filtering to retain only conceptual illustrations (excluding data-driven charts)
2. Ensured each key visual element is explicitly described in the source text
3. Two independent annotators evaluated each pair
4. Only pairs approved by both annotators were included
5. Achieved high Inter-Rater Reliability (IRR) of 0.91

Citation

If you use FigureBench or AutoFigure in your research, please cite:

@inproceedings{
zhu2026autofigure,
title={AutoFigure: Generating and Refining Publication-Ready Scientific Illustrations},
author={Minjun Zhu and Zhen Lin and Yixuan Weng and Panzhong Lu and Qiujie Xie and Yifan Wei and Sifan Liu and Qiyao Sun and Yue Zhang},
booktitle={The Fourteenth International Conference on Learning Representations},
year={2026},
url={https://openreview.net/forum?id=5N3z9JQJKq}
}

License

This dataset is released under CC BY 4.0.

The curation process adhered to open-source licenses. Please refer to the original papers for the copyright of individual figures.