Can Vision-Language Models Assess Graphic Design Aesthetics? A Benchmark, Evaluation, and Dataset Perspective

This repository contains the evaluation code for DesignBench, a benchmark for studying whether vision-language models can assess graphic design aesthetics.

Overview

DesignBench evaluates graphic design aesthetics across four dimensions and twelve task categories:

The code supports three evaluation modes:

Repository Layout

design_bench/
|-- benchmark_data/          # Benchmark data release
|-- main.py                  # Entry point
|-- evaluator.py             # Evaluation pipeline
|-- models/                  # Model adapters
|-- tasks/                   # Dimension/task definitions
|-- requirements.txt         # Python dependencies
|-- README.md
`-- results/                 # Saved evaluation outputs

Dataset Download

1. Download the DesignBench benchmark release from this link.
2. Extract it under the repository root as benchmark_data/.
3. If you store the data elsewhere, pass its location through --benchmark_path.

Expected layout:

benchmark_data/
`-- {sample_id}-perturbs_new/
    |-- preview.png
    |-- preview_highlight.png
    |-- simplified_meta_info.json
    |-- GT.json
    |-- changes.json
    |-- 0.png, 1.png, ...
    `-- meta_info.json

Notes:

• The evaluator scans sample folders whose names contain _new.
• GT.json stores labels using keys such as layout-balance and color-harmony.
• You can point the code to any compatible data directory with --benchmark_path /path/to/benchmark_data.

Installation

pip install -r requirements.txt

Supported Models

OpenAI-compatible API models

These models are called through the OpenAI Chat Completions API interface and support concurrent requests.

Local Qwen2.5-VL models

These models are loaded locally and use device_map="auto" for single-GPU or multi-GPU inference.

By default, the local Qwen adapter sets HF_ENDPOINT=https://hf-mirror.com. Override it in your environment if you need a different endpoint.

Running the Evaluation

API-based models

Set your credentials through environment variables:

export OPENAI_API_KEY="your_api_key"
export OPENAI_BASE_URL="https://api.openai.com/v1"

Run evaluation:

python3 main.py \
    --model_type gpt-4o-mini \
    --eval_mode text-preview \
    --benchmark_path ./benchmark_data \
    --output_dir ./results \
    --max_workers 4

Other modes use the same command with a different --eval_mode:

python3 main.py \
    --model_type gpt-4o-mini \
    --eval_mode text-preview-choice \
    --benchmark_path ./benchmark_data \
    --output_dir ./results

python3 main.py \
    --model_type gpt-4o-mini \
    --eval_mode text-preview-bbox \
    --benchmark_path ./benchmark_data \
    --output_dir ./results

You can also pass API settings by CLI:

python3 main.py \
    --model_type gpt-4o \
    --eval_mode text-preview \
    --benchmark_path ./benchmark_data \
    --api_key your_api_key \
    --api_base https://api.openai.com/v1

Local Qwen2.5-VL models

Single GPU:

python3 main.py \
    --model_type qwen2.5-vl-7b \
    --eval_mode text-preview-choice \
    --benchmark_path ./benchmark_data \
    --output_dir ./results \
    --device_ids 0

Multi-GPU automatic sharding:

python3 main.py \
    --model_type qwen2.5-vl-72b \
    --eval_mode text-preview \
    --benchmark_path ./benchmark_data \
    --output_dir ./results \
    --device_ids 0,1,2,3

Use a custom local checkpoint path:

python3 main.py \
    --model_type qwen2.5-vl-72b \
    --model_path /path/to/Qwen2.5-VL-72B-Instruct \
    --eval_mode text-preview-bbox \
    --benchmark_path ./benchmark_data \
    --output_dir ./results \
    --device_ids 0,1,2,3

Command-Line Arguments

Output

Each run writes one JSON file to:

results/{model_type}_{eval_mode}.json

The output includes:

• Per-task accuracy.
• Per-dimension averages.
• Overall accuracy and sample-level accuracy.
• Prediction details for every evaluated sample.
• API failure counts when applicable.
• Mean IoU for non-None ground-truth boxes in text-preview-bbox mode.

Extending the Codebase

Add a new OpenAI-compatible model

Add the mapping in models/gpt_model.py:

MODEL_NAME_MAP = {
    ...,
    "my-model": "my-model-api-name",
}

Then register it in models/__init__.py:

MODEL_REGISTRY["my-model"] = GPTModel

Add a new Qwen variant

Add the checkpoint mapping in models/qwen_model.py:

MODEL_REPO_MAP = {
    ...,
    "qwen2.5-vl-14b": "Qwen/Qwen2.5-VL-14B-Instruct",
}