ArtifactsBench: Bridging the Visual-Interactive Gap in LLM Code Generation Evaluation

Tencent Hunyuan Team

---

📖 Paper • 🏠 Home Page • 💻 Code • 🏆 Leaderboard • 📜 Citation

Figure 1: Automation level versus human–alignment across evaluation frameworks. The red star marks the fully manual WebDev Arena (100% human effort), while the blue bubble denotes our checklist-guided MLLM evaluation, ArtifactsBench, which achieves 94.4% agreement with human votes with 100% automation.

Introduction

The generative capabilities of Large Language Models (LLMs) are rapidly expanding from static code to dynamic, interactive visual artifacts. This progress is bottlenecked by a critical evaluation gap: established benchmarks focus on algorithmic correctness and are blind to the visual fidelity and interactive integrity that define modern user experiences.

To bridge this gap, we introduce ArtifactsBench, a new benchmark and paradigm for the automated, multimodal evaluation of visual code generation. Our framework programmatically renders each generated artifact and captures its dynamic behavior, which is then assessed by an MLLM-as-Judge guided by a fine-grained, per-task checklist to ensure holistic and reproducible scoring.

ArtifactsBench is open-sourced, including the benchmark with 1,825 diverse tasks, the evaluation harness, and baseline results, to provide the community with a scalable and accurate tool to accelerate the development of user-centric generative models.

Core Features

• A Diverse and Hierarchical Benchmark Suite: ArtifactsBench comprises a rich set of tasks derived from real-world applications, including component-based web development, SVG-based data visualization, and interactive mini-games. Tasks are stratified by complexity (simple, medium, hard) to robustly measure model capabilities across a meaningful difficulty gradient.
• A Novel Multimodal and Automated Evaluation Pipeline: We propose a new evaluation strategy that synergizes automated interaction with Multimodal Large Language Model (MLLM)-based assessment. Our framework programmatically interacts with the generated artifacts (e.g., clicking buttons, dispatching events) and captures visual states (e.g., screenshots, GIFs). An MLLM-as-Judge then evaluates these visual and textual traces against fine-grained, per-task criteria.
• In-depth Analysis and Gold-Standard Validation: We conducted a large-scale evaluation of over 30 prominent LLMs. Our automated evaluation achieves a striking 94.4% ranking consistency with WebDev Arena, the gold-standard for human preference, validating our approach as a highly reliable proxy for human-perceived quality.

Benchmark Comparison

ArtifactsBench is the first to offer high-granularity evaluation (GR), strong human-judgment consistency (CHA), automated assessment (AF), and direct visual evaluation (VE), addressing critical gaps in prior work.

Dataset Distribution

Figure 2: An overview of the ArtifactsBench dataset, illustrating the distribution of tasks across nine primary categories.

Evaluation Pipeline

Our evaluation process is designed to ensure objectivity and high consistency with human expert judgment.

Figure 3: The ArtifactsBench evaluation pipeline. The process hinges on a two-stage evaluation: (Step 5) we first validate our MLLM-as-Judge by confirming its high pairwise scoring agreement with human experts. (Step 6) Once its reliability is established, the automated judge is deployed at scale to evaluate all model outputs across the entire benchmark.

Quick Start

pip install vllm==0.8.3
pip install pytest-playwright
playwright install
playwright install-deps
pip install transformers
pip install requests
pip install tqdm

{
    "index": "unique identifier in the dataset that corresponds one-to-one with 'question'",
    "question": "each 'question' in ArtifactsBench",
    "answer": "The answer inferred by your model based on the 'question'"
}

api_key=xxx
model_marker=xxx
api_url=xxx
screenshots_count=3
path_with_index=xxx
save_path=xxx
screenshots_dir=xxx
tokenizer_dir=xxx
num_processes=16
python3 src/infer_gemini.py \
    $path_with_index \
    $save_path \
    $screenshots_dir \
    $screenshots_count \
    $api_key \
    $model_marker \
    $api_url \
    $tokenizer_dir \
    --num_processes $num_processes

# Deploy Qwen2.5-VL-72B-Instruct using vllm
MODEL_DIR="/xxx/Qwen2.5-VL-72B-Instruct"
HOST_IP=$(hostname -i)
model_name=$(basename $MODEL_DIR)
nohup python3 -m vllm.entrypoints.openai.api_server \
    --enforce-eager --swap-space 50 --disable-log-requests \
    --dtype float16 --trust-remote-code \
    --model ${MODEL_DIR} --served-model-name ${model_name} \
    --gpu-memory-utilization 0.9 --port 8088 \
    --max-model-len 32768 --max-seq-len 32768 \
    --limit-mm-per-prompt "image=5"\
    --tensor-parallel-size 8 \
    --seed 1024 > /root/log.ds_server 2> /root/err.ds_server &
sleep 10m

# Evaluate the answers with Qwen2.5-VL-72B-Instruct.
screenshots_count=3
path_with_index=xxx
save_path=xxx
screenshots_dir=xxx
tokenizer_dir=$MODEL_DIR
ip_file_path=$HOST_IP # ip or ip_list_file
num_processes=16
python3 src/infer_qvl.py \
  $path_with_index \
  $save_path \
  $screenshots_dir \
  $screenshots_count \
  $model_name \
  $tokenizer_dir \
  $ip_file_path \
  --num_processes $num_processes

🏆 Leaderboard

For the latest results, please visit the live leaderboard.

The following are the main results on ArtifactsBench, scored by the Gemini-2.5-Pro-0506 referee. A higher score indicates better overall capability in generating visual and interactive artifacts.

• SV: Static Visual, MMD: Mild-to-Moderate Dynamics, HD: High Dynamics, II: Intensive Interactive
• GAME: Game, SVG: SVG Generation, WEB: Web Application, SI: Simulation, MS: Management System
• AVG: Global Average Score

Citation

If you find our project helpful, please cite:

@article{zhang2025artifactsbench,
  title={ArtifactsBench: Bridging the Visual-Interactive Gap in LLM Code Generation Evaluation},
  author={Zhang, Chenchen and Li, Yuhang and Xu, Can and Liu, Jiaheng and Liu, Ao and Zhou, Changzhi and Deng, Ken and Wu, Dengpeng and Huang, Guanhua and Li, Kejiao and others},
  journal={arXiv preprint arXiv:2507.04952},
  year={2025}
}

Contributions

First Authors

• Chenchen Zhang, Tencent
• Yuhang Li, Tencent

Core Contributors

• Can Xu, Tencent
• Jiaheng Liu, NJU
• Ao Liu, Tencent
• Shihui Hu, Tencent

Contributors | Algorithm Support (Alphabet Order)

• Dengpeng Wu, Tencent
• Guanhua Huang, Tencent
• Kejiao Li, Tencent
• Qi Yi, Tencent
• Ruibin Xiong, Tencent
• Haotian Zhu, Tencent
• Yuanxing Zhang, PKU
• Yuhao Jiang, Tencent
• Yue Zhang, Tencent
• Zenan Xu, Tencent

Contributors | Data and Front-End Technical Support (Alphabet Order)

• Bohui Zhai, Tencent
• Guoxiang He, Tencent
• Hebin Li, Tencent
• Jie Zhao, Tencent
• Le Zhang, Tencent
• Lingyun Tan, Tencent
• Pengyu Guo, Tencent
• Xianshu Pang, Tencent
• Yang Ruan, Tencent
• Zhifeng Zhang, Tencent
• Zhonghu Wang, Tencent
• Ziyan Xu, Tencent
• Zuopu Yin, Tencent

Corresponding Authors

• Wiggin Zhou, Tencent
• Chayse Zhou, Tencent
• Fengzong Lian, Tencent
• {wigginzhou,chaysezhou,faxonlian}@tencent.com

If you encounter any issues during testing, please contact adamwzhang@tencent.com.

License

This repository is licensed under the terms of the LICENSE file.