VTBench: Evaluating Visual Tokenizers for Autoregressive Image Generation

[Paper, Dataset, Space Demo, GitHub Repo]

This repository provides the official implementation of VTBench, a benchmark designed to evaluate the performance of visual tokenizers (VTs) in the context of autoregressive (AR) image generation. VTBench enables fine-grained analysis across three core tasks: image reconstruction, detail preservation, and text preservation, isolating the tokenizer's impact from the downstream generation model.

Our goal is to encourage the development of strong, general-purpose open-source visual tokenizers that can be reliably reused across autoregressive image generation and broader multimodal tasks.

For more details, please visit the official GitHub repository: https://github.com/huawei-lin/VTBench

🔍 Why VTBench?

Recent AR models such as GPT-4o demonstrate impressive image generation quality, which we hypothesize is made possible by a highly capable visual tokenizer. However, most existing VTs significantly lag behind continuous VAEs, leading to:

• Poor reconstruction fidelity
• Loss of structural and semantic detail
• Failure to preserve symbolic information (e.g., text in multilingual images)

VTBench isolates and evaluates VT quality, independent of the downstream model, using standardized tasks and metrics.

✨ Features

• Evaluation on three tasks:
  1. Image Reconstruction (ImageNet, High-Res, Varying-Res)
  2. Detail Preservation (patterns, fine textures)
  3. Text Preservation (posters, academic abstracts, multilingual scripts)
• Supports VTs from models like FlowMo, MaskBiT, OpenMagViT2, VAR, BSQ-ViT, etc.
• Includes baselines from continuous VAEs (e.g., SD3.5L, FLUX.1) and GPT-4o.
• Metrics: PSNR, SSIM, LPIPS, FID, CER, WER

GitHub Repo

For more details, please visit the official GitHub repository: https://github.com/huawei-lin/VTBench

Note: Some portions of the dataset are synthetic, created using GPT-4o.