Artifact-Bench is a comprehensive benchmark for evaluating whether Multimodal Large Language Models (MLLMs) can truly detect and reason about the artifacts of AI-generated videos. Instead of focusing only on semantic understanding, Artifact-Bench emphasizes artifact-aware realism perception and fine-grained video analysis across photorealistic, animated, and CG-style domains.
Artifact-Bench defines three complementary tasks:
The following commands can be used to run evaluation on the benchmark using the pipeline provided in the official repository:
python eval/infer_qwen3_vl.py \
--model-path Qwen/Qwen3-VL-8B-Instruct \
--task-dir task \
--output-dir results \
--gpu-ids 0 \
--fps 5
python eval/result_process.py \
--input-dir results \
--output-dir results
Artifact-Bench is only used for academic research. Commercial use in any form is prohibited. The copyright of all videos belongs to the video owners. If there is any infringement in Artifact-Bench, please email frankyang1517@gmail.com and we will remove it immediately. Without prior approval, you cannot distribute, publish, copy, disseminate, or modify Artifact-Bench in whole or in part. You must strictly comply with the above restrictions.
@misc{tang2026artifactbenchevaluatingmllmsdetecting,
title={Artifact-Bench: Evaluating MLLMs on Detecting and Assessing the Artifacts of AI-Generated Videos},
author={Yuqi Tang and Yang Shi and Zhuoran Zhang and Qixun Wang and Xuehai Bai and Yue Ding and Ruizhe Chen and Bohan Zeng and Xinlong Chen and Xuanyu Zhu and Bozhou Li and Yuran Wang and Yifan Dai and Chengzhuo Tong and Xinyu Liu and Yiyan Ji and Yujie Wei and Yuhao Dong and Shilin Yan and Fengxiang Wang and Yi-Fan Zhang and Haotian Wang and Yuanxing Zhang and Pengfei Wan},
year={2026},
eprint={2605.18984},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2605.18984},
}
10 commits
1 commits
Artifact-Bench is a comprehensive benchmark for evaluating whether Multimodal Large Language Models (MLLMs) can truly detect and reason about the artifacts of AI-generated videos. Instead of focusing only on semantic understanding, Artifact-Bench emphasizes artifact-aware realism perception and fine-grained video analysis across photorealistic, animated, and CG-style domains.
Artifact-Bench defines three complementary tasks:
The following commands can be used to run evaluation on the benchmark using the pipeline provided in the official repository:
python eval/infer_qwen3_vl.py \
--model-path Qwen/Qwen3-VL-8B-Instruct \
--task-dir task \
--output-dir results \
--gpu-ids 0 \
--fps 5
python eval/result_process.py \
--input-dir results \
--output-dir results
Artifact-Bench is only used for academic research. Commercial use in any form is prohibited. The copyright of all videos belongs to the video owners. If there is any infringement in Artifact-Bench, please email frankyang1517@gmail.com and we will remove it immediately. Without prior approval, you cannot distribute, publish, copy, disseminate, or modify Artifact-Bench in whole or in part. You must strictly comply with the above restrictions.
@misc{tang2026artifactbenchevaluatingmllmsdetecting,
title={Artifact-Bench: Evaluating MLLMs on Detecting and Assessing the Artifacts of AI-Generated Videos},
author={Yuqi Tang and Yang Shi and Zhuoran Zhang and Qixun Wang and Xuehai Bai and Yue Ding and Ruizhe Chen and Bohan Zeng and Xinlong Chen and Xuanyu Zhu and Bozhou Li and Yuran Wang and Yifan Dai and Chengzhuo Tong and Xinyu Liu and Yiyan Ji and Yujie Wei and Yuhao Dong and Shilin Yan and Fengxiang Wang and Yi-Fan Zhang and Haotian Wang and Yuanxing Zhang and Pengfei Wan},
year={2026},
eprint={2605.18984},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2605.18984},
}
10 commits
1 commits