Awesome Evaluation of Visual Generation

This repository collects methods for evaluating visual generation.

Overview

What You'll Find Here

Within this repository, we collect works that aim to answer some critical questions in the field of evaluating visual generation, such as:

• Model Evaluation: How does one determine the quality of a specific image or video generation model?
• Sample/Content Evaluation: What methods can be used to evaluate the quality of a particular generated image or video?
• User Control Consistency Evaluation: How to tell how well the generated images and videos align with the user controls or inputs?

Updates

This repository is updated periodically. If you have suggestions for additional resources, updates on methodologies, or fixes for expiring links, please feel free to do any of the following:

• raise an Issue,
• nominate awesome related works with Pull Requests,
• We are also contactable via email (ZIQI002 at e dot ntu dot edu dot sg).

Table of Contents

• 1. Evaluation Metrics of Generative Models
  • 1.1. Evaluation Metrics of Image Generation
  • 1.2. Evaluation Metrics of Video Generation
  • 1.3. Evaluation Metrics for Latent Representation
• 2. Evaluation Metrics of Condition Consistency
  • 2.1 Evaluation Metrics of Multi-Modal Condition Consistency
  • 2.2. Evaluation Metrics of Image Similarity
• 3. Evaluation Systems of Generative Models
  • 3.1. Evaluation of Unconditional Image Generation
  • 3.2. Evaluation of Text-to-Image Generation
  • 3.3. Evaluation of Text-Based Image Editing
  • 3.4. Evaluation of Neural Style Transfer
  • 3.5. Evaluation of Video Generation
  • 3.6. Evaluation of Text-to-Motion Generation
  • 3.7. Evaluation of Model Trustworthiness
  • 3.8. Evaluation of Entity Relation
  • 3.9. Agentic Evaluation
  • 3.10. Evaluation of Unified Multimodal Models
• 4. Improving Visual Generation with Evaluation / Feedback / Reward
• 5. Quality Assessment for AIGC
• 6. Study and Rethinking
• 7. Other Useful Resources

1. Evaluation Metrics of Generative Models

1.1. Evaluation Metrics of Image Generation

• Physics-Based Benchmarking Metrics for Multimodal Synthetic Images (2025-11-19)

• Making Reconstruction FID Predictive of Diffusion Generation FID (2026-03-05)

  Note: Introduces iFID.

• Gram-MMD: A Texture-Aware Metric for Image Realism Assessment (2026-04-03)

  Note: Introduces Gram-MMD (GMMD).

• Relative Classification Accuracy: A Calibrated Metric for Identity Consistency in Fine-Grained K-pop Face Generation (2026-01-22)

  Note: Introduces Relative Classification Accuracy (RCA).

• Towards a Scalable Reference-Free Evaluation of Generative Models (2024-07-03)

• FaceScore: Benchmarking and Enhancing Face Quality in Human Generation (2024-06-24)

  Note: Face Score introduced

• Global-Local Image Perceptual Score (GLIPS): Evaluating Photorealistic Quality of AI-Generated Images (2024-05-15)

• Unifying and extending Precision Recall metrics for assessing generative models (2024-05-02)

• Enhancing Plausibility Evaluation for Generated Designs with Denoising Autoencoder (2024-03-08)

  Note: Fréchet Denoised Distance introduced

• Virtual Classifier Error (VCE) from Virtual Classifier: A Reversed Approach for Robust Image Evaluation (2024-03-04)

• An Interpretable Evaluation of Entropy-based Novelty of Generative Models (2024-02-27)

• Semantic Shift Rate from Discovering Universal Semantic Triggers for Text-to-Image Synthesis (2024-02-12)

• Optimizing Prompts Using In-Context Few-Shot Learning for Text-to-Image Generative Models (2024-01-01)

  Note: Quality Loss introduced

• Attribute Based Interpretable Evaluation Metrics for Generative Models (2023-10-26)

• On quantifying and improving realism of images generated with diffusion (2023-09-26)

  Note: Image Realism Score introduced

• Probabilistic Precision and Recall Towards Reliable Evaluation of Generative Models (2023-09-04)

  Note: P-precision and P-recall introduced

• Learning to Evaluate the Artness of AI-generated Images (2023-05-08)

  Note: ArtScore, metric for images resembling authentic artworks by artists

• Training-Free Location-Aware Text-to-Image Synthesis (2023-04-26)

  Note: New evaluation metric for control capability of location aware generation task

• Feature Likelihood Divergence: Evaluating the Generalization of Generative Models Using Samples (2023-02-09)

• LGSQE: Lightweight Generated Sample Quality Evaluatoin (2022-11-08)

• SSD: Towards Better Text-Image Consistency Metric in Text-to-Image Generation (2022-10-27)

  Note: Semantic Similarity Distance introduced

• Layout-Bridging Text-to-Image Synthesis (2022-08-12)

  Note: Layout Quality Score (LQS), new metric for evaluating the generated layout

• Rarity Score: A New Metric to Evaluate the Uncommonness of Synthesized Images (2022-06-17)

• Mutual Information Divergence: A Unified Metric for Multimodal Generative Models (2022-05-25)

  Note: evaluates text to image and utilizes vision language models (VLM)

• TREND: Truncated Generalized Normal Density Estimation of Inception Embeddings for GAN Evaluation (2021-04-30, ECCV 2022)

• CFID from Conditional Frechet Inception Distance (2021-03-21)

• On Self-Supervised Image Representations for GAN Evaluation (2021-01-12)

  Note: SwAV, self-supervised image representation model

• Random Network Distillation as a Diversity Metric for Both Image and Text Generation (2020-10-13)

  Note: RND metric introduced

• The Vendi Score: A Diversity Evaluation Metric for Machine Learning (2022-10-05)

• CIS from Evaluation Metrics for Conditional Image Generation (2020-04-26)

• Text-To-Image Synthesis Method Evaluation Based On Visual Patterns (2020-04-09)

• Cscore: A Novel No-Reference Evaluation Metric for Generated Images (2020-03-25)

• SceneFID from Object-Centric Image Generation from Layouts (2020-03-16)

• Reliable Fidelity and Diversity Metrics for Generative Models (2020-02-23, ICML 2020)
  

• Effectively Unbiased FID and Inception Score and where to find them (2019-11-16, CVPR 2020)
  

• On the Evaluation of Conditional GANs (2019-07-11)

  Note:Fréchet Joint Distance (FJD), which is able to assess image quality, conditional consistency, and intra-conditioning diversity within a single metric.

• Quality Evaluation of GANs Using Cross Local Intrinsic Dimensionality (2019-05-02)

  CrossLID, assesses the local intrinsic dimensionality

• A domain agnostic measure for monitoring and evaluating GANs (2018-11-13)

• Learning to Generate Images with Perceptual Similarity Metrics (2015-11-19)

  Multiscale structural-similarity score introduced

• A No-Reference Image Blur Metric Based on the Cumulative Probability of Blur Detection (CPBD) (2011-03-28)

1.2. Evaluation Metrics of Video Generation

1.3. Evaluation Metrics for Latent Representation

• Linear Separability & Perceptual Path Length (PPL) from A Style-Based Generator Architecture for Generative Adversarial Networks (2020-01-09)

2. Evaluation Metrics of Condition Consistency

2.1 Evaluation Metrics of Multi-Modal Condition Consistency

• NexusScore, NaturalScore and GmeScore from OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generation (2025-06-03)

  Note: NexusScore - Identity Consistency - image retrieval + cosine similarity; NaturalScore - Identity Naturalness - prompting gpt4o; GmeScore - Text - cosine similarity between the GME image and text embeddings.

• FaceSim-Cur from Identity-Preserving Text-to-Video Generation by Frequency Decomposition (2024-11-26)

  Note: NFaceSim-Cur - Face image of human - cosine similarity between the curricularface embeddings of the generated face and the input face.

• Manipulation Direction (MD) from Manipulation Direction: Evaluating Text-Guided Image Manipulation Based on Similarity between Changes in Image and Text Modalities (2023-11-20)

• Semantic Similarity Distance: Towards better text-image consistency metric in text-to-image generation (2022-12-02)

• On the Evaluation of Conditional GANs (2019-07-11)

  Note: Fréchet Joint Distance (FJD), which is able to assess image quality, conditional consistency, and intra-conditioning diversity within a single metric.

• Classification Accuracy Score for Conditional Generative Models (2019-05-26)

  Note: New metric Classification Accuracy Score (CAS)

• Visual-Semantic (VS) Similarity from Photographic Text-to-Image Synthesis with a Hierarchically-nested Adversarial Network (2018-12-26)

• Semantically Invariant Text-to-Image Generation (2018-09-06)

  Note: They evaluate image-text similarity via image captioning

• Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis (2018-01-16)

  Note: An object detector based metric is proposed.

2.2. Evaluation Metrics of Image Similarity

• CLPIPS: A Personalized Metric for AI-Generated Image Similarity (2026-03-26)

• MEt3R: Measuring Multi-View Consistency in Generated Images (2025-01-10)

• Pathology-Guided Virtual Staining Metric for Evaluation and Training (2025-07-16)

  Note: Introduces the Pathology-Aware Perceptual Image Similarity (PaPIS) metric.

The community has also been using DINO or CLIP features to measure the semantic similarity of two images / frames.

There are also recent works on new methods to measure visual similarity (more will be added):

• DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data (2023-06-15)
  

3. Evaluation Systems of Generative Models

3.1. Evaluation of Unconditional Image Generation

• Synthetic Melanoma Image Generation and Evaluation Using Generative Adversarial Networks (2026-03-13)

• AesBench: An Expert Benchmark for Multimodal Large Language Models on Image Aesthetics Perception (2024-01-16)

• A Lightweight Generalizable Evaluation and Enhancement Framework for Generative Models and Generated Samples (2024-04-16)

• Anomaly Score: Evaluating Generative Models and Individual Generated Images based on Complexity and Vulnerability (2023-12-17, CVPR 2024)

• Using Skew to Assess the Quality of GAN-generated Image Features (2023-10-31)

Note: Skew Inception Distance introduced

• StudioGAN: A Taxonomy and Benchmark of GANs for Image Synthesis (2022-06-19)

• HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative Models (2019-04-01)
  

• An Improved Evaluation Framework for Generative Adversarial Networks (2018-03-20)

Note: Class-Aware Frechet Distance introduced

3.2. Evaluation of Text-to-Image Generation

• Simile Understanding in Text-to-Image Models: An Evaluation Framework (2026-08-05)

• Can Text-to-Image Models Draw from the Right Frame of Reference? (2026-08-04)

• ExpertVerse: A General-Purpose Benchmark for Expert-Level Reasoning in Knowledge-Intensive Visual Synthesis (2026-07-21)

• DynEval: Holistic Evaluations of T2I Generative Models in the Wild (2026-07-13)

• TIIF-Bench: How Does Your T2I Model Follow Your Instructions? (2025-06-02)

• PIPBench: A Profile-Inclusive Framework for Personalized Image Generation Evaluation (2026-07-07)

• KathaTrace: Diagnosing Semantic Trajectory Collapse in Generated Visual Narratives (2026-07-01)

• Are Text-to-Image Models Inductivist Turkeys? A Counterfactual Benchmark for Causal Reasoning (2026-06-23)

• DSH-Bench: A Difficulty- and Scenario-Aware Benchmark with Hierarchical Subject Taxonomy for Subject-Driven Text-to-Image Generation (2026-03-09)

• Scaling Multi-Reference Image Generation with Dynamic Reward Optimization (2026-06-25)

• Qwen-Image-Bench: From Generation to Creation in Text-to-Image Evaluation (2026-05-27)

• Can AI Draw Science? A Benchmark for Evaluating Scientific Figure Generation by Text-to-Image and Multimodal Models (2026-06-24)

• GeoFidelity-Bench: Evaluating Segment-Level Geographic Fidelity in Text-to-Image Street-View Generation (2026-06-22)

• WeGenBench: A Multidimensional Diagnostic Benchmark towards Text-to-Image Model Optimization (2026-06-18)

• GarmentSketch: Large-scale Sketch-to-Fashion Benchmark (2026-06-12)

• The 1st PortraitCraft Challenge: A CVPR 2026 Workshop Competition on Portrait Composition Understanding and Generation (2026-06-09)

• Faithful, Enriched, and Precise: Benchmarking Natural-Science Illustration Generation by T2I models (2026-06-04)

• TASTE: A Designer-Annotated Multi-Dimensional Preference Dataset for AI-Generated Graphic Design (2026-05-20)

• TextAlign: Preference Alignment for Text Rendering with Hierarchical Rewards (2026-05-19)

• DetailMaster: Can Your Text-to-Image Model Handle Long Prompts? (2025-05-22)

• Benchmarking and Enhancing Text-to-Image Models for Generating Visual Representations in Early Arithmetic Education (2026-05-29)

• DyCoRM: Dynamic Criterion-Aware Reward Modeling for Text-to-Image Generation (2026-05-25)

• MaSC: A Masked Similarity Metric for Evaluating Concept-Driven Generation (2026-05-21)

• A Framework for Evaluating Zero-Shot Image Generation in Concept-based Explainability (2026-05-19)

• Benchmarking and Evolving Reason-Reflect-Rectify for Reflective Visual Generation (2026-05-19)

• ColorConceptBench: A Benchmark for Probabilistic Color-Concept Understanding in Text-to-Image Models (2026-01-23)

• TIQA: Human-Aligned Perceptual Text Quality Assessment in Generated Images (2026-03-07)

• Benchmarking Layout-Guided Diffusion Models through Unified Semantic-Spatial Evaluation in Closed and Open Settings (2026-04-28)

• Knowledge Visualization: A Benchmark and Method for Knowledge-Intensive Text-to-Image Generation (2026-04-24)

• Pref-GRPO: Pairwise Preference Reward-based GRPO for Stable Text-to-Image Reinforcement Learning (2025-08-28)

• ViStoryBench: Comprehensive Benchmark Suite for Story Visualization (2025-05-30)

• When Identities Collapse: A Stress-Test Benchmark for Multi-Subject Personalization (2026-03-27)

• BizGenEval: A Systematic Benchmark for Commercial Visual Content Generation (2026-03-26)

• Diffusion Probe: Generated Image Result Prediction Using CNN Probes (2026-02-27)

• TechImage-Bench: Rubric-Based Evaluation for Technical Image Generation (2025-12-13)

• Evaluating Generative Models via One-Dimensional Code Distributions (2026-03-09)

• The Art That Poses Back: Assessing AI Pastiches after Contemporary Artworks (2026-03-06)

• Toward Early Quality Assessment of Text-to-Image Diffusion Models (2026-03-03)

• Enhancing Spatial Understanding in Image Generation via Reward Modeling (2026-02-27)

• UniGenBench++: A Unified Semantic Evaluation Benchmark for Text-to-Image Generation (2025-10-21)

• Everything in Its Place: Benchmarking Spatial Intelligence of Text-to-Image Models (2026-01-28)

• Text-to-Image Diffusion Models Cannot Count, and Prompt Refinement Cannot Help (2025-03-10)

• TempViz: On the Evaluation of Temporal Knowledge in Text-to-Image Models (2026-01-21)

• Scientific Image Synthesis: Benchmarking, Methodologies, and Downstream Utility (2026-01-17)

• Architecture inside the mirage: evaluating generative image models on architectural style, elements, and typologies (2026-01-14)

• GenEval 2: Addressing Benchmark Drift in Text-to-Image Evaluation (2025-12-18)

• Color Bind: Exploring Color Perception in Text-to-Image Models (2025-08-27)

• Human-Centred Evaluation of Text-to-Image Generation Models for Self-expression of Mental Distress: A Dataset Based on GPT-4o (2025-10-26)

• Prompt fidelity of ChatGPT4o / Dall-E3 text-to-image visualisations (2025-10-21)

• CAP: Evaluation of Persuasive and Creative Image Generation (2024-12-10)

• TIT-Score: Evaluating Long-Prompt Based Text-to-Image Alignment via Text-to-Image-to-Text Consistency (2025-10-03)

• AcT2I: Evaluating and Improving Action Depiction in Text-to-Image Models (2025-09-19)

• STRICT: Stress Test of Rendering Images Containing Text (2025-05-25)

• MagicMirror: A Large-Scale Dataset and Benchmark for Fine-Grained Artifacts Assessment in Text-to-Image Generation (2025-09-12)

• FLUX-Reason-6M & PRISM-Bench: A Million-Scale Text-to-Image Reasoning Dataset and Comprehensive Benchmark (2025-09-11)

• Evaluating Text-to-Image and Text-to-Video Synthesis with a Conditional Fréchet Distance (2025-03-27)

• T2I-ReasonBench: Benchmarking Reasoning-Informed Text-to-Image Generation (2025-08-24)

• 7Bench: a Comprehensive Benchmark for Layout-guided Text-to-image Models (2025-08-18)

• VISTAR: A User-Centric and Role-Driven Benchmark for Text-to-Image Evaluation (2025-08-08)

• Scendi Score: Prompt-Aware Diversity Evaluation via Schur Complement of CLIP Embeddings (2024-12-24)

• KITTEN: A Knowledge-Intensive Evaluation of Image Generation on Visual Entities (2024-10-15)

• Enhancing Reward Models for High-quality Image Generation: Beyond Text-Image Alignment (2025-07-25)

• Hierarchical Fusion and Joint Aggregation: A Multi-Level Feature Representation Method for AIGC Image Quality Assessment (2025-07-23)

• Evaluating Attribute Confusion in Fashion Text-to-Image Generation (2025-07-09)

• OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generation (2025-06-09)

• Q-Eval-100K: Evaluating Visual Quality and Alignment Level for Text-to-Vision Content (2025-03-04)

• GenSpace: Benchmarking Spatially-Aware Image Generation (2025-05-30)

• DIMCIM: A Quantitative Evaluation Framework for Default-mode Diversity and Generalization in Text-to-Image Generative Models (2025-06-05)

• T2I-FactualBench: Benchmarking the Factuality of Text-to-Image Models with Knowledge-Intensive Concepts (2024-12-05)

• MIBE: Multi-subject Interaction Benchmark and Evaluator for Personalized Image Generation (2026-07-01)

• CogCanvas: A Benchmark for Evaluating Multi-Subject Reference-Based Image Generation (2026-06-14)

• GlazyBench: A Benchmark for Ceramic Glaze Property Prediction and Image Generation (2026-05-07)

• PortraitCraft: A Benchmark for Portrait Composition Understanding and Generation (2026-04-04)

• AIBench: Evaluating Visual-Logical Consistency in Academic Illustration Generation (2026-03-30)

• MultiBind: A Benchmark for Attribute Misbinding in Multi-Subject Generation (2026-03-23)

• Counting Hallucinations in Diffusion Models (2025-10-15)

• Beyond the Pixels: VLM-based Evaluation of Identity Preservation in Reference-Guided Synthesis (2025-11-11)

• SinSEMI: A One-Shot Image Generation Model and Data-Efficient Evaluation Framework for Semiconductor Inspection Equipment (2025-11-10)

• Mind-the-Glitch: Visual Correspondence for Detecting Inconsistencies in Subject-Driven Generation (2025-09-26)

• Visual Personalization Turing Test (2026-01-30, CVPR 2026)

• Finer-Personalization Rank: Fine-Grained Retrieval Examines Identity Preservation for Personalized Generation (2025-12-22)

• Human Preference-Aligned Concept Customization Benchmark via Decomposed Evaluation (2025-09-03)

• GenColorBench: A Color Evaluation Benchmark for Text-to-Image Generation Models (2025-10-23)

• SCIEval: Evaluating and Benchmarking the Faithfulness of Scientific Image Generation and Interpretation with Large Multimodal Models (CVPR 2026)

• ServImage: An Image Generation and Editing Benchmark from Real-World Commercial Imaging Services (2026-04)

  Note: Introduces ServImageScore.

• LongT2IBench: A Benchmark for Evaluating Long Text-to-Image Generation with Graph-Structured Annotations (2025-12-10)

• OverLayBench: A Benchmark for Layout-to-Image Generation with Dense Overlaps (2025-09-23)

  Note: Introduces OverLayScore.

• ICE-Bench: A Unified and Comprehensive Benchmark for Image Creating and Editing (2025-03)

  Note: Introduces VLLM-QA.

• MultiHuman-Testbench: Benchmarking Image Generation for Multiple Humans (2025-06-25)

• MMMG: A Massive, Multidisciplinary, Multi-Tier Generation Benchmark for Text-to-Image Reasoning (2025-06-12)

  Note: Introduces MMMG-Score.

• MMIG-Bench: Towards Comprehensive and Explainable Evaluation of Multi-Modal Image Generation Models (2025-05)

  Note: Introduces the Aspect Matching Score (AMS).

• SciFlow-Bench: Evaluating Structure-Aware Scientific Diagram Generation via Inverse Parsing (2026-02-10)

• IGenBench: Benchmarking the Reliability of Text-to-Infographic Generation (2026-01-08)

• Automatic and Reliable Evaluation for Academic Caption-to-Figure Generation with LMMs (ACL 2026)

• Scendi Score from Scendi Score: Prompt-Aware Diversity Evaluation via Schur Complement of CLIP Embeddings (ICCV 2025)

  Note: Introduces the Scendi Score for prompt-aware diversity evaluation.

• P2P: Automated Paper-to-Poster Generation and Fine-Grained Benchmark (ICLR 2026)

• Too Vivid to Be Real? Benchmarking and Calibrating Generative Color Fidelity (CVPR 2026)

  Note: Introduces the Color Fidelity Metric (CFM).

• PosterIQ: A Design Perspective Benchmark for Poster Understanding and Generation (2026-03-25)

• MultiBanana: A Challenging Benchmark for Multi-Reference Text-to-Image Generation (2025-11-28)

• GenExam: A Multidisciplinary Text-to-Image Exam (2025-09-17)

• What Makes a Scene ? Scene Graph-based Evaluation and Feedback for Controllable Generation (2024-05-26)

• Commonsense-T2I Challenge: Can Text-to-Image Generation Models Understand Commonsense? (2024-08-12)

• WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generation (2025-03-10)

• Why Settle for One? Text-to-ImageSet Generation and Evaluation (2025-06-29)

• LMM4LMM: Benchmarking and Evaluating Large-multimodal Image Generation with LMMs (2025-04-11)

• Robust and Discriminative Speaker Embedding via Intra-Class Distance Variance Regularization (2018-09)

  Note: IntraClass Average Distance(ICAD) - Diversity

• REAL: Realism Evaluation of Text-to-Image Generation Models for Effective Data Augmentation. (2025-02-15)

• ABHINAW: A method for Automatic Evaluation of Typography within AI-Generated Images (2024-09-18)

• Finding the Subjective Truth: Collecting 2 Million Votes for Comprehensive Gen-AI Model Evaluation (2024-09-18)

• Beyond Aesthetics: Cultural Competence in Text-to-Image Models (2024-07-09)

  Note: CUBE benchmark introduced

• MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation? (2024-07-05)

  Note: MJ-Bench introduced

• MIGC++: Advanced Multi-Instance Generation Controller for Image Synthesis (2024-07-02)

  Note: Benchmark COCO-MIG and Multimodal-MIG introduced

• Analyzing Quality, Bias, and Performance in Text-to-Image Generative Models (2024-06-28)

• EvalAlign: Evaluating Text-to-Image Models through Precision Alignment of Multimodal Large Models with Supervised Fine-Tuning to Human Annotations (2024-06-24)

• DreamBench++: A Human-Aligned Benchmark for Personalized Image Generation (2024-06-24)

• Six-CD: Benchmarking Concept Removals for Benign Text-to-image Diffusion Models (2024-06-21)

• Evaluating Numerical Reasoning in Text-to-Image Models (2024-06-20)

  Note: GeckoNum introduced

• Holistic Evaluation for Interleaved Text-and-Image Generation (2024-06-20)

  Note: InterleavedBench and InterleavedEval metric introduced

• GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generation (2024-06-19)

• Decomposed evaluations of geographic disparities in text-to-image models (2024-06-17)

  Note: new metric Decomposed Indicators of Disparities introduced

• PhyBench: A Physical Commonsense Benchmark for Evaluating Text-to-Image Models (2024-06-17)

  Note: PhyBench introduced

• Make It Count: Text-to-Image Generation with an Accurate Number of Objects (2024-06-14)

• Commonsense-T2I Challenge: Can Text-to-Image Generation Models Understand Commonsense? (2024-06-11)

  Note: Commonsense-T2I, benchmark for real-life commonsense reasoning capabilities of T2I models

• Unified Text-to-Image Generation and Retrieval (2024-06-09)

  Note: TIGeR-Bench, benchmark for evaluation of unified text-to-image generation and retrieval.

• PQPP: A Joint Benchmark for Text-to-Image Prompt and Query Performance Prediction (2024-06-07)

• GenAI Arena: An Open Evaluation Platform for Generative Models (2024-06-06)

• A-Bench: Are LMMs Masters at Evaluating AI-generated Images? (2024-06-05)
  

• Multidimensional Preference Score from Learning Multi-dimensional Human Preference for Text-to-Image Generation (2024-05-23)

• Evolving Storytelling: Benchmarks and Methods for New Character Customization with Diffusion Models (2024-05-20)

  Note: NewEpisode benchmark introduced

• Training-free Subject-Enhanced Attention Guidance for Compositional Text-to-image Generation (2024-05-11)

  Note: GroundingScore metric introduced

• TheaterGen: Character Management with LLM for Consistent Multi-turn Image Generation (2024-04-29)

  Note: consistent score r introduced

• Exposing Text-Image Inconsistency Using Diffusion Models (2024-04-28)

• Revisiting Text-to-Image Evaluation with Gecko: On Metrics, Prompts, and Human Ratings (2024-04-25)

• Multimodal Large Language Model is a Human-Aligned Annotator for Text-to-Image Generation (2024-04-23)

• Infusion: Preventing Customized Text-to-Image Diffusion from Overfitting (2024-04-22)

  Note: Latent Fisher divergence and Wasserstein metric introduced

• TAVGBench: Benchmarking Text to Audible-Video Generation (2024-04-22)
  

• Object-Attribute Binding in Text-to-Image Generation: Evaluation and Control (2024-04-21)

• Magic Clothing: Controllable Garment-Driven Image Synthesis (2024-04-15)

  Note: new metric Matched-Points-LPIPS introduced

• GenAI-Bench: A Holistic Benchmark for Compositional Text-to-Visual Generation (2024-04-09)

  Note: GenAI-Bench was introduced in a previous paper 'Evaluating Text-to-Visual Generation with Image-to-Text Generation'

• Detect-and-Compare from Identity Decoupling for Multi-Subject Personalization of Text-to-Image Models (2024-04-05)

• Enhancing Text-to-Image Model Evaluation: SVCS and UCICM (2024-04-02)

  Note: Evaluation metrics: Semantic Visual Consistency Score and User-Centric Image Coherence Metric

• Evaluating Text-to-Visual Generation with Image-to-Text Generation (2024-04-01)
  

• Measuring Style Similarity in Diffusion Models (2024-04-01)
  

• AAPMT: AGI Assessment Through Prompt and Metric Transformer (2024-03-28)

• FlashEval: Towards Fast and Accurate Evaluation of Text-to-image Diffusion Generative Models (2024-03-25)

• Refining Text-to-Image Generation: Towards Accurate Training-Free Glyph-Enhanced Image Generation (2024-03-25)

  Note: LenCom-Eval introduced

• Exploring GPT-4 Vision for Text-to-Image Synthesis Evaluation (2024-03-20)

• DialogGen: Multi-modal Interactive Dialogue System for Multi-turn Text-to-Image Generation (2024-03-13)

  Note: DialogBen introduced

• Evaluating Text-to-Image Generative Models: An Empirical Study on Human Image Synthesis (2024-03-08)

• An Information-Theoretic Evaluation of Generative Models in Learning Multi-modal Distributions (2024-02-13)
  

• MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesis (2024-02-08)

  Note: COCO-MIG benchmark introduced

• CAS: A Probability-Based Approach for Universal Condition Alignment Score (2024-01-16)
  

  Note: Condition alignment of text-to-image, {instruction, image}-to-image, edge-/scribble-to-image, and text-to-audio

• EmoGen: Emotional Image Content Generation with Text-to-Image Diffusion Models (2024-01-09)

  Note: emotion accuracy, semantic clarity and semantic diversity are not core contributions of this paper

• VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluation (2023-12-22)
  

• PIA: Your Personalized Image Animator via Plug-and-Play Modules in Text-to-Image Models (2023-12-21)

  Note: AnimateBench, benchmark for comparisons in the field of personalized image animation

• Stellar: Systematic Evaluation of Human-Centric Personalized Text-to-Image Methods (2023-12-11)
  

• A Contrastive Compositional Benchmark for Text-to-Image Synthesis: A Study with Unified Text-to-Image Fidelity Metrics (2023-12-04)
  

• The Challenges of Image Generation Models in Generating Multi-Component Images (2023-11-22)

• SelfEval: Leveraging the discriminative nature of generative models for evaluation (2023-11-17)

• GPT-4V(ision) as a Generalist Evaluator for Vision-Language Tasks (2023-11-02)

• Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generation (2023-10-27, ICLR 2024)
  

• DEsignBench: Exploring and Benchmarking DALL-E 3 for Imagining Visual Design (2023-10-23)
  

• GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment (2023-10-17)
  

• Hypernymy Understanding Evaluation of Text-to-Image Models via WordNet Hierarchy (2023-10-13)
  

• SingleInsert: Inserting New Concepts from a Single Image into Text-to-Image Models for Flexible Editing (2023-10-12)
  

  Note: New Metric: Editing Success Rate

• ImagenHub: Standardizing the evaluation of conditional image generation models (2023-10-02)
  

• Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluation (2023-09-26, ICLR 2024)
  

• Concept Score from Text-to-Image Generation for Abstract Concepts (2023-09-26)

• OpenLEAF: Open-Domain Interleaved Image-Text Generation and Evaluation (2023-09-23) GenAI-Arena

  Note: Evaluates task of image and text generation

• Progressive Text-to-Image Diffusion with Soft Latent Direction (2023-09-18)

  Note: Benchmark for text-to-image generation tasks

• AltDiffusion: A Multilingual Text-to-Image Diffusion Model (2023-08-19, AAAI 2024)

  Note: Benchmark with focus on multilingual generation aspect

• LEICA from Likelihood-Based Text-to-Image Evaluation with Patch-Level Perceptual and Semantic Credit Assignment (2023-08-16)

• Let's ViCE! Mimicking Human Cognitive Behavior in Image Generation Evaluation (2023-07-18)

• T2I-CompBench: A Comprehensive Benchmark for Open-world Compositional Text-to-image Generation (2023-07-12)
  

• TIAM -- A Metric for Evaluating Alignment in Text-to-Image Generation (2023-07-11, WACV 2024)
  

• Divide, Evaluate, and Refine: Evaluating and Improving Text-to-Image Alignment with Iterative VQA Feedback (2023-07-10, NeurIPS 2023)
  

• Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis (2023-06-15)
  

• ConceptBed: Evaluating Concept Learning Abilities of Text-to-Image Diffusion Models (2023-06-07, AAAI 2024)
  

• Visual Programming for Text-to-Image Generation and Evaluation (2023-05-24, NeurIPS 2023)
  

• LLMScore: Unveiling the Power of Large Language Models in Text-to-Image Synthesis Evaluation (2023-05-18, NeurIPS 2023)
  

• X-IQE: eXplainable Image Quality Evaluation for Text-to-Image Generation with Visual Large Language Models (2023-05-18)
  

• What You See is What You Read? Improving Text-Image Alignment Evaluation (2023-05-17, NeurIPS 2023)
  

• Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation (2023-05-02)
  

• Analysis of Appeal for Realistic AI-Generated Photos (2023-04-17)

• Appeal and quality assessment for AI-generated images (2023-06-22)

• Diagnostic Benchmark and Iterative Inpainting for Layout-Guided Image Generation (2023-04-13)

• HRS-Bench: Holistic, Reliable and Scalable Benchmark for Text-to-Image Models (2023-04-11, ICCV 2023)
  

• Human Preference Score: Better Aligning Text-to-Image Models with Human Preference (2023-03-25, ICCV 2023)
  

• A study of the evaluation metrics for generative images containing combinational creativity (2023-03-23)

  Note: Consensual Assessment Technique and Turing Test used in T2I evaluation

• TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering (2023-03-21, ICCV 2023)
  

• Is This Loss Informative? Faster Text-to-Image Customization by Tracking Objective Dynamics (2023-02-09)

  Note: an evaluation approach for early stopping criterion in T2I customization

• Benchmarking Spatial Relationships in Text-to-Image Generation (2022-12-20)
  

• MMI and MOR from from Benchmarking Robustness of Multimodal Image-Text Models under Distribution Shift (2022-12-15)

• TeTIm-Eval: a novel curated evaluation data set for comparing text-to-image models (2022-12-15)

• Human Evaluation of Text-to-Image Models on a Multi-Task Benchmark (2022-11-22)

• UPainting: Unified Text-to-Image Diffusion Generation with Cross-modal Guidance (2022-10-28)

  Note: UniBench, benchmark contains prompts for simple-scene images and complex-scene images in Chinese and English

• Re-Imagen: Retrieval-Augmented Text-to-Image Generator (2022-09-29)

  Note: EntityDrawBench, benchmark to evaluates image generation for diverse entities

• Vision-Language Matching for Text-to-Image Synthesis via Generative Adversarial Networks (2022-08-20)

  Note: new metric, Vision-Language Matching Score (VLMS)

• Scaling Autoregressive Models for Content-Rich Text-to-Image Generation (2022-06-22)

• GR-GAN: Gradual Refinement Text-to-image Generation (2022-05-23)

  Note: new metric Cross-Model Distance introduced

• DrawBench from Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (2022-05-23)

• StyleT2I: Toward Compositional and High-Fidelity Text-to-Image Synthesis (2022-03-29, CVPR 2024)

  Note: Evaluation metric for compositionality of T2I models

• Benchmark for Compositional Text-to-Image Synthesis (2021-07-29)
  

• TISE: Bag of Metrics for Text-to-Image Synthesis Evaluation (2021-12-02, ECCV 2022)
  

• Improving Generation and Evaluation of Visual Stories via Semantic Consistency (2021-05-20)

• Leveraging Visual Question Answering to Improve Text-to-Image Synthesis (2020-10-28)

• Image Synthesis from Locally Related Texts (2020-06-08)

  Note: VQA accuracy as a new evaluation metric.

• Semantic Object Accuracy for Generative Text-to-Image Synthesis (2019-10-29)
  

  Note: new evaluation metric, Semantic Object Accuracy (SOA)

• GPT-ImgEval: A Comprehensive Benchmark for Diagnosing GPT4o in Image Generation (2025-04-03)

• R2I-Bench: Benchmarking Reasoning-Driven Text-to-Image Generation (2025-05-28)

3.3. Evaluation of Text-Based Image Editing

• MIEScore: Human-Aligned Evaluation for Multi-Source Image Editing (2026-08-03)

• MPIE-Bench: Benchmarking Anatomically Plausible Multi-Person Interaction Editing (2026-07-30)

• RS-RIE-Bench: Benchmarking Reasoning-Guided Remote Sensing Image Editing (2026-07-22)

• WeEdit: A Dataset, Benchmark and Glyph-Guided Framework for Text-centric Image Editing (2026-03-12)

• Do Image Editing Models Understand Lighting? (2026-06-25)

• ChartSync: A Benchmark for Visuo-Logical Cascading Chart Editing (2026-07-11)

• PhyEditBench: A Real-World Multi-Stage Benchmark for Physics-Aware Image Editing (2026-06-25)

• An Extensive Benchmark for Single-round and Multi-round Instruction-based Image Editing (2026-06-14)

• VDE Bench: Evaluating The Capability of Image Editing Models to Modify Visual Documents (2026-01-27)

• PaintBench: Deterministic Evaluation of Precise Visual Editing (2026-05-29)

• TextSculptor: Training and Benchmarking Scene Text Editing (2026-05-20)

• MULTITEXTEDIT: Benchmarking Cross-Lingual Degradation in Text-in-Image Editing (2026-05-04)

• DLEBench: Evaluating Small-scale Object Editing Ability for Instruction-based Image Editing Model (2026-02-27)

• StyleText: A Large-Scale Dataset and Benchmark for Stylized Scene Text Inpainting (2026-05-17)

• Editor's Choice: Evaluating Abstract Intent in Image Editing through Atomic Entity Analysis (2026-05-14)

• Do-Undo Bench: Reversibility for Action Understanding in Image Generation (2025-12-15)

• Edit-Compass & EditReward-Compass: A Unified Benchmark for Image Editing and Reward Modeling (2026-05-13)

• PlanViz: Evaluating Planning-Oriented Image Generation and Editing for Computer-Use Tasks (2026-02-06)

• Towards In-Context Tone Style Transfer with A Large-Scale Triplet Dataset (2026-04-17)

• Is This Edit Correct? A Multi-Dimensional Benchmark for Reasoning-Aware Image Editing (2026-04-16)

• Redefining Quality Criteria and Distance-Aware Score Modeling for Image Editing Assessment (2026-04-14)

• AIM-Bench: Benchmarking and Improving Affective Image Manipulation via Fine-Grained Hierarchical Control (2026-04-12)

• SpatialEdit: Benchmarking Fine-Grained Image Spatial Editing (2026-04-06)

• MIRAGE: Benchmarking and Aligning Multi-Instance Image Editing (2026-04-06)

• FED-Bench: A Cross-Granular Benchmark for Disentangled Evaluation of Facial Expression Editing (2026-03-31)

• GEditBench v2: A Human-Aligned Benchmark for General Image Editing (2026-03-30)

• ImagenWorld: Stress-Testing Image Generation Models with Explainable Human Evaluation on Open-ended Real-World Tasks (2026-03-29)

• VIBE: How Well Do Models Follow Visual Instructions? (2026-02-02)

• Evaluating Image Editing with LLMs: A Comprehensive Benchmark and Intermediate-Layer Probing Approach (2026-03-20)

• Edit Spillover as a Probe: Do Image Editing Models Implicitly Understand World Relations? (2026-03-18)

• Scalable Evaluation of the Realism of Synthetic Environmental Augmentations in Images (2026-03-04)

• InEdit-Bench: Benchmarking Intermediate Logical Pathways for Intelligent Image Editing Models (2026-03-04)

• Human-Aligned MLLM Judges for Fine-Grained Image Editing Evaluation: A Benchmark, Framework, and Analysis (2026-02-13)

• WorldEdit: Towards Open-World Image Editing with a Knowledge-Informed Benchmark (2026-02-06)

• LocateEdit-Bench: A Benchmark for Instruction-Based Editing Localization (2026-02-05)

• HYPE-EDIT-1: Benchmark for Measuring Reliability in Frontier Image Editing Models (2026-01-25)

• Fine-Grained Human Pose Editing Assessment via Layer-Selective MLLMs (2026-01-15)

• PICABench: How Far Are We from Physically Realistic Image Editing? (2025-10-20)

• RealDrag: The First Dragging Benchmark with Real Target Image (2025-12-13)

• WiseEdit: Benchmarking Cognition- and Creativity-Informed Image Editing (2025-11-29)

• UniREditBench: A Unified Reasoning-based Image Editing Benchmark (2025-11-03)

• MedGEN-Bench: Contextually Entangled Benchmark for Open-ended Multimodal Medical Generation (2025-11-17)

• MedEBench: Diagnosing Reliability in Text-Guided Medical Image Editing (2025-06-02)

• SpotEdit: Evaluating Visually-Guided Image Editing Methods (2025-08-25)

• EditGarment: An Instruction-Based Garment Editing Dataset Constructed with Automated MLLM Synthesis and Semantic-Aware Evaluation (2025-08-05)

• ADIEE: Automatic Dataset Creation and Scorer for Instruction-Guided Image Editing Evaluation (2025-07-09)

• GIE-Bench: Towards Grounded Evaluation for Text-Guided Image Editing (2025-05-16)

• ComplexBench-Edit: Benchmarking Complex Instruction-Driven Image Editing via Compositional Dependencies (2025-06-15)

• ByteMorph: Benchmarking Instruction-Guided Image Editing with Non-Rigid Motions (2025-06-03)

• RefEdit: A Benchmark and Method for Improving Instruction-based Image Editing Model on Referring Expressions (2025-06-03)

• A Systematic Benchmark of Intraoperative Ultrasound-to-MR Synthesis for Brain Tumour Surgery (2026-05-30)

• PhysEditBench: A Protocol-Conditioned Benchmark for Dense Physical-Map Prediction with Image Editors (2026-05-13)

• WildRelight: A Real-World Benchmark and Physics-Guided Adaptation for Single-Image Relighting (2026-05-12)

• StableI2I: Spotting Unintended Changes in Image-to-Image Transition (2026-05-06)

• ViGoR-Bench: How Far Are Visual Generative Models From Zero-Shot Visual Reasoners? (2026-03-26)

• GEBench: Benchmarking Image Generation Models as GUI Environments (2026-02-09)

• Vectra: A New Metric, Dataset, and Model for Visual Quality Assessment in E-Commerce In-Image Machine Translation (2026-01-31)

• Contrast-X: A Multi-Modal Contrast Image Synthesis Benchmark and Universal Modality Flow Matching (2026-01-22)

• Lightweight CycleGAN Models for Cross-Modality Image Transformation and Experimental Quality Assessment in Fluorescence Microscopy (2025-10-17)

• Human Body Restoration with One-Step Diffusion Model and A New Benchmark (2025-02-03)

• Probing Visual Planning in Image Editing Models (2026-04-23)

• EditInspector: A Benchmark for Evaluation of Text-Guided Image Edits (2025-06-11, ACL 2025)

• Reference-Free Image Quality Assessment for Virtual Try-On via Human Feedback (2026-03-13, ECCV 2026)

  Note: Introduces the reference-free VTON-IQA framework and the human-annotated VTON-QBench.

• VTEdit-Bench: A Comprehensive Benchmark for Multi-Reference Image Editing Models in Virtual Try-On (2026-03-12, ECCV 2026)

  Note: Introduces VTEdit-QA to assess model consistency, cloth consistency, and overall image quality.

• Virtual Try-On for Cultural Clothing: A Benchmarking Study (2026-03-07)

  Note: Introduces BD-VITON for benchmarking virtual try-on on culturally diverse Bangladeshi garments.

• OpenVTON-Bench: A Large-Scale High-Resolution Benchmark for Controllable Virtual Try-On Evaluation (2026-01-30)

  Note: Evaluates background consistency, identity fidelity, texture fidelity, shape plausibility, and overall realism.

• VTONQA: A Multi-Dimensional Quality Assessment Dataset for Virtual Try-on (2026-01-06)

  Note: Provides human opinion scores for clothing fit, body compatibility, and overall quality across images from 11 VTON models.

• Omni IIE Bench: Benchmarking the Practical Capabilities of Image Editing Models (2026-03)

• I2I-Bench: A Comprehensive Benchmark Suite for Image-to-Image Editing Models (2025-12-04)

• Inter-Edit: First Benchmark for Interactive Instruction-Based Image Editing (CVPR 2026)

• CREval: An Automated Interpretable Evaluation for Creative Image Manipulation under Complex Instructions (2026-03)

• LMM4Edit: Benchmarking and Evaluating Multimodal Image Editing with LMMs (2025-07)

  Note: Introduces LMM4Edit.

• CompBench: Benchmarking Complex Instruction-Guided Image Editing (2025-05)

• Does FLUX Already Know How to Perform Physically Plausible Image Composition? (2025-09)

• Taming I2V Models for Image HOI Editing: A Cognitive Benchmark and Agentic Self-Correcting Framework (2026-06)

• MICE-Bench: A Challenging and Comprehensive Benchmark for Multi-Reference Image Creation and Editing (ICML 2026)

• UnicEdit-10M: A Dataset and Benchmark Breaking the Scale-Quality Barrier via Unified Verification for Reasoning-Enriched Edits (CVPR 2026)

• Learning Action and Reasoning-Centric Image Editing from Videos and Simulations (2024-07-03)

  Note: AURORA-Bench introduced

• GIM: A Million-scale Benchmark for Generative Image Manipulation Detection and Localization (2024-06-24)

• MultiEdits: Simultaneous Multi-Aspect Editing with Text-to-Image Diffusion Models (2024-06-03)

  Note: PIE-Bench++, evaluating image-editing tasks involving multiple objects and attributes

• DiffUHaul: A Training-Free Method for Object Dragging in Images (2024-06-03)

  Note: foreground similarity, object traces and realism metric introduced

• HQ-Edit: A High-Quality Dataset for Instruction-based Image Editing (2024-04-15)

• FlexEdit: Flexible and Controllable Diffusion-based Object-centric Image Editing (2024-03-27)

  Note: novel automatic mask-based evaluation metric tailored to various object-centric editing scenarios

• TransformationOriented Paired Benchmark from InstructBrush: Learning Attention-based Instruction Optimization for Image Editing (2024-03-27)

• ImageNet Concept Editing Benchmark from Editing Massive Concepts in Text-to-Image Diffusion Models (2024-03-20)

• Editing Massive Concepts in Text-to-Image Diffusion Models (2024-03-20)

  Note: ImageNet Concept Editing Benchmark (ICEB), for evaluating massive concept editing for T2I models

• Make Me Happier: Evoking Emotions Through Image Diffusion Models (2024-03-13)

  Note: EMR, ESR, ENRD, ESS metric introduced

• Diffusion Model-Based Image Editing: A Survey (2024-02-27)
  

  Note: EditEval, benchmark for text-guided image editing and LLM Score

• Towards Efficient Diffusion-Based Image Editing with Instant Attention Masks (2024-01-15, AAAI 2024)

  Note: Editing-Mask, new benchmark to examine the mask accuracy and local editing ability

• RotationDrag: Point-based Image Editing with Rotated Diffusion Features (2024-01-12)

  Note: RotationBench introduced

• LEDITS++: Limitless Image Editing using Text-to-Image Models (2023-11-28)
  

  Note: TEdBench++, revised benchmark of TEdBench

• Emu Edit: Precise Image Editing via Recognition and Generation Tasks (2023-11-16)

• EditVal: Benchmarking Diffusion Based Text-Guided Image Editing Methods (2023-10-03)
  

• PIE-Bench from Direct Inversion: Boosting Diffusion-based Editing with 3 Lines of Code (2023-10-02)

• Iterative Multi-granular Image Editing using Diffusion Models (2023-09-01)

• DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editing (2023-06-26)

  Note: drawbench benchmark introduced

• DreamEdit: Subject-driven Image Editing (2023-06-22)

  Note: DreamEditBench benchmark introduced

• MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing (2023-06-16)

  Note: dataset only

• Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpainting (2022-12-13, CVPR 2023)
  

• Imagic: Text-Based Real Image Editing with Diffusion Models (2022-10-17)
  

  Note: TEdBench, image editing benchmark

• Predict, Prevent, and Evaluate: Disentangled Text-Driven Image Manipulation Empowered by Pre-Trained Vision-Language Model (2021-11-26)

• Knowledge-Driven Generative Adversarial Network for Text-to-Image Synthesis (2021-09-29)

  Note: New evaluation system, Pseudo Turing Test (PTT)

• ManiGAN: Text-Guided Image Manipulation (2019-12-12)

  Note: manipulative precision metric introduced

• Text Guided Person Image Synthesis (2019-04-10)

  Note: VQA perceptual score introduced

3.4. Evaluation of Neural Style Transfer

• ArtFID: Quantitative Evaluation of Neural Style Transfer (2022-07-25)

3.5. Evaluation of Video Generation

3.5.1. Evaluation of Text-to-Video Generation

• CultureVidBench: Benchmarking Cultural Understanding in Text-to-Video Generation (2026-08-03)

• FilmBench: A Film-Grade Benchmark for Cinematic Video Generation (2026-07-27)

• When Cultures Move: Measuring and Improving Multicultural Text-to-Video Generation (2026-05-16)

• Multi-Dimensional Quality Assessment for AI-Generated Human-Centric Videos: Dataset and Model (2026-07-18)

• MultiRef-Compass: Towards Comprehensive Evaluation of Multi-Reference-to-Audio-Video Generation (2026-07-15)

• VGIF-Score: Interpretable and Diagnostic Evaluation of Spatio-Temporal Instruction Following in Video Generation (2026-07-15)

• GeoT2V-Bench: Benchmarking 3D Consistency in Text-to-Video Models via 3D Reconstruction (2026-06-23)

• Measuring 3D Spatial Geometric Consistency in Dynamic Video Generation (2026-03-19)

• MSAVBench: Towards Comprehensive and Reliable Evaluation of Multi-Shot Audio-Video Generation (2026-05-19)

• CultureScore: Evaluating Cultural Faithfulness in Video Generation Models (2026-06-05)

• BRITE: A Benchmark for Reliable and Interpretable T2V Evaluation on Implausible Scenarios (2026-04-24)

• Evaluating Design Video Generation: Metrics for Compositional Fidelity (2026-05-15)

• PEDRA: Evaluating the Realism of Pedestrian Dynamics in Video Generation (2025-10-23)

• T2AV-Compass: Towards Unified Evaluation for Text-to-Audio-Video Generation (2025-12-24)

• OmniHuman: A Large-scale Dataset and Benchmark for Human-Centric Video Generation (2026-04-20)

• LoCoT2V-Bench: Benchmarking Long-Form and Complex Text-to-Video Generation (2025-10-30)

• LongAV-Compass: Towards Unified Evaluation of Minute-Scale Audio-Visual Generation Across T2AV, I2AV, and V2AV (2026-05-25)

• AVBench: Human-Aligned and Automated Evaluation Benchmark for Audio-Video Generative Models (2026-05-23)

• EduVQA: Towards Concept-Aware Assessment of Educational AI-Generated Videos (2026-03-03)

• WorldJen: An End-to-End Multi-Dimensional Benchmark for Generative Video Models (2026-05-05)

• HumanScore: Benchmarking Human Motions in Generated Videos (2026-04-22)

• AVGen-Bench: A Task-Driven Benchmark for Multi-Granular Evaluation of Text-to-Audio-Video Generation (2026-04-09)

• MSG Score: Automated Video Verification for Reliable Multi-Scene Generation (2024-11-28)

• VABench: A Comprehensive Benchmark for Audio-Video Generation (2025-12-10)

• MSVBench: Towards Human-Level Evaluation of Multi-Shot Video Generation (2026-02-27)

• Q-Save: Towards Scoring and Attribution for Generated Video Evaluation (2025-11-24)

• PhyEduVideo: A Benchmark for Evaluating Text-to-Video Models for Physics Education (2026-01-02)

• GRADEO: Towards Human-Like Evaluation for Text-to-Video Generation via Multi-Step Reasoning (2025-03-04)

• Are AI-Generated Driving Videos Ready for Autonomous Driving? A Diagnostic Evaluation Framework (2025-12-06)

• RULER-Bench: Probing Rule-based Reasoning Abilities of Next-level Video Generation Models for Vision Foundation Intelligence (2025-12-02)

• V-ReasonBench: Toward Unified Reasoning Benchmark Suite for Video Generation Models (2025-11-20)

• Temporal Realism Evaluation of Generated Videos Using Compressed-Domain Motion Vectors (2025-11-17)

• How Far Are Surgeons from Surgical World Models? A Pilot Study on Zero-shot Surgical Video Generation with Expert Assessment (2025-11-03)

• SeqBench: Benchmarking Sequential Narrative Generation in Text-to-Video Models (2025-10-14)

• DynamicEval: Rethinking Evaluation for Dynamic Text-to-Video Synthesis (2025-10-08)

• Stable Cinemetrics : Structured Taxonomy and Evaluation for Professional Video Generation (2025-09-30)

• GeneVA: A Dataset of Human Annotations for Generative Text to Video Artifacts (2025-09-10)

• T2VEval: Benchmark Dataset and Objective Evaluation Method for T2V-generated Videos (2025-01-15)

• MagicAnime: A Hierarchically Annotated, Multimodal and Multitasking Dataset with Benchmarks for Cartoon Animation Generation (2025-07-27)

• T2VWorldBench: A Benchmark for Evaluating World Knowledge in Text-to-Video Generation (2025-07-24)

• FIFA: Unified Faithfulness Evaluation Framework for Text-to-Video and Video-to-Text Generation (2025-07-09)

• Beyond Time Shifts: Adapting Omni-LLM as a Reference-Free Evaluator for Generative Audio-Visual Models (2026-07-10)

• Artifact-Bench: Evaluating MLLMs on Detecting and Assessing the Artifacts of AI-Generated Videos (2026-05-18)

• EntityBench: Towards Entity-Consistent Long-Range Multi-Shot Video Generation (2026-05-14)

• Omni-Judge: Can Omni-LLMs Serve as Human-Aligned Judges for Text-Conditioned Audio-Video Generation? (2026-02-02)

• VideoASMR-Bench: Can AI-Generated ASMR Videos Fool VLMs and Humans? (2025-12-15)

• Spotlight: Identifying and Localizing Video Generation Errors Using VLMs (2025-11-22)

• NarrLV: Towards a Comprehensive Narrative-Centric Evaluation for Long Video Generation (2025-07-15, ICLR 2026)

• Comparison Drives Preference: Reference-Aware Modeling for AI-Generated Video Quality Assessment (2026-04-18)

  Note: Introduces Reference-aware Video Quality Assessment (RefVQA).

• Towards Holistic Visual Quality Assessment of AI-Generated Videos: A LLM-Based Multi-Dimensional Evaluation Model (2025-06-05, CVPR Workshop 2025)

• Physics Question Scene Graph: Fine-grained Evaluation of Physical Plausibility in Text-to-Video Generation (2026-06-24)

• Trimming the Long-Tail of Visual World Modeling Evaluation (2026-06-23)

• Quantitative Video World Model Evaluation for Geometric-Consistency (2026-05-14)

• PhyGround: Benchmarking Physical Reasoning in Generative World Models (2026-05-11)

• Physion-Eval: Evaluating Physical Realism in Generated Video via Human Reasoning (2026-03-20)

• 3DSPA: A 3D Semantic Point Autoencoder for Evaluating Video Realism (2026-02-23)

• Benchmarking Scientific Understanding and Reasoning for Video Generation using VideoScience-Bench (2025-12-02)

• PhyDetEx: Detecting and Explaining the Physical Plausibility of T2V Models (2025-12-01)

• Objects in Generated Videos Are Slower Than They Appear: Models Suffer Sub-Earth Gravity and Don't Know Galileo's Principle...for now (2025-12-01)

• World Consistency Score: A Unified Metric for Video Generation Quality (2025-07-31)

• Can Your Model Separate Yolks with a Water Bottle? Benchmarking Physical Commonsense Understanding in Video Generation Models (2025-07-21)

• "PhyWorldBench": A Comprehensive Evaluation of Physical Realism in Text-to-Video Models (2025-07-17)

• VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulness (2025-03-27)

• Artifact-Aware Evaluation for High-Quality Video Generation (2026-01-28)

• How Far are AI-generated Videos from Simulating the 3D Visual World: A Learned 3D Evaluation Approach (2024-06-27)

• EvalVerse: Pipeline-Aware and Expert-Calibrated Benchmarking for Professional Cinematic Video Generation (2026-05-22)

  Note: Introduces a pipeline-aware, expert-calibrated framework for evaluating cinematic quality, acting, aesthetics, multi-shot sequencing, and audio-visual integration.

• MuSS: A Large-Scale Dataset and Cinematic Narrative Benchmark for Multi-Shot Subject-to-Video Generation (2026-04-26)

  Note: Introduces the Cinematic Narrative Benchmark and Anti-Copy-Paste Variance (ACP-Var) metric for evaluating continuous storytelling and 3D structural consistency.

• VGA-Bench: A Unified Benchmark and Multi-Model Framework for Video Aesthetics and Generation Quality Evaluation (2026-04-11, CVPR 2026)

  Note: Jointly evaluates aesthetic quality, aesthetic tagging, and generation quality with VAQA-Net, VTag-Net, and VGQA-Net.

• Ref4D-VideoBench: Four-Dimensional Reference-Based Evaluation of Text-to-Video Generative Models (CVPR 2026)

• SVBench: Evaluation of Video Generation Models on Social Reasoning (2025-12-25)

• FingER: Content Aware Fine-Grained Evaluation with Reasoning for AI-Generated Videos (2025-04)

• OSCBench: Benchmarking Object State Change in Text-to-Video Generation (2026-03-12)

• MTAVG-Bench: A Diagnostic Benchmark for Multi-Talker Dialogue-Centric Audio-Video Generation (2026-02)

• VQ-Insight: Teaching VLMs for AI-Generated Video Quality Understanding via Progressive Visual Reinforcement Learning (2025-06-23)

• ViStoryBench: Comprehensive Benchmark Suite for Story Visualization (CVPR 2026)

• LikePhys: Evaluating Intuitive Physics Understanding in Video Diffusion Models via Likelihood Preference (2025-10-13, ICLR 2026)

  Note: likelihood-preference benchmark for evaluating intuitive physics understanding in video diffusion models.

• VideoScore2: Think before You Score in Generative Video Evaluation (2025-09-26)

• Stable Cinemetrics: Structured Taxonomy and Evaluation for Professional Video Generation (2025-09)

• Are Synthetic Videos Useful? A Benchmark for Retrieval-Centric Evaluation of Synthetic Videos (2025-07-03)

• AIGVE-MACS: Unified Multi-Aspect Commenting and Scoring Model for AI-Generated Video Evaluation (2025-07-02)

• BrokenVideos: A Benchmark Dataset for Fine-Grained Artifact Localization in AI-Generated Videos (2025-06-25)

• OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generation (2025-06-03)

  Note: The first open-sourced infrastructure (OpenS2V-Eval & OpenS2V-5M) for Subject-to-Video generation

• CineTechBench: A Benchmark for Cinematographic Technique Understanding and Generation (2025-05)

• LOVE: Benchmarking and Evaluating Text-to-Video Generation and Video-to-Text Interpretation (2025-05-17)

• On the Consistency of Video Large Language Models in Temporal Comprehension (2025-05-17)

• AIGVE-Tool: AI-Generated Video Evaluation Toolkit with Multifaceted Benchmark (2025-04-18)

• VideoGen-Eval: Agent-based System for Video Generation Evaluation (2025-03-30)
  

• Video-Bench: Human Preference Aligned Video Generation Benchmark (2025-04-07)
  

• Morpheus: Benchmarking Physical Reasoning of Video Generative Models with Real Physical Experiments (2025-04-03)

• Envisioning Beyond the Pixels: Benchmarking Reasoning-Informed Visual Editing (2025-04-03)

• VinaBench: Benchmark for Faithful and Consistent Visual Narratives (2025-03-26)

• ETVA: Evaluation of Text-to-Video Alignment via Fine-grained Question Generation and Answering (2025-03-21)

• UVE: Are MLLMs Unified Evaluators for AI-Generated Videos? (2025-03-13)

• Is Your World Simulator a Good Story Presenter? A Consecutive Events-Based Benchmark for Future Long Video Generation (2024-12-17)

  Note: focus on storytelling.

• Neuro-Symbolic Evaluation of Text-to-Video Models using Formal Verification (2024-12-03)

  Note: focus on temporally text-video alignment (event order, accuracy)

• AIGV-Assessor: Benchmarking and Evaluating the Perceptual Quality of Text-to-Video Generation with LMM (2024-11-26)

  Note: fuild motion, light change, motion speed, event order.

• Human-Activity AGV Quality Assessment: A Benchmark Dataset and an Objective Evaluation Metric (2024-11-25)

  Note: Introduces the AI-Generated Human Activity Video Quality metric (GHVQ).

• What You See Is What Matters: A Novel Visual and Physics-Based Metric for Evaluating Video Generation Quality (2024-11-24)

  Note: texture evaluation scheme introduced

• The Dawn of Video Generation: Preliminary Explorations with SORA-like Models (2024-10-10)

• Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation (2024-10-07)

  Note: Comprehensive physical (optical, mechanic, thermal, material) benchmark introduced

• Benchmarking AIGC Video Quality Assessment: A Dataset and Unified Model (2024-07-31)

• T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation (2024-07-19)

• T2VSafetyBench: Evaluating the Safety of Text-to-Video Generative Models (2024-07-08)

  Note: T2VSafetyBench introduced

• Evaluation of Text-to-Video Generation Models: A Dynamics Perspective (2024-07-01)

• T2VBench: Benchmarking Temporal Dynamics for Text-to-Video Generation (2024-06)

• Evaluating and Improving Compositional Text-to-Visual Generation (2024-06)

• TlTScore: Towards Long-Tail Effects in Text-to-Visual Evaluation with Generative Foundation Models (2024-06)

• ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of Text-to-Time-lapse Video Generation (2024-06-26)

  Note: Comprehensive time-lapse (biological, human created, meteorological, physical) benchmark introduced

• VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generation (2024-06-21)

• TC-Bench: Benchmarking Temporal Compositionality in Text-to-Video and Image-to-Video Generation (2024-06-12)

  Note: TC-Bench, TCR and TC-Score introduced

• VideoPhy: Evaluating Physical Commonsense for Video Generation (2024-06-05)

• Illumination Histogram Consistency Metric for Quantitative Assessment of Video Sequences (2024-05-15)

• The Lost Melody: Empirical Observations on Text-to-Video Generation From A Storytelling Perspective (2024-05-13)

  Note: New evaluation framework T2Vid2T, Evaluation for storytelling aspects of videos

• Exposing AI-generated Videos: A Benchmark Dataset and a Local-and-Global Temporal Defect Based Detection Method (2024-05-07)

• Sora Detector: A Unified Hallucination Detection for Large Text-to-Video Models (2024-05-07)

  Note: hallucination detection

• Exploring AIGC Video Quality: A Focus on Visual Harmony, Video-Text Consistency and Domain Distribution Gap (2024-04-21)
  

• Subjective-Aligned Dataset and Metric for Text-to-Video Quality Assessment (2024-03-18)
  

• A dataset of text prompts, videos and video quality metrics from generative text-to-video AI models (2024-02-22)
  

• Sora Generates Videos with Stunning Geometrical Consistency (2024-02-27)
  

• STREAM: Spatio-TempoRal Evaluation and Analysis Metric for Video Generative Models (2024-01-30)
  

• Towards A Better Metric for Text-to-Video Generation (2024-01-15)
  

• PEEKABOO: Interactive Video Generation via Masked-Diffusion (2023-12-12)
  

  Note: Benchmark for interactive video generation

• VBench: Comprehensive Benchmark Suite for Video Generative Models (2023-11-29)
  

• SmoothVideo: Smooth Video Synthesis with Noise Constraints on Diffusion Models for One-shot Video Tuning (2023-11-29)

• FETV: A Benchmark for Fine-Grained Evaluation of Open-Domain Text-to-Video Generation (2023-11-03)
  

• EvalCrafter: Benchmarking and Evaluating Large Video Generation Models (2023-10-17)
  

• Measuring the Quality of Text-to-Video Model Outputs: Metrics and Dataset (2023-09-14)

• StoryBench: A Multifaceted Benchmark for Continuous Story Visualization (2023-08-22, NeurIPS 2023)
  

• Exploring Video Quality Assessment on User Generated Contents from Aesthetic and Technical Perspectives (2023-03-07, ICCV 2023)
  

  Note: Aesthetic View & Technical View

• CelebV-Text: A Large-Scale Facial Text-Video Dataset (2023-03-26, CVPR 2023)
  

  Note: Benchmark on Facial Text-to-Video Generation

• Make It Move: Controllable Image-to-Video Generation with Text Descriptions (2021-12-06, CVPR 2022)
  

3.5.2. Evaluation of Image-to-Video Generation

• GAUGE: A Measurement-Grounded Benchmark for Physical Fidelity in Simulation Engines and Video World Models (2026-08-06)

• MBench: A Comprehensive Benchmark on Memory Capability for Video World Models (2026-05-30)

• WBench: A Comprehensive Multi-turn Benchmark for Interactive Video World Model Evaluation (2026-05-25)

• VideoVerse: Does Your T2V Generator Have World Model Capability to Synthesize Videos? (2025-10-09)

• WorldReasonBench: Human-Aligned Stress Testing of Video Generators as Future World-State Predictors (2026-05-11)

• Towards Generative Predictive Display for Vision-Based Teleoperation: A Zero-Shot Benchmark of Off-the-Shelf Video Models (2026-05-10)

• MME-CoF-Pro: Evaluating Reasoning Coherence in Video Generative Models with Text and Visual Hints (2026-03-20)

• RISE-Video: Can Video Generators Decode Implicit World Rules? (2026-02-05)

• AnimeAgent: Is the Multi-Agent via Image-to-Video models a Good Disney Storytelling Artist? (2026-02-24)

• Thinking in Video: Can Video Generators Really Reason About the Real World? (2026-07-20)

• KeyFrame-Compass: Towards Comprehensive Evaluation of Keyframe-Conditioned Video Generation (2026-07-15)

• Beyond the Last Frame: Process-aware Evaluation for Generative Video Reasoning (2025-12-31)

• Can World Simulators Reason? Gen-ViRe: A Generative Visual Reasoning Benchmark (2025-11-17)

• Drive&Gen: Co-Evaluating End-to-End Driving and Video Generation Models (2025-10-07)

• SimWorld: A Unified Benchmark for Simulator-Conditioned Scene Generation via World Model (2025-03-18)

  Note: Benchmarks simulator-conditioned world-model image generation for autonomous-driving scenes.

• MechVerse: Evaluating Physical Motion Consistency in Video Generation Models (2026-05-14)

• WorldMark: A Unified Benchmark Suite for Interactive Video World Models (2026-04-23)

• DrivingGen: A Comprehensive Benchmark for Generative Video World Models in Autonomous Driving (2026-01-04)

• WorldLens: Full-Spectrum Evaluations of Driving World Models in Real World (2025-12-11)

• Evaluating Newtonian Mechanics in Video Generative Models with Real Physical Systems (2025-04-03, ICML 2026)

• WorldExam: Benchmarking World Models from Apparent Appearance to Inherent Reactivity (2026-08-03)

  Note: Evaluates visual quality, control adherence, spatial consistency, and inherent world reactivity.

• Learning Explicit Physical Parameter Control and Benchmarking for Video Generation (2026-07-21)

  Note: Introduces PhyParam-Bench for temporal dynamics, spatial stability, and semantic-physical alignment in I2V generation.

• AnimationBench: Are Video Models Good at Character-Centric Animation? (2026-04-16)

  Note: Evaluates animation principles, IP preservation, semantic consistency, motion rationality, and camera-motion consistency.

• Out of Sight, Out of Mind? Evaluating State Evolution in Video World Models (2026-03-13)

  Note: Introduces STEVO-Bench to evaluate whether state evolution continues correctly under interrupted observation.

• TiViBench: Benchmarking Think-in-Video Reasoning for Video Generative Models (2025-11-17)

  Note: Benchmarks structural, spatial, logical, and action-planning reasoning in I2V models.

• UI2V-Bench: An Understanding-based Image-to-video Generation Benchmark (2025-09-29)

  Note: Evaluates spatial understanding, attribute binding, category understanding, and reasoning in I2V models.

• HyperMotionX: The Dataset and Benchmark with DiT-Based Pose-Guided Human Image Animation of Complex Motions (2025-05-29)

  Note: Introduces HyperMotionX Bench for pose-guided human image animation under complex motions.

• VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Models (2024-11-20)

• I2V-Bench from ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generation (2024-02-06)
  

• AIGCBench: Comprehensive Evaluation of Image-to-Video Content Generated by AI (2024-01-03)
  

• A Benchmark for Controllable Text-Image-to-Video Generation (2023-06-12)

• Temporal Shift GAN for Large Scale Video Generation (2020-04-04)

  Note: Symmetric-Similarity-Score introduced

• Video Imagination from a Single Image with Transformation Generation (2017-06-13)

  Note: RIQA metric introduced

3.5.3. Evaluation of Talking Face Generation

• Temporally-Aligned Evaluation for Audio-Driven Talking Head Generation (2026-05-31)

• REACT 2026: The Fourth Multiple Appropriate Facial Reaction Generation Challenge: Personalised MAFRG and Appropriate EEG Reaction Prediction (2026-06-06)

• EvalTalker: Learning to Evaluate Real-Portrait-Driven Multi-Subject Talking Humans (2025-12-01)

• THEval. Evaluation Framework for Talking Head Video Generation (2025-11-06, CVPR 2026 Findings)

• Assessing Identity Leakage in Talking Face Generation: Metrics and Evaluation Framework (2025-11-05, ICASSP 2026)

• Preview WB-DH: Towards Whole Body Digital Human Bench for the Generation of Whole-body Talking Avatar Videos (2025-08-12)

• Who is a Better Talker: Subjective and Objective Quality Assessment for AI-Generated Talking Heads (2025-07-31)

• JWB-DH-V1: Benchmark for Joint Whole-Body Talking Avatar and Speech Generation Version 1 (2025-07-28)

• OpFlowTalker: Realistic and Natural Talking Face Generation via Optical Flow Guidance (2024-05-23)

  Note: VTCS to measures lip-readability in synthesized videos

• Audio-Visual Speech Representation Expert for Enhanced Talking Face Video Generation and Evaluation (2024-05-07)

• VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time (2024-04-16)

  Note: Contrastive Audio and Pose Pretraining (CAPP) score introduced

• THQA: A Perceptual Quality Assessment Database for Talking Heads (2024-04-13)
  

• A Comparative Study of Perceptual Quality Metrics for Audio-driven Talking Head Videos (2024-03-11)
  

• Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expert (2023-03-29, CVPR 2023)
  

  Note: Measuring intelligibility of the generated videos

• Sparse in Space and Time: Audio-visual Synchronisation with Trainable Selectors (2022-10-13)
  

• Responsive Listening Head Generation: A Benchmark Dataset and Baseline (2021-12-27, ECCV 2022)
  

• A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild (2020-08-23)

  Note: new metric LSE-D and LSE-C introduced

• What comprises a good talking-head video generation?: A Survey and Benchmark (2020-05-07)

3.5.4. Evaluation of World Generation

• WorldModelBench: Judging Video Generation Models As World Models (2025-02)

• WorldScore: A Unified Evaluation Benchmark for World Generation (2025-04-01)

3.5.5. Evaluation of Video Editing

• Decouple and Track: Benchmarking and Improving Video Diffusion Transformers for Motion Transfer (2025-03-21)

  Note: Introduces MTBench and a hybrid motion-fidelity metric.

• A new dataset and comparison for multi-camera frame synthesis (2025-08-12)

• OmniEdit-Bench: A Comprehensive Benchmark for Instruction-based Video Editing (2026-08-05)

• MEDit-Bench: A Dataset for Evaluating Message-Driven Narrative Video Editing (2026-07-28)

• AVE-Compass: Towards Holistic Evaluation for Audio-Video Editing Abilities (2026-07-17)

• Goku: A Million-Scale Universal Dataset and Benchmark for Instruction-Based Video Editing (2026-06-29)

• CoVEBench: Can Video Editing Models Handle Complex Instructions? (2026-06-07)

• V2V-Bench: A Comprehensive Benchmark for Video-to-Video Generation Evaluation (2026-06-04, ICML 2026 Workshop)

• PROVE: A Perceptual RemOVal cohErence Benchmark for Visual Media (2026-05-14, ACM MM 2026)

• VEFX-Bench: A Holistic Benchmark for Generic Video Editing and Visual Effects (2026-04-17)

• UniEditBench: A Unified and Cost-Effective Benchmark for Image and Video Editing via Distilled MLLMs (2026-04-17)

• Physics-Aware Video Instance Removal Benchmark (2026-04-07)

• Hi-Light: A Path to high-fidelity, high-resolution video relighting with a Novel Evaluation Paradigm (2026-01-30)

• Beyond Rigid: Benchmarking Non-Rigid Video Editing (2026-01-26)

• DirectSwap: Mask-Free Cross-Identity Training and Benchmarking for Expression-Consistent Video Head Swapping (2025-12-10)

• EgoEdit: Dataset, Real-Time Streaming Model, and Benchmark for Egocentric Video Editing (2025-12-05)

• IVEBench: Modern Benchmark Suite for Instruction-Guided Video Editing Assessment (2025-10-13, ICLR 2026)

• FiVE: A Fine-grained Video Editing Benchmark for Evaluating Emerging Diffusion and Rectified Flow Models (2025-03-17)

• EditBoard: Towards a Comprehensive Evaluation Benchmark for Text-Based Video Editing Models (2024-09)

• Multi-modal Segment Assemblage Network for Ad Video Editing with Importance-Coherence Reward (2022-09-25, ACCV 2022)

3.6. Evaluation of Text-to-Motion Generation

• Self-Consistency for LLM-Based Motion Trajectory Generation and Verification (2026-03-31, CVPR 2026)

  Note: Uses transformation-aware self-consistency to generate and verify motion-graphics trajectories.

• Bilingual Text-to-Motion Generation: A New Benchmark and Baselines (2026-03-26)

  Note: Introduces the bilingual BiHumanML3D benchmark and the BiMD baseline.

• T2MBench: A Benchmark for Out-of-Distribution Text-to-Motion Generation (2026-02-14)

  Note: Evaluates semantic alignment, generalizability, physical quality, and fine-grained accuracy under OOD prompts.

• IRG-MotionLLM: Interleaving Motion Generation, Assessment and Refinement for Text-to-Motion Generation (2025-12-11)

  Note: Interleaves motion generation, assessment, and refinement through iterative text-motion dialogue.

• ReAlign: Text-to-Motion Generation via Step-Aware Reward-Guided Alignment (2025-11-24, AAAI 2026)

  Note: Introduces a step-aware reward model for text-motion semantic consistency and motion realism.

• The Quest for Generalizable Motion Generation: Data, Model, and Evaluation (2025-10-30, ICLR 2026)

  Note: Introduces MBench for motion quality, prompt fidelity, and generalization ability.

• FineMotion: A Dataset and Benchmark with both Spatial and Temporal Annotation for Fine-grained Motion Generation and Editing (2025-07-26, ICCV 2025)

  Note: Provides fine-grained spatial and temporal annotations for motion generation and editing.

• ReAlign: Bilingual Text-to-Motion Generation via Step-Aware Reward-Guided Alignment (2025-05-08)

  Note: Combines the BiHumanML3D benchmark with step-aware reward-guided alignment.

• VMBench: A Benchmark for Perception-Aligned Video Motion Generation (2024-03-13)

• MoDiPO: text-to-motion alignment via AI-feedback-driven Direct Preference Optimization (2024-05-06)

• What is the Best Automated Metric for Text to Motion Generation? (2023-09-19)

• Text-to-Motion Retrieval: Towards Joint Understanding of Human Motion Data and Natural Language (2023-05-25)
  

  Note: Evaluation protocol for assessing the quality of the retrieved motions

• Establishing a Unified Evaluation Framework for Human Motion Generation: A Comparative Analysis of Metrics (2024-05-13)

• Evaluation of text-to-gesture generation model using convolutional neural network (2021-10-11)

3.7. Evaluation of Model Trustworthiness

3.7.1. Evaluation of Visual-Generation-Model Trustworthiness

• Innocent Panels, Hateful Stories: Evaluating and Detecting Hateful Intent in Multi-Turn Visual Story Generation (2026-08-05)

• Child-Oriented AIGC Video Risk Reviewing: A Benchmark and Knowledge-Supported Iterative Reasoning Framework (2026-07-21)

• NAMESAKES: Probing Identity Memorization in Text-to-Image Models (2026-06-18)

• Inference-Time Concept Suppression and Video-Centric Evaluation for Text-to-Video Models (2026-07-15)

• Beyond wheelchairs and blindfolds: Investigating disability stereotypes in T2I models with INCLUDE-BENCH (2026-07-09)

• AI's Blind Spots: Geographic Knowledge and Diversity Deficit in Generated Urban Scenario (2025-06-20)

• VPA-Guard: Defending and Benchmarking Image-to-Video Generation Against Visual Prompt Attacks (2026-06-24)

• Happy Young Women, Grumpy Old Men? Emotion-Driven Demographic Biases in Synthetic Face Generation (2026-01-18)

• When Safe Concepts Become Unsafe: Multi-Concept Compositional Vulnerabilities in Text-to-Image Models (2026-04-17)

• BAFIS: Dataset + Framework to assess occupational Bias and Human Preference in modern Text-to-image Models (2026-06-18)

• Generating Fearful Images: Investigating Potential Emotional Biases in Image-Generation Models (2024-11-08)

• When Cars Have Stereotypes: Auditing Demographic Bias in Objects from Text-to-Image Models (2025-08-05)

• CheXGenBench: A Unified Benchmark For Fidelity, Privacy and Utility of Synthetic Chest Radiographs (2025-05-15)

• Aligned but Stereotypical? How System Prompts Shape Demographic Bias in LLM-Based Text-to-Image Models (2025-12-04)

• Gender Artifacts from Art History to Text-to-Image Generation (2026-06-04)

• Toward Trustworthy Portrait Editing: Evaluation of Demographic Misrepresentation in I2I Models (2026-02-18)

• Exposing Blindspots: Cultural Bias Evaluation in Generative Image Models (2025-10-22)

• SynCred-Bench: Benchmarking Synthetic Credibility in AI-Generated Visual Misinformation (2026-06-02)

• RoboTrustBench: Benchmarking the Trustworthiness of Video World Models for Robotic Manipulation (2026-06-01)

• SafeGen-Bench: Benchmarking Safety in Image-Conditioned Text-to-Video Generation (2026-05-31)

• Broken Memories: Detecting and Mitigating Memorization in Diffusion Models with Degraded Generations (2026-05-21)

• Hallucination Behavior in Multimodal LLMs Across Agricultural Image Interpretation and Generation Tasks (2026-05-26)

• FairJudge: Abstention-Aware Multimodal Judges for Fairness and Alignment Evaluation in Text-to-Image Models (2025-10-26)

• The Silent Brush: Evaluating Artistic Style Leakage in AI Art Generation (2026-05-17)

• Red-Teaming Text-to-Image Models via In-Context Experience Replay and Semantic-Preserving Prompt Rewriting (2024-11-25)

• Latent Space Probing for Adult Content Detection in Video Generative Models (2026-04-25)

• T2I-BiasBench: A Multi-Metric Framework for Auditing Demographic and Cultural Bias in Text-to-Image Models (2026-04-14)

• Text-to-Image Models and Their Representation of People from Different Nationalities Engaging in Activities (2025-04-08)

• GLEaN: A Text-to-image Bias Detection Approach for Public Comprehension (2026-04-10)

• FlowGuard: Towards Lightweight In-Generation Safety Detection for Diffusion Models via Linear Latent Decoding (2026-04-09)

• DialectGen: Benchmarking and Improving Dialect Robustness in Multimodal Generation (2025-10-16)

• Erasure or Erosion? Evaluating Compositional Degradation in Unlearned Text-To-Image Diffusion Models (2026-04-06)

• From Unlearning to UNBRANDING: A Benchmark for Trademark-Safe Text-to-Image Generation (2025-12-15)

• PROBE: Diagnosing Residual Concept Capacity in Erased Text-to-Video Diffusion Models (2026-03-23)

• REFORGE: Multi-modal Attacks Reveal Vulnerable Concept Unlearning in Image Generation Models (2026-03-17)

• When Generative Augmentation Hurts: A Benchmark Study of GAN and Diffusion Models for Bias Correction in AI Classification Systems (2026-03-17)

• Culture in Action: Evaluating Text-to-Image Models through Social Activities (2025-11-07)

• Investigating Disability Representations in Text-to-Image Models (2026-02-04)

• GeoDiv: Framework For Measuring Geographical Diversity In Text-To-Image Models (2026-02-25)

• Synthetic History: Evaluating Visual Representations of the Past in Diffusion Models (2025-05-18)

• From Preferences to Prejudice: The Role of Alignment Tuning in Shaping Social Bias in Video Diffusion Models (2025-10-20)

• Aesthetics as Structural Harm: Algorithmic Lookism Across Text-to-Image Generation and Classification (2026-01-15)

• Are Video Generation Models Geographically Fair? An Attraction-Centric Evaluation of Global Visual Knowledge (2026-01-26)

• Wukong Framework for Not Safe For Work Detection in Text-to-Image systems (2025-08-01)

• CulturalFrames: Assessing Cultural Expectation Alignment in Text-to-Image Models and Evaluation Metrics (2025-06-10)

• Beautiful Images, Toxic Words: Understanding and Addressing Offensive Text in Generated Images (2025-02-07)

• Prompting4Debugging: Red-Teaming Text-to-Image Diffusion Models by Finding Problematic Prompts (2023-09-12)

• Bias Detection and Rotation-Robustness Mitigation in Vision-Language Models and Generative Image Models (2026-01-09)

• How Many Images Does It Take? Estimating Imitation Thresholds in Text-to-Image Models (2024-10-19)

• MineTheGap: Automatic Mining of Biases in Text-to-Image Models (2025-12-15)

• DREAM: Scalable Red Teaming for Text-to-Image Generative Systems via Distribution Modeling (2025-07-22) [](https