banana100-additional-iqa-models

This repository evaluates iteratively generated images with learned image/text quality and preference metrics. The original generation prompts are not assumed to be available, so a vision-language model captions one reference image per image group. Those captions are then reused as prompts for the IQA models.

This repo only considers no-reference metrics: prompt-conditioned learned metrics, image-only learned quality/preference metrics, and a COCO-detector compositional diagnostic.

Aggregate Figures

The tracked aggregate figures summarize metric failures across generators and score streams. Per-metric/per-group figures are generated artifacts and are ignored by Git.

Models

The captioning step uses Qwen2.5-VL-7B-Instruct.

Repository Layout

data/
    raw/
        images/
            <generator>/<group>/run_idx_<1-5>_step_<001-020>.png
    processed/
        captions/captions.json
        scores/<metric>/scores.csv
configs/
    metrics.json       # metric labels and score directions used by plots
figures/
    combined_failure_*.png/.pdf
    <metric>/<generator>/<group>.png
logs/
    *.log
models/
    caption/        # Qwen cache if captions are regenerated
    pickscore/      # PickScore files and HF cache
    hpsv2/          # HPS checkpoint and HF/OpenCLIP cache
    imagereward/    # ImageReward checkpoint/config and HF cache
    geneval/        # Cascade Mask R-CNN config/checkpoint
    hpsv3/          # HPSv3 checkpoint and HF cache
    vqascore/       # VQAScore / CLIP-FlanT5 HF cache
    mps_overall/    # MPS source checkout, checkpoint, and HF cache
    imagedoctor/    # ImageDoctor HF cache
    icthp/          # ICT/HP source checkout, checkpoints, and HF cache
    tifa/           # TIFA source checkout, question cache, and HF cache
src/
    caption/
        download_and_test.sh
        generate_captions.py
    models/<model>/
        download_and_test.sh
        evaluate.py
    plot_scores.py
    run_eval_parallel.sh
    run_iqa.py
venvs/
    caption/
    pickscore/
    hpsv2/
    imagereward/
    geneval/
    hpsv3/
    vqascore/
    mps_overall/
    imagedoctor/
    icthp/
    tifa/

Model files live under models/. Python environments live under venvs/. Transient runtime caches are kept under logs/.cache/ where possible.

Pipeline

1. Put generated images under data/raw/images/<generator>/<group>/.
2. Caption one reference image per group: run_idx_1_step_001.png.
3. Set up each IQA model in its own venv and download model files under models/<model>/.
4. Run scoring. src/run_iqa.py resolves the caption for each image group, calls each model wrapper, and appends scores to data/processed/scores/<metric>/scores.csv.
5. Plot scores. src/plot_scores.py groups each 100-image set by generator/group, averages the five runs at each step, and plots mean plus or minus one standard deviation.

Current expected data shape is 48 image groups, 100 images per group, for 4,800 scored images per metric.

Metric Streams and Score Direction

The repository distinguishes between evaluator wrappers and metric streams. A wrapper under src/models/<model>/ may emit one score stream, such as pickscore, or several streams from the same model invocation. ImageDoctor emits imagedoctor_alignment, imagedoctor_aesthetics, imagedoctor_plausibility, and imagedoctor_overall. ICT/HP emits ict and hp.

Each stream is saved independently as data/processed/scores/<metric>/scores.csv. configs/metrics.json is the source of truth for display labels and score direction. All current streams are higher_is_better, so larger scores indicate better quality, preference, or alignment for that metric.

For reproducibility, this repo uses one repo-standard official checkpoint or backend per metric. Alternative official checkpoints are not scored as additional streams unless they are deliberately added later as separate metric names in configs/metrics.json.

Prerequisites

• Python 3.10+
• CUDA-capable GPU environment. The setup scripts install PyTorch CUDA 12.4 wheels where applicable.
• Enough disk space for the model weights, venvs, generated scores, and plots.

Step 0: Add Images

Use this structure:

data/raw/images/
    <generator>/
        <group>/
            run_idx_1_step_001.png
            run_idx_1_step_002.png
            ...
            run_idx_5_step_020.png

Each group must include run_idx_1_step_001.png; this is the reference image captioned for that group.

Step 1: Generate Captions

./src/caption/download_and_test.sh

This creates venvs/caption/, downloads Qwen2.5-VL-7B-Instruct under models/caption/, and writes captions to data/processed/captions/captions.json.

Captioning is incremental. To caption newly added groups after setup:

venvs/caption/bin/python src/caption/generate_captions.py

Step 2: Set Up IQA Models

Run each setup script once:

./src/models/pickscore/download_and_test.sh
./src/models/hpsv2/download_and_test.sh
./src/models/imagereward/download_and_test.sh
./src/models/geneval/download_and_test.sh
./src/models/hpsv3/download_and_test.sh
./src/models/vqascore/download_and_test.sh
./src/models/mps_overall/download_and_test.sh
./src/models/imagedoctor/download_and_test.sh
./src/models/icthp/download_and_test.sh
./src/models/tifa/download_and_test.sh

Each script creates venvs/<model>/, installs dependencies, downloads model files to models/<model>/, and runs a smoke test.

The repo-standard VQAScore backend is clip-flant5-xl. The VQAScore setup script narrows the local t2v_metrics import path to the CLIP-FlanT5 backend used here, avoiding unrelated optional LLaVA/InternVL/video dependencies. HPSv3 defaults to HPSV3_BATCH_SIZE=1 because the 7B reward model can run close to the memory limit on 48 GB GPUs without flash-attn.

The MPS authors currently release only the overall-preference checkpoint, so this repo exposes it as mps_overall. ImageDoctor is a multi-output evaluator: one run of python src/run_iqa.py --models imagedoctor writes four score CSVs: imagedoctor_alignment, imagedoctor_aesthetics, imagedoctor_plausibility, and imagedoctor_overall.

ICT/HP is also multi-output: one run of python src/run_iqa.py --models icthp writes ict and hp score CSVs. ICT uses the caption as text input; HP is image-only but is emitted alongside ICT by the same evaluator.

TIFA uses two stages. The setup script downloads the official LLaMA 2 question generator and writes models/tifa/question_answers.json for the captions in data/processed/captions/captions.json, then the evaluator runs an official TIFA-supported BLIP VQA backend. The repo-standard backend is blip-large.

Step 3: Score Images

Batch mode scores every image under data/raw/images/:

python src/run_iqa.py
python src/run_iqa.py --models pickscore hpsv2 hpsv3 vqascore mps_overall imagedoctor icthp tifa

Scores are written as:

data/processed/scores/<metric>/scores.csv

CSV format:

image,score
flux_2_dev/banana_dongpo_replica/run_idx_1_step_001.png,21.384700

Runs are resumable. Rows are flushed immediately, and images already present in the CSV are skipped on restart.

Single-image checks:

python src/run_iqa.py --image data/raw/images/flux_2_dev/banana_dongpo_replica/run_idx_3_step_008.png
python src/run_iqa.py --image <path> --prompt "a plate of braised pork with banana"
python src/run_iqa.py --image <path> --output-json
python src/run_iqa.py --image <path> --no-save

To run all evaluator wrappers in parallel with at most one active model per GPU:

./src/run_eval_parallel.sh
GPU_PICKSCORE=0 GPU_HPSV2=1 GPU_IMAGEREWARD=2 GPU_GENEVAL=3 GPU_HPSV3=4 GPU_VQASCORE=5 GPU_MPS_OVERALL=6 GPU_IMAGEDOCTOR=7 GPU_ICTHP=8 GPU_TIFA=9 ./src/run_eval_parallel.sh

The parallel driver writes logs to logs/<model>_eval.log, where <model> is the wrapper name passed to src/run_iqa.py.

Step 4: Plot Scores

python src/plot_scores.py
python src/plot_scores.py --models geneval
python src/combined_failure_heatmap.py

For each metric and each <generator>/<group>, the plotter averages the five runs at every step and shades plus or minus one standard deviation. Figures are written to:

figures/<metric>/<generator>/<group>.png

With the current 48 groups and fourteen score streams, this produces 672 figure PNGs once all metrics have completed.

src/combined_failure_heatmap.py creates an additional cross-metric summary. For each generator/group/metric cell, it compares the mean score at steps 5, 10, and 20 against step 1 and counts how many selected step deltas are non-negative. Because the expected image quality direction is a drop, these non-negative deltas mark cases where a metric did not reflect that drop. Metric labels and score directions are configured in configs/metrics.json. Outputs are written to figures/combined_failure_heatmap_vertical_*.png and .pdf. The script also writes figures/combined_failure_metric_ranking_*.png and .pdf, ranking metrics by total failure count.

GenEval Limitations

The GenEval wrapper in this repo is a simplified COCO-detector diagnostic. It does not understand arbitrary scenes, architecture, styles, textures, colors, or open-vocabulary objects. It does three things:

1. Searches the VLM caption for literal COCO class names.
2. Runs a COCO-trained Cascade Mask R-CNN detector.
3. Scores the fraction of expected COCO classes that were detected.

This makes the score useful for prompts that mention unambiguous COCO objects, but unreliable for scene-heavy or architecture-heavy captions.

For example, flux_2_dev/banana_building_replica receives consistent zeros because its caption describes a skyscraper/city scene with warm orange/pink lighting. orange is a COCO class, so the wrapper interprets that word as an expected orange fruit, while building, skyscraper, cityscape, bridge, and sunset are not COCO classes. No orange fruit is detected, so the score is 0.0. This is a metric coverage limitation, not evidence that the images are empty or that the implementation failed.

When no COCO class is found in a caption, the wrapper falls back to a detector activity proxy: 1.0 if any COCO object is detected, otherwise 0.0. Treat those values as diagnostic only, not as compositional correctness.

Adding a New IQA Model

1. Create src/models/<name>/download_and_test.sh. It should create venvs/<name>/, install dependencies, store model files under models/<name>/, and run a smoke test.
2. Create src/models/<name>/evaluate.py. It must support:
   • --image <path> --prompt <text> --output-json
   • --batch-jsonl <path> for efficient full-dataset scoring
3. Add the model name to ALL_MODELS in src/run_iqa.py.
4. Add each output metric stream to configs/metrics.json with a label and score_direction.
5. If the wrapper emits more than one stream, add the mapping to MODEL_OUTPUTS in src/run_iqa.py.

Batch JSONL input records look like:

{"image": "/absolute/path/to/image.png", "prompt": "caption text"}

The evaluator should print one JSON object per scored image. Single-stream evaluators should include at least model, image, and score. Multi-stream evaluators should print a scores object whose keys match the metric names in configs/metrics.json.