AnalogyGen

A pipeline for generating and evaluating images on visual analogy and other visual reasoning tasks using vision-language models and image generation models.

Overview

Given images A, A', and B (forming the analogy A:A'::B:?), this pipeline:

1. Captions all images (A, A', B, and ground-truth D)
2. Reasons about the transformation pattern (A→A')
3. Generates evaluation rubrics
4. Creates a new D image by applying the pattern to B
5. Evaluates the generated image against the rubrics

The pipeline supports three prompting strategies for image generation, described below.

---

Supported Tasks

---

Prompting Strategies

Three strategies control how image prompts are constructed and refined:

1. naive

Passes images A, A', B directly to the image generation model along with a fixed template prompt. No VLM reasoning is used to construct the prompt.

Best for: Quick baselines; testing image generation quality without prompt engineering.

2. guided

The VLM first reasons about the analogy from captions, then produces a text-only description of the target image D. The image generation model generates from text only (no input images).

Best for: Situations where images may be misleading or when clean text-based prompting is preferred.

3. feedback

Extends guided by iteratively refining the generated image. After each generation, the VLM evaluates the result against the rubric and produces feedback. A new, improved prompt is constructed and the image is regenerated. The best-scoring image across all iterations is selected.

Best for: Maximizing generation quality; exploring how iterative refinement affects the final result.

Note: guided is equivalent to feedback with --n_iteration 1.

---

Requirements

conda create -n analogygen python=3.10
conda activate analogygen
pip install -r requirements.txt

For the Bagel image generation model, use the separate Bagel environment:

conda activate bagel

---

Setup

1. Start the vLLM Server

The pipeline requires a running vLLM endpoint for captioning, reasoning, rubric generation, and evaluation. The default endpoint is http://localhost:8010/v1/chat/completions.

# Example: start Qwen2.5-VL-7B-Instruct (or similar) on port 8010
tmux new-session -d -s vllm_server bash -c "
  source \$(conda info --base)/etc/profile.d/conda.sh
  conda activate analogygen
  vllm serve Qwen/Qwen3-VL-4B-Instruct \
    --served-model-name thinker \
    --port 8010
"

Alternatively, pass --is_openai_endpoint to use the OpenAI API (requires ./utils/openai_key.txt).

2. Prepare Pre-computed Rubrics (Optional)

If using --preload, place a pre-computed rubric file at:

vasr/gpt5_rubrics/reasoning_rubrics_train-gold.json

Each entry in the JSON has the following structure:

{
  "sample_index": 0,
  "captions": {
    "A": "A lion leaping at a wildebeest...",
    "A_prime": "A lion resting on grass...",
    "B": "A polar bear swimming...",
    "D": "A polar bear resting on ice..."
  },
  "reasoning": "The transformation changes the animal from active hunting to resting...",
  "rubric": [
    {
      "criterion": "Species substitution (0-1)",
      "description": "The subject is unmistakably a polar bear (ice bear), not a lion...",
      "max_score": 1
    },
    ...
  ]
}

Download Data for Tasks

To download the task problems, rubrics, and input images directly from the Huggingface dataset, run

sh download_data.sh

To download it in the local layout needed by the code.

Alternatively, download in a parquet format from Huggingface for custom usage:

mivigenbench/mivigenbench

---

Running the Pipeline

Basic Command Structure

python3 pipeline.py \
  --task <analogy|clothing|object_attributes|human_action> \
  --prompt_strategy <naive|guided|feedback> \
  --image_gen_pipeline <bagel|qwenimageeditplus> \
  --model_name <thinker|gpt-4o-mini|...> \
  --output_dir <path/to/results> \
  [--preload] \
  [--n_iteration N] \
  [--limit N] \
  [--data_split <train|val>]

Strategy-specific Examples

Naive strategy (direct image + template prompt):

CUDA_VISIBLE_DEVICES=1,2 python3 pipeline.py \
  --task analogy \
  --prompt_strategy naive \
  --image_gen_pipeline bagel \
  --model_name thinker \
  --output_dir results/bagel_naive \
  --preload \
  --limit 10

Guided strategy (VLM-generated text prompt, no input images):

CUDA_VISIBLE_DEVICES=1,2 python3 pipeline.py \
  --task analogy \
  --prompt_strategy guided \
  --image_gen_pipeline bagel \
  --model_name thinker \
  --output_dir results/bagel_guided \
  --preload \
  --limit 10

Feedback strategy (iterative refinement, N iterations):

CUDA_VISIBLE_DEVICES=1,2 python3 pipeline.py \
  --task analogy \
  --prompt_strategy feedback \
  --image_gen_pipeline bagel \
  --model_name thinker \
  --output_dir results/bagel_feedback \
  --preload \
  --n_iteration 3 \
  --limit 10

All Arguments

---

Output Directory Structure

All outputs are organized per-index under {output_dir}/index_{N}/.

Naive strategy output

results/bagel_naive/
└── index_0/
    ├── generated.jpg        # Generated D image
    └── results.json         # Evaluation scores and metadata

Guided strategy output

results/bagel_guided/
└── index_0/
    ├── iter_0.png           # Generated D image (iteration 0)
    └── results.json         # Evaluation scores and metadata

Feedback strategy output

results/bagel_feedback/
└── index_0/
    ├── iter_0.png           # Generated image, iteration 0
    ├── iter_1.png           # Generated image, iteration 1 (after VLM feedback)
    ├── iter_2.png           # Generated image, iteration 2
    └── results.json         # Best-iteration scores + full feedback history

results.json format

{
  "index": 0,
  "A_str": "lion_hunting.jpg",
  "B_str": "polar_bear_swimming.jpg",
  "captions": { "A": "...", "B": "...", "C": "...", "D": "..." },
  "reasoning": "...",
  "rubric": [...],
  "generated_instructions": "A polar bear pounces on a wildebeest...",
  "criteria_scores": [
    { "criterion": "Species substitution (0-1)", "score": 1, "max_score": 1, "explanation": "..." },
    { "criterion": "Action continuity from C (0-2)", "score": 2, "max_score": 2, "explanation": "..." }
  ],
  "total_score": 3,
  "max_score": 5,
  "score_percentage": 60.0,
  "best_iteration": 0,
  "feedback_history": [
    { "iteration": 0, "score": 3, "feedback": "...", "instructions": "..." }
  ]
}

---

Visualizing Results

python3 visualization.py

The visualization/ folder contains pre-computed example outputs for reference.

---

Clothing Owner Task

Overview

Four people (A, B, C, D) each start with a clothing item in a specific color (provided as images). After a sequence of transfers, the model is asked to generate the final state of one person's clothing.

Dataset

Generate the dataset (problems + images):

python3 clothing_owner_task/create_data.py

To generate clothing images from scratch, you need access to:

stabilityai/stable-diffusion-3.5-medium

and place them in ./clothing_owner_task/problem_data/images/.

Run Pipeline

python3 pipeline.py \
  --task clothing \
  --preload \
  --image_gen_pipeline bagel \
  --output_dir results/clothing

---

Object Attributes Task

Dataset

python3 object_attributes_task/create_data.py

Run Pipeline

python3 pipeline.py \
  --task object_attributes \
  --preload \
  --image_gen_pipeline bagel \
  --output_dir results/object_attributes

---

Human Action Task

Dataset

python3 human_action_task/create_data.py

Run Pipeline

python3 pipeline.py \
  --task human_action \
  --preload \
  --image_gen_pipeline bagel \
  --output_dir results/human_action

---

Project Structure

AnalogyGen/
├── pipeline.py                        # Main entry point
├── pipeline_utils/
│   ├── __init__.py
│   ├── base_pipeline.py               # Base Pipeline class (VLM calls, generation, eval)
│   ├── visual_analogy_pipeline.py     # VisualAnalogyPipeline (VASR)
│   ├── clothing_owner_pipeline.py     # ClothingOwnerPipeline
│   ├── object_attributes_pipeline.py  # ObjectAttributesPipeline
│   └── human_action_pipeline.py       # HumanActionPipeline
├── clothing_owner_task/
│   └── create_data.py                 # Problem + image generation for clothing task
├── object_attributes_task/
│   └── create_data.py
├── human_action_task/
│   └── create_data.py
├── Bagel/                             # Bagel model integration
│   ├── inferencer.py
│   └── bagel_init.py
├── utils/
│   └── utils.py                       # OpenAI key loading, multimodal API helpers
├── vasr/
│   ├── train_gold.csv
│   └── dev_gold.csv
├── results/                           # All pipeline outputs (auto-created)
├── visualization.py
└── requirements.txt

---

GPU Requirements

• Bagel (BAGEL-7B-MoT): Requires ~2 GPUs (uses infer_auto_device_map). Set CUDA_VISIBLE_DEVICES to select specific GPUs.
• vLLM server (Qwen VL): Requires ~1 GPU. Run on a separate GPU from the image generation model.

Example: if GPUs 0 and 5 are in use by the vLLM server:

CUDA_VISIBLE_DEVICES=1,2 python3 pipeline.py ...