Ask, Solve, Generate: Self-Evolving Unified Multimodal Understanding and Generation via Self-Consistency Rewards

Ritesh Thawkar¹ · Shravan Venkatraman¹ · Omkar Thawakar¹ · Abdelrahman M Shaker¹ · Fahad Shahbaz Khan^1,3 · Hisham Cholakkal¹ · Salman Khan¹ · Rao Muhammad Anwer^1,2

¹ Mohamed bin Zayed University of Artificial Intelligence    ² Aalto University    ³ Linköping University

---

Ask, Solve, Generate studies whether unified multimodal models can improve visual understanding and image generation from unlabeled images alone. The training loop turns each image into self-generated questions, selects reliable reference answers through self-consistency, and uses the same interaction signal to update both understanding and generation components.

---

🔎 Overview

The self-evolving loop uses three internal roles:

The implementation uses internal consistency signals rather than external answer labels. For understanding, Solver Token Entropy (STE) provides a token-level uncertainty signal for selecting useful updates. For generation, question-answer fidelity and cycle-consistent captioning connect generated images back to the understanding loop.

The main implementation is built around BLIP3o. BAGEL and VARGPT-v1.1 integrations are included to evaluate the same self-evolving recipe on different unified model families. The released code covers self-questioning, reference-answer selection, generation supervision, cycle consistency, and backend-specific training/evaluation adapters.

---

🏗️ Architecture

The framework alternates understanding and generation steps. In the understanding step, the Proposer creates image-grounded questions, the Solver answers perturbed prompts, and self-consistency plus token-level entropy provide training signals. In the generation step, question-answer-derived generation specifications guide image synthesis, while the Solver evaluates QA fidelity and cycle consistency.

---

📊 Main Results

Results are reported as base checkpoint -> self-evolved checkpoint under matched evaluation settings. MME-P and MME-C use the raw MME perception and cognition scores; other understanding metrics are percentages.

🧠 Visual Understanding

🎨 Image Generation

GenEval scores are percentages.

🖼️ Qualitative Results

---

🗂️ Repository Layout

Generated outputs are ignored by git. Keep datasets, model weights, checkpoints, logs, and caches outside the tracked source tree or under ignored paths such as data/, models/, outputs/, and logs/.

---

⚙️ Setup

Use separate environments for the three backends. The dependency stacks differ, and mixing them in one environment can create version conflicts.

Copy .env.example to .env for local secrets and cache paths. Common keys are HF_TOKEN, OPENAI_API_KEY, WANDB_API_KEY, and HF_HOME; do not commit filled environment files.

🔧 BLIP3o

Install a PyTorch build that matches your CUDA or ROCm stack first, then install the BLIP3o dependencies and local packages:

conda create -n uug-blip3o python=3.10 -y
conda activate uug-blip3o
# Install torch/torchvision/torchaudio for your CUDA or ROCm stack first.
pip install -r BLIP3o/requirements.safe.txt
pip install -e BLIP3o
pip install -e BLIP3o/eval/lmms-eval

BLIP3o/requirements.safe.txt intentionally avoids reinstalling PyTorch, xformers, flash-attn, deepspeed, and bitsandbytes. If you want the full upstream CUDA stack instead, use:

pip install -r BLIP3o/requirements.txt
pip install -e BLIP3o
pip install -e BLIP3o/eval/lmms-eval

🔧 BAGEL

conda create -n uug-bagel python=3.10 -y
conda activate uug-bagel
pip install -r Bagel/requirements.txt

Generation benchmark scoring may also require benchmark-specific detector assets and setup from Bagel/EVAL.md.

🔧 VARGPT-v1.1

conda create -n uug-vargpt python=3.10 -y
conda activate uug-vargpt
cd vargpt_1_1/VARGPT-family-training
pip install -r requirements.txt
pip install -e .
cd ../..

For VARGPT understanding evaluation, install the local evaluation package when needed:

pip install -e vargpt_1_1/understand_eval

The standalone upstream inference examples under vargpt_1_1/inference_v1_1 use vargpt_1_1/requirements.txt, which pins a PyTorch/flash-attn stack. Use that file in a separate environment if it conflicts with the training setup.

---

🚀 Quick Start

After installing the BLIP3o environment, run a small launcher smoke test on any local folder of images:

cp .env.example .env

TOTAL_STEPS=20 \
ALLOW_SMALL_DATA=1 \
DATA_DIR=/path/to/small/image/folder \
OUTPUT_DIR=$PWD/outputs/smoke/blip3o \
bash scripts/E1_main_joint.sh

---

🤗 Model Zoo

Released checkpoints are available in the Ask, Solve, Generate Hugging Face collection. The training and evaluation scripts expect local checkpoint paths.

---

🔮 Inference

Use these entry points for qualitative checks:

python BLIP3o/gradio/app.py /path/to/BLIP3o-Model-8B
python BLIP3o/inference.py /path/to/BLIP3o-Model-8B
python vargpt_1_1/inference_v1_1/understanding_vargpt_v1_1.py
python vargpt_1_1/inference_v1_1/generation_vargpt_v1_1.py

BAGEL inference utilities are exposed through Bagel/inferencer.py; use the benchmark-generation wrappers for batch sampling and scoring.

---

🏋️ Training

BLIP3o training launchers are under scripts/. BAGEL and VARGPT-v1.1 use their backend-native launchers, shown below.

Common BLIP3o launchers:

▶️ Direct BLIP3o Launcher

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
NPROC_PER_NODE=8 \
DATA_DIR=/path/to/unlabeled/images \
OUTPUT_DIR=$PWD/outputs/blip3o/E1_main_joint \
bash scripts/E1_main_joint.sh

▶️ BAGEL Launcher

MODEL_PATH=/path/to/BAGEL-7B-MoT \
DATA_DIR=/path/to/unlabeled/images \
OUTPUT_DIR=$PWD/outputs/bagel/B1_unified_training \
NPROC_PER_NODE=8 \
bash Bagel/scripts/B1_unified_training.sh

MODEL_PATH must point to the local BAGEL model checkout or checkpoint directory.

▶️ VARGPT-v1.1 Launcher

cd vargpt_1_1/VARGPT-family-training

IMAGE_FOLDER=/path/to/unlabeled/images \
OUTPUT_DIR=/path/to/output/vargpt/joint \
bash examples/train_self_evolving/run_self_evolving.sh joint 8

Available VARGPT modes are joint, u_only, and gen_only.

For BLIP3o resume, set RESUME_FROM=/path/to/output_dir or RESUME_FROM=/path/to/output_dir/step_010000; complete checkpoints contain SAVE_OK. Monitor runs through status.json, iter_log.jsonl, and logs/training_watch.log inside OUTPUT_DIR.

---

📏 Evaluation

Use completed checkpoints as input. The exact benchmark data setup follows the underlying benchmark and upstream evaluation tool requirements.

🧠 BLIP3o Understanding

CHECKPOINT_DIR=/path/to/step_010000 \
NUM_GPUS=8 \
bash BLIP3o/eval/understanding_eval_our.sh

Override benchmark tasks when needed:

CHECKPOINT_DIR=/path/to/step_010000 \
TASKS="realworldqa,textvqa" \
bash BLIP3o/eval/understanding_eval_our.sh

🎨 BLIP3o Generation

CHECKPOINT_DIR=/path/to/step_010000 \
bash BLIP3o/eval/geneval/generation_our.sh

CHECKPOINT_DIR=/path/to/step_010000 \
bash BLIP3o/eval/dpg_bench/generate_dpg_our.sh

CHECKPOINT_DIR=/path/to/step_010000 \
bash BLIP3o/eval/wise/generate_wise_our.sh

WISE scoring requires OPENAI_API_KEY.

📋 BAGEL Evaluation

Understanding benchmarks are launched through the BAGEL VLM evaluation dispatcher:

bash Bagel/eval/vlm/evaluate.sh mmbench-dev-en
bash Bagel/eval/vlm/evaluate.sh mmmu-val
bash Bagel/eval/vlm/evaluate.sh mme

For generation benchmarks, follow the dependency and detector setup described in Bagel/EVAL.md.

📋 VARGPT-v1.1 Evaluation

CHECKPOINT_DIR=/path/to/se_checkpoint_10000 \
NUM_GPUS=8 \
bash vargpt_1_1/understand_eval/understanding_eval_our.sh

Generation evaluation wrappers are under:

vargpt_1_1/VARGPT-family-training/run_scripts/

For trained checkpoints, start with:

cd vargpt_1_1/VARGPT-family-training

CHECKPOINT_DIR=/path/to/se_checkpoint_10000 \
bash run_scripts/run_eval_vargpt_generation_our.sh

---

📚 Citation

If this codebase is useful for your research, please cite:

@misc{thawkar2026asksolvegenerate,
  title = {Ask, Solve, Generate: Self-Evolving Unified Multimodal Understanding and Generation via Self-Consistency Rewards},
  author = {Thawkar, Ritesh and Venkatraman, Shravan and Thawakar, Omkar and Shaker, Abdelrahman M and Khan, Fahad Shahbaz and Cholakkal, Hisham and Khan, Salman and Anwer, Rao Muhammad},
  year = {2026},
  note = {Manuscript}
}

---

🙏 Acknowledgements

This repository builds on BLIP3o, BAGEL, VARGPT-v1.1, lmms-eval, GenEval, DPG-Bench, and WISE evaluation tooling. We thank the authors and maintainers of these projects for releasing their code and models.

---

📄 License

This repository is released under the Apache License 2.0. See LICENSE.

Third-party code and model integrations under Bagel/, BLIP3o/, and vargpt_1_1/ may retain their upstream licenses and usage terms. Check the corresponding upstream projects and model cards before redistributing model weights or derived checkpoints.