UniWorld: High-Resolution Semantic Encoders for
Unified Visual Understanding and Generation

📣 News

• [2025.06.03] 🤗 We release UniWorld, a unified framework for understanding, generation, and editing. All data, models, training code, and evaluation code are open-sourced. Checking our report for more details. Welcome to watch 👀 this repository for the latest updates.

😍 Gallery

UniWorld shows excellent performance in 20+ tasks.

UniWorld, trained on only 2.7M samples, consistently outperforms BAGEL (trained on 2665M samples) on the ImgEdit-Bench for image manipulation. It also surpasses the specialized image editing model Step1X-Edit across multiple dimensions, including add, adjust, and extract on ImgEdit-Bench.

Click to play

😮 Highlights

1. All Resources Fully Open-Sourced

• We fully open-source the models, data, training and evaluation code to facilitate rapid community exploration of unified architectures.

• We curate 10+ CV downstream tasks, including canny, depth, sketch, MLSD, segmentation and so on.

• We annotate 286K long-caption samples using Qwen2-VL-72B. We use GPT-4o to filter ImgEdit, result in 724K high-quality editing samples (all shortedge ≥ 1024 pix). Additionally, we organize and filter existing open-sourced datasets. The details can be found here.

2. Contrastive Semantic Encoders as Reference Control Signals

• Unlike prior approaches that use VAE-encoded reference images for low-level control, we advocate using contrastive visual encoders as control signals for reference images.

• For such encoders, we observe that as resolution increases, global features approach saturation and model capacity shifts toward preserving fine details, which is crucial for maintaining fidelity in non-edited regions.

3. Image Priors via VLM Encoding Without Learnable Tokens

• We find that multimodal features encoded by VLMs can interpret instructions while retaining image priors. Due to causal attention, the format <instruction><image> is particularly important.

🤗 Demo

Gradio Web UI

Highly recommend trying out our web demo by the following command.

MODEL_PATH="path/to/model"
FLUX_PATH="path/to/flux"
SIGLIP_PATH="path/to/siglip"
CUDA_VISIBLE_DEVICES=0 python -m univa.serve.gradio_web_server \
    --model_path ${MODEL_PATH} \
    --flux_path ${FLUX_PATH} \
    --siglip_path ${SIGLIP_PATH}

CLI Inference

MODEL_PATH="path/to/model"
FLUX_PATH="path/to/flux"
SIGLIP_PATH="path/to/siglip"
CUDA_VISIBLE_DEVICES=1 python -m univa.serve.cli \
    --model_path ${MODEL_PATH} \
    --flux_path ${FLUX_PATH} \
    --siglip_path ${SIGLIP_PATH}

ComfyUI

Coming soon...

⚙️ Requirements and Installation

1. Clone this repository and navigate to UniWorld folder

git clone https://github.com/PKU-YuanGroup/UniWorld
cd UniWorld

2. Install required packages

conda create -n univa python=3.10 -y
conda activate univa
pip install -r requirements.txt

🗝️ Training

Data preparation

Download the data from LanguageBind/UniWorld-V1. The dataset consists of two parts: source images and annotation JSON files.

Prepare a data.txt file in the following format:

1. The first column is the root path to the image.

2. The second column is the corresponding annotation JSON file.

3. The third column indicates whether to enable the region-weighting strategy. We recommend setting it to True for edited data and False for others.

data/BLIP3o-60k,json/blip3o_t2i_58859.json,false
data/coco2017_caption_canny-236k,coco2017_canny_236574.json,false
data/imgedit,json/imgedit/laion_add_part0_edit.json,true

We provide a simple online verification tool to check whether your paths are set in data.txt correctly.

python univa/serve/check_data.py

Data details

Training

Prepare pretrained weights

Download black-forest-labs/FLUX.1-dev to $FLUX_PATH. Download Qwen/Qwen2.5-VL-7B-Instruct to $QWENVL_PATH. We also support other sizes of Qwen2.5-VL.

SAVE_PATH="path/to/save/UniWorld-Qwen2.5-VL-7B-Instruct-FLUX.1-dev-fp32"
python scripts/make_univa_qwen2p5vl_weight.py \
    --origin_flux_ckpt_path $FLUX_PATH \
    --origin_qwenvl_ckpt_path $QWENVL_PATH \
    --save_path ${SAVE_PATH}

# stage1
bash scripts/denoiser/flux_qwen2p5vl_7b_vlm_stage1_512.sh

Download flux-redux-siglipv2-512.bin and set its path to pretrained_siglip_mlp_path in stage2.yaml. The weight is sourced from ostris/Flex.1-alpha-Redux, we just re-organize the weight.

# stage2
bash scripts/denoiser/flux_qwen2p5vl_7b_vlm_stage2_512.sh

⚡️ Evaluation

Text-to-Image Generation

cd univa/eval/geneval
# follow the instruction in univa/eval/geneval/README.md

cd univa/eval/wise
# follow the instruction in univa/eval/wise/README.md

cd univa/eval/genai
# follow the instruction in univa/eval/genai/README.md

cd univa/eval/dpgbench
# follow the instruction in univa/eval/dpgbench/README.md

Image Editing

cd univa/eval/imgedit
# follow the instruction in univa/eval/imgedit/README.md

cd univa/eval/gdit
# follow the instruction in univa/eval/gdit/README.md

📊 Benchmarks

💡 How to Contribute

We greatly appreciate your contributions to the UniWorld open-source community and helping us make it even better than it is now!

For more details, please refer to the Contribution Guidelines.

👍 Acknowledgement and Related Work

• ImgEdit: ImgEdit is a large-scale, high-quality image-editing dataset comprising 1.2 million carefully curated edit pairs.
• Open-Sora Plan: An open‑source text-to-image/video foundation model, which provides a lot of caption data.
• SEED-Data-Edit: A hybrid dataset for instruction-guided image editing.
• Qwen2.5-VL: The new flagship vision-language model of Qwen.
• FLUX.1-Redux-dev: Given an input image, FLUX.1 Redux can reproduce the image with slight variation, allowing to refine a given image.
• SigLIP 2: New multilingual vision-language encoders.
• Step1X-Edit: A state-of-the-art image editing model.
• BLIP3-o: A unified multimodal model that combines the reasoning and instruction following strength of autoregressive models with the generative power of diffusion models.
• BAGEL: An open‑source multimodal foundation model with 7B active parameters (14B total) trained on large‑scale interleaved multimodal data.

🔒 License

• See LICENSE for details. The FLUX weights fall under the FLUX.1 [dev] Non-Commercial License.

✨ Star History

✏️ Citing

@misc{lin2025uniworldhighresolutionsemanticencoders,
      title={UniWorld: High-Resolution Semantic Encoders for Unified Visual Understanding and Generation}, 
      author={Bin Lin and Zongjian Li and Xinhua Cheng and Yuwei Niu and Yang Ye and Xianyi He and Shenghai Yuan and Wangbo Yu and Shaodong Wang and Yunyang Ge and Yatian Pang and Li Yuan},
      year={2025},
      eprint={2506.03147},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2506.03147}, 
}

@article{niu2025wise,
  title={Wise: A world knowledge-informed semantic evaluation for text-to-image generation},
  author={Niu, Yuwei and Ning, Munan and Zheng, Mengren and Lin, Bin and Jin, Peng and Liao, Jiaqi and Ning, Kunpeng and Zhu, Bin and Yuan, Li},
  journal={arXiv preprint arXiv:2503.07265},
  year={2025}
}

@article{lin2024open,
  title={Open-Sora Plan: Open-Source Large Video Generation Model},
  author={Lin, Bin and Ge, Yunyang and Cheng, Xinhua and Li, Zongjian and Zhu, Bin and Wang, Shaodong and He, Xianyi and Ye, Yang and Yuan, Shenghai and Chen, Liuhan and others},
  journal={arXiv preprint arXiv:2412.00131},
  year={2024}
}

🤝 Community contributors

This model is presented in the paper: UniWorld: High-Resolution Semantic Encoders for Unified Visual Understanding and Generation