UIPro: Unleashing Superior Interaction Capability For GUI Agents

Model Details

Model Description

• Developed by: Brave Group, CASIA
• Model type: Vision-Language Model
• Language(s) (NLP): English
• License: Apache License 2.0
• Finetuned from model [optional]: Qwen2-VL-7B-Instruct

Model Sources [optional]

UIPro_1stage-7B is a GUI grounding model finetuned from Qwen2-VL-7B-Instruct.

• Repository: https://github.com/ZJULiHongxin/UIPro
• Paper [optional]: https://arxiv.org/abs/2509.17328

Uses

Direct Use

First, ensure that the necessary dependencies are installed:

pip install transformers
pip install qwen-vl-utils

Inference code example:

from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
from qwen_vl_utils import process_vision_info

# Default: Load the model on the available device(s)
model = Qwen2VLForConditionalGeneration.from_pretrained(
    "HongxinLi/UIPro_1stage", torch_dtype="auto", device_map="auto"
)
processor = AutoProcessor.from_pretrained("HongxinLi/UIPro_1stage")

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "./web_6f93090a-81f6-489e-bb35-1a2838b18c01.png",
            },

            # For ScreenSpot-v2, MOTIF, RefExp, and VisualWebBench Action Grounding
            {"type": "text", "text": "I want to {goal_info}. Please locate the target element I should interact with. (Output the center coordinates of the target)"},
            # For AutoGUI
            {"type": "text", "text": "Locate the element according to its detailed functionality description. {goal_info} (Output the center coordinates of the target)"},
            # For VisualWebBench Element Grounding
            {"type": "text", "text": "Locate the text "{goal_info}" (Output the center coordinates of the target)"},
        ],
    }
]

Evaluation

It is recommended to use AutoGUI evaluation suite based on LMMS-EVAL to evaluate it on multiple GUI Grounding benchmarks.

Results

Comparison on the GUI element grounding benchmarks. UIPro achieves impressive grounding accuracy, especially on FuncPred, RefExp, and VWB EG. AnyRes means using an image division strategy to handle images with variable resolutions.

References: [1] Qwen2VL [2] CogAgent [3] SeeClick [4] Ferret-UI [5] UGround [6] OS-ATLAS-Base

Citation

BibTeX:

@InProceedings{Li_2025_ICCV,
    author    = {Li, Hongxin and Su, Jingran and Chen, Jingfan and Ju, Zheng and Chen, Yuntao and Li, Qing and Zhang, Zhaoxiang},
    title     = {UIPro: Unleashing Superior Interaction Capability For GUI Agents},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2025},
    pages     = {1613-1623}
}

Framework versions

• PEFT 0.11.1