📚 Table of Contents

• Overview
• Architecture
• Installation
• Download Data
• Project Structure
• Supported Benchmarks & Models
• Reproduction Tips
• Quick Start
• Script Parameters
• Adding a New Model
• Data Format
• Reproduction Results
• Roadmap
• License

📖 Overview

ClawGUI-Eval is the evaluation module of ClawGUI. GUI grounding evaluation is harder to reproduce than it looks: prompt order, coordinate systems, temperature, and image resolution all interact to shift numbers by several points. ClawGUI-Eval pins all of these choices per model and adopts a three-stage pipeline — Infer → Judge → Metric — to evaluate how accurately a model can locate UI elements based on natural language instructions. The result is a 95.8% reproduction rate against official baselines, making cross-paper comparisons meaningful.

Key Features:

• Dual backend support — Local GPU via transformers or remote API via OpenAI-compatible endpoints
• 6 benchmarks — ScreenSpot-Pro, ScreenSpot-V2, UIVision, MMBench-GUI, OSWorld-G, AndroidControl
• 12+ models — Qwen3-VL, Qwen2.5-VL, UI-TARS, MAI-UI, GUI-G2, UI-Venus, Gemini, Seed 1.8, Kimi K2.5, and more
• Multi-GPU & multi-thread — NUM_GPUS processes launched via Python multiprocessing, each pinned to one GPU via CUDA_VISIBLE_DEVICES. Shard files are automatically split and merged; interrupted runs resume from the last completed shard.
• Easily extensible — Add new models by inheriting a simple base class; shared architectures (e.g. UI-TARS extends Qwen2.5-VL) reuse parent model loading and only override prompt building and output parsing
• Faithful reproduction — Comprehensive reproduction results with detailed official vs. reproduced comparisons (see details)
• Frontier model evaluation — Successfully reproduced Gemini 3.0 Pro and Seed 1.8 official results on ScreenSpot-Pro using a Zoom paradigm (2-stage crop-then-ground: Gemini uses 25% crop tiles, Seed uses 50% crop tiles), and added Gemini 3.1 Pro evaluation
• ClawGUI-Agent integration — Pair with ClawGUI-Agent to launch the full evaluation pipeline with a single natural language command (env check → inference → judging → metrics). See ClawGUI-Agent README for setup details

🏗️ Architecture

🔧 Installation

Option A: Docker (recommended for reproducibility)

Docker eliminates dependency conflicts and makes it easy to share exact evaluation environments.

Prerequisites: NVIDIA Container Toolkit

cd ClawGUI/clawgui-eval

# Build the image (first build is slow due to flash-attn compilation)
docker build -t clawgui-eval .

Then create a .env file to point at your data and model directories:

# .env
DATA_DIR=/data/clawgui-eval/data
IMAGE_DIR=/data/clawgui-eval/image
OUTPUT_DIR=/data/clawgui-eval/output
MODEL_DIR=/data/models           # HuggingFace model cache or local weights

Run any inference script inside the container:

# Inference
docker compose run clawgui-eval \
    bash scripts/infer/transformers/qwen3vl_run_transformers.sh

# Judge
docker compose run clawgui-eval \
    bash scripts/judge/screenspot-pro_run_judge.sh

# Metric
docker compose run clawgui-eval \
    bash scripts/metric/run_metric_screenspot_pro.sh

Note: Edit MODEL_PATH inside the shell scripts to point to /models/<your-model-dir> (the container-side path of MODEL_DIR).

Option B: Conda + pip

cd ClawGUI/clawgui-eval
conda create -n opengui python=3.12 -y
conda activate opengui
pip install -r requirements.txt
# Recommended: FlashAttention-2 for better precision (falls back to SDPA if not installed)
pip install flash-attn==2.8.1 --no-build-isolation
# Optional: vLLM support
pip install vllm==0.11.0

💡 Tip: If building flash-attn from source is too slow, you can download a prebuilt wheel from the flash-attn releases page and install it directly.

📥 Download Data

Benchmark images and data files are hosted on Hugging Face and ModelScope. Download them before running evaluations.

From Hugging Face:

pip install -U huggingface_hub

# If you have trouble accessing HF, use the mirror:
# export HF_ENDPOINT=https://hf-mirror.com

huggingface-cli download johnzqlu/clawgui-eval --repo-type dataset --local-dir .

From ModelScope:

pip install -U modelscope

modelscope download --dataset Matrix0602/clawgui-eval --local_dir .

Then extract the archives under the clawgui-eval/ directory:

cd clawgui-eval
unzip image.zip
unzip data.zip
unzip output.zip

⚠️ Important: All zip files (image.zip, data.zip, output.zip) must be extracted under the clawgui-eval/ directory to ensure the relative paths resolve correctly.

📁 Project Structure

clawgui-eval/
├── 📄 main.py                          # Inference entry point
├── 📂 inference/                        # Model inferencers
│   ├── base_inferencer.py               # Abstract base class
│   ├── qwen3vl_inferencer.py            # Qwen3-VL
│   ├── qwen25vl_inferencer.py           # Qwen2.5-VL
│   ├── maiui_inferencer.py              # MAI-UI
│   ├── stepgui_inferencer.py            # StepGUI
│   ├── guiowl15_inferencer.py           # GUI-Owl 1.5
│   ├── guig2_inferencer.py              # GUI-G2
│   ├── uitars_inferencer.py             # UI-TARS (extends Qwen2.5-VL)
│   ├── uivenus15_inferencer.py          # UI-Venus 1.5 (extends Qwen3-VL)
│   ├── uivenus_inferencer.py            # UI-Venus (extends GUI-G2)
│   ├── gemini_inferencer.py             # Gemini (API, optional Zoom)
│   ├── seed_inferencer.py               # Seed 1.8 (API, optional Zoom)
│   └── kimi_inferencer.py               # Kimi K2.5 (API, optional Zoom)
├── 📂 judge/                            # Judgment module
│   ├── base_judge.py                    # Abstract base class
│   ├── grounding_judge.py               # Point-in-box judge (most benchmarks)
│   ├── osworld_g_judge.py               # OSWorld-G judge (bbox/polygon/refusal)
│   └── androidcontrol_judge.py          # AndroidControl judge (multi-action)
├── 📂 metric/                           # Metric calculation
│   ├── base_metric.py
│   ├── screenspotpro_metric.py
│   ├── screenspotv2_metric.py
│   ├── mmbenchgui_metric.py
│   ├── osworldg_metric.py
│   ├── uivision_metric.py
│   └── androidcontrol_metric.py
├── 📂 data/                             # Benchmark data & prompt injection
│   ├── convert_any_models.py            # Prompt injection script
│   └── *.json                           # Base & model-specific data files
├── 📂 scripts/
│   ├── infer/
│   │   ├── transformers/                # Local GPU inference scripts
│   │   ├── api/                         # API inference scripts
│   │   └── vllm_depoly/                 # vLLM server deployment
│   ├── judge/                           # Judge scripts (one per benchmark)
│   └── metric/                          # Metric scripts
├── 📂 image/                            # Benchmark images (downloaded)
└── 📂 output/                           # Inference & judge output

📊 Supported Benchmarks & Models

Benchmarks

Open-Source Models

Frontier / Closed-Source Models

We have also reproduced GUI grounding results for frontier models on ScreenSpot-Pro using the Zoom paradigm (crop-then-ground). For details on the Zoom pipeline, see the MAI-UI blog: A Practical Guide to GUI Grounding for Frontier Models.

📐 Coordinate Systems:

• Absolute — Output is in raw pixel coordinates of the original (or smart_resize'd) image
• [0, 1000] — Output is normalized to a 1000×1000 coordinate space, then mapped back to the original image
• [0, 1] — Output is a ratio in [0, 1] relative to the original image dimensions
• [0, 999] — Similar to [0, 1000] but with a 999 divisor

💡 Reproduction Tips

Click to expand 9 key lessons for faithful reproduction

1. 🔀 Message Format (tv_or_vt)

Different models are sensitive to the order of image and text in the input message. Our framework provides the TV_OR_VT parameter to control this:

• vt = image first, then text (default for most models)
• tv = text first, then image (required by MAI-UI)

⚠️ Always align with the model's official implementation. Using the wrong order can cause significant accuracy drops.

2. 🌡️ Temperature

For grounding tasks, always set TEMPERATURE=0.0 (greedy decoding). Non-zero temperatures introduce randomness that hurts coordinate precision.

3. 📝 Prompt Alignment

Most GUI grounding models are highly sensitive to prompt format. Ensure strict alignment with the official prompt template. Even minor wording differences can affect results. The data/convert_any_models.py script handles this for all supported models.

4. 🖼️ Image Resolution (MIN_PIXELS / MAX_PIXELS)

Models are sensitive to image resolution bounds. Always match the official values:

• Different models use different default resolutions
• Changing these values can significantly shift accuracy

5. 📊 Sampling Parameters (TOP_P / TOP_K)

These parameters have minimal impact on grounding results — typically ±0.1% fluctuation. Not a major concern for reproduction.

6. 📐 Coordinate Systems

Understanding each model's output coordinate format is critical for correct parsing:

• Qwen2.5-VL family (qwen25vl, uitars) → outputs absolute pixel coordinates
• Qwen3-VL family (qwen3vl, guiowl15, uivenus15, maiui) → outputs [0, 1000] normalized coordinates
• GUI-G2 family (guig2, uivenus) → outputs [0, 1000] normalized bounding boxes
• StepGUI → outputs [0, 999] normalized coordinates

🔑 Mismatched coordinate parsing is the #1 cause of zero-accuracy results.

7. 💬 System Prompt

The Qwen-VL series models are notably sensitive to system prompts:

• qwen3vl, qwen25vl, guiowl15, maiui → require a specific tool-call system prompt
• uitars, guig2, uivenus, uivenus15, stepgui → inject prompts into the user question instead

Set SYSTEM_PROMPT="call_user" for models that require it; the prompt content is pre-injected into the data files.

8. 🪄 Default System Prompt Boost

Some models are sensitive to even the most generic system prompt. Simply adding "You are a helpful assistant." as a default system prompt can improve accuracy by ~1% on certain models. If a model's official code includes any system prompt, always replicate it — even if it seems trivial.

9. 📱 AndroidControl: Scroll Direction Convention

AndroidControl defines scroll direction from the screen's perspective — scroll_direction=down means the screen scrolls down (content moves up). However, some models (trained on human gesture data) output swipe directions from the finger's perspective — a finger swipe up causes the screen to scroll down. Always verify which convention a model follows and normalize accordingly.

Additionally, since OS-Atlas, most subsequent works evaluate on the 7,708-sample subset of AndroidControl. For click accuracy, the ground-truth target is parsed from the original AndroidControl accessibility tree as a bounding box (point-in-box judgment) — this differs from GUI-Odyssey, which computes Euclidean distance between the predicted point and the GT point, using a threshold of 0.14 (normalized by screen size).

🚀 Quick Start

Step 1: Inference (Infer)

Two backends are supported:

🖥️ Transformers Backend (Local GPU)

bash scripts/infer/transformers/qwen3vl_run_transformers.sh

🌐 API Backend (Remote Service)

# 1. Deploy vLLM service first
bash scripts/infer/vllm_depoly/vllm_serve.sh

# 2. Run inference
bash scripts/infer/api/qwen3vl_run_api.sh

# Kimi K2.5 API
bash scripts/infer/api/kimi_run_api.sh

Output is saved to:

output/<experiment_name>/<benchmark>/predictions.jsonl

Step 2: Judgment (Judge)

# GUI Grounding benchmarks
bash scripts/judge/screenspot-pro_run_judge.sh

# AndroidControl benchmark
bash scripts/judge/androidcontrol_run_judge.sh

Each record gets a correct field (true/false). Output:

output/<experiment_name>/<benchmark>/predictions_judge.jsonl

Step 3: Metric Calculation (Metric)

# GUI Grounding benchmarks
bash scripts/metric/run_metric_screenspot_pro.sh

# AndroidControl benchmark
bash scripts/metric/run_metric_androidcontrol.sh

Reports accuracy broken down by platform, UI type, etc.

⚙️ Script Parameters

🖥️ Transformers Backend

🌐 API Backend

In addition to the parameters above:

🔍 Judge Parameters

🧩 Adding a New Model

1. Create inference/<name>_inferencer.py, extending BaseInferencer (or an existing inferencer if architectures match).

2. Implement four methods: _init_model(), _build_prompt(), _generate(), _post_process().

3. Register in inference/__init__.py:

   INFERENCER_REGISTRY = {
       ...
       "your_model": YourModelInferencer,
   }
   

4. Add prompt injection logic in data/convert_any_models.py, then generate data files.

5. Add parsing logic in judge/grounding_judge.py (and osworld_g_judge.py if needed).

6. Create launch scripts under scripts/infer/transformers/ and scripts/infer/api/.

📋 Data Format

Each input sample must contain the following fields:

📈 Reproduction Results

A key goal of ClawGUI-Eval is faithful reproduction of officially reported numbers. Below we compare our reproduced results against official baselines across all supported benchmarks.

📂 All inference results are publicly available on our dataset page: 🤗 HuggingFace: johnzqlu/clawgui-eval | 🤖 ModelScope: Matrix0602/clawgui-eval

Criterion: A result is considered successfully reproduced (✅) if the reproduced number meets or exceeds the official number, or the absolute difference is ≤ 2%. - means no official baseline is available.

GUI Grounding Benchmarks

Open-Source GUI Grounding Reproduction Rate: 44 / 46 cells with official baselines = 95.7%

Frontier Model ScreenSpot-Pro Reproduction Rate: 2 / 2 = 100.0%

Overall Reproduction Rate: 46 / 48 = 95.8%

AndroidControl (HIGH Split — Step Success Rate)

AndroidControl evaluates offline navigation with multi-action prediction (click, type, scroll, etc.). We currently support Qwen3-VL and Qwen2.5-VL on this benchmark.

Note: Official AndroidControl baselines for these models are not yet publicly available. We will update the comparison once official numbers are released.

🗺️ Roadmap

• Support ScreenSpot-Pro, ScreenSpot-V2, UIVision, MMBench-GUI, OSWorld-G benchmarks
• Support AndroidControl benchmark (Qwen3-VL, Qwen2.5-VL)
• Transformers & API dual backend inference
• Multi-GPU parallel inference with automatic resume
• Frontier model reproduction (Claude 4.5 Sonnet, Gemini 3.1/3.0 Pro, Seed 1.8) with Zoom paradigm
• Integrate vLLM offline inference (non-server mode)
• Add more GUI-specific models
• GUI offline navigation evaluation (e.g. GUI-Odyssey)

📄 License

This project is licensed under the Apache License 2.0.