---

📖 Overview

ScreenRef and Distillation-to-Reinforcement Learning (D2RL) are the core contributions of GroundGUI. ScreenRef is a high-quality GUI grounding dataset spanning desktop, web, and mobile environments. It pairs diverse screenshots with referring expressions and localization annotations, covering text recognition, visual matching, spatial understanding, and refusal.

Building on ScreenRef, we train GroundGUI-8B from Qwen3-VL-8B-Instruct with supervised fine-tuning and D2RL, improving precise localization in complex interfaces.

Key Features

🗂️ High-Quality Cross-Platform Dataset

• ScreenRef provides 586K screenshots and 633K referring expressions across desktop, web, and mobile interfaces.
• Two complementary collection pipelines combine screenshot crawling with tutorial video processing, followed by filtering and verification.
• Diverse supervision covers text recognition, visual matching, spatial understanding, and absent-target refusal.

🧠 Distillation-to-Reinforcement Learning

• OPSD provides dense guidance from a frozen teacher with privileged target information, preparing a stronger starting policy for RL.
• RLOO then optimizes click success with a binary point-in-box reward, completing the SFT → OPSD → RLOO training pipeline.

🌐 Cross-Platform Grounding

• GroundGUI-8B is evaluated on five benchmarks spanning general interfaces, professional software, and spatial reasoning.
• Consistent gains over Qwen3-VL-8B include +14.0 percentage points on ScreenSpot-Pro and +20.8 on UI-Vision.

🗂️ ScreenRef Dataset

ScreenRef (Screen Referring Grounding Dataset) pairs GUI screenshots with natural language referring expressions and bounding-box or point annotations.

Dataset Features

• 📊 Scale: 585,981 screenshots and 633,369 referring expressions.
• 🖥️ Coverage: Desktop, web, and mobile interfaces from both screenshots and tutorial videos.
• ✅ Quality: Multi-stage filtering and model-based verification of instruction–target consistency.
• 🎯 Capabilities: Text recognition, visual matching, spatial understanding, and refusal.
• 📐 Resolution: Images range from 0.1 to 8.3 megapixels.
• 🛑 Refusal Support: 22,475 absent-target examples teach the model when to abstain.

Data construction

The data is built through two complementary pipelines: screenshot crawling with metadata filtering and instruction verification, and tutorial video processing with keyframe extraction, action annotation, and visual grounding.

_{ScreenRef data construction through screenshot crawling and tutorial video crawling. Click the figure to view it at full resolution.}

Grounding capabilities

Platform distribution

Data comparison: ScreenRef, GroundCUA, and ScaleCUA

Qwen3-VL-8B is fine-tuned on each dataset under its recommended settings. UI-Vision is excluded from this comparison because of source overlap with GroundCUA.

Explore ScreenRef on Hugging Face →

🧠 D2RL: Distillation-to-Reinforcement Learning

After supervised fine-tuning on ScreenRef, D2RL strengthens the grounding policy in two stages: on-policy self-distillation (OPSD) provides dense teacher guidance, then REINFORCE Leave-One-Out (RLOO) optimizes click success.

📊 Results

GroundGUI-8B improves over Qwen3-VL-8B on all five evaluated benchmarks and outperforms the larger Qwen3-VL-32B baseline in this comparison.

🚀 Getting Started

1. Get the code and resources

git clone https://github.com/PrentisAI/GroundGUI.git
cd GroundGUI

2. Prepare the stage-specific environments

The recorded environments use Python 3.12.13. Prepare a separate environment for each stage using its requirements.lock.txt as the version reference. These are environment snapshots with CUDA-specific packages and builds; use wheel sources and CUDA tooling appropriate to your GPU system.

Keep these environments separate. SFT requires Transformers 5.x for correct sequence packing; using 4.x can silently produce incorrect training. The RL and evaluation stacks use 4.57.6. Use the lock files above as the reference for the tested versions.

3. Prepare the data paths

• SFT: download ScreenRef and ScreenRef-Annotations. Set SCREENREF_PARQUET to the ScreenRef directory and DATA_ROOT to the ScreenRef-Annotations directory; the screenshots are unpacked into $DATA_ROOT/images/ on the first run (about 330 GB).
• Post-training: set DATASET to the RL corpus, either its Parquet directory or a JSONL file. Parquet is converted once into rl/data/.
• Evaluation: organize the benchmarks according to eval/dataset_info.template.json. The evaluation wrapper creates eval/dataset_info.json from this template on its first run.

🏋️ Training

Run the commands below from the repository root and replace /path/to/... with your local paths. The paper uses 4 B300 GPUs for SFT and 8 B300 GPUs for D2RL, with 12K grounding samples for post-training.

Stage 1 · Supervised fine-tuning

CONDA_ENV="/path/to/sft-env" \
DATA_ROOT="/path/to/ScreenRef-Annotations" \
SCREENREF_PARQUET="/path/to/ScreenRef" \
GPUS=4 \
bash scripts/sft.sh out/sft "/path/to/Qwen3-VL-8B-Instruct"

The default data configuration is meta/meta_non_refusal.json in ScreenRef-Annotations. Pass a custom configuration as the third argument to scripts/sft.sh if needed. The trained model is saved under out/sft/.

Stages 2–3 · D2RL: OPSD followed by RLOO

export ENV_BIN="/path/to/rl-env/bin"
export DATASET="/path/to/rl-corpus"

# The SFT model from Stage 1.
SFT_MODEL="/path/to/sft-model"

# Stage 2: distill from the privileged teacher.
bash scripts/opsd.sh "$SFT_MODEL" groundgui-opsd

# Stage 3: optimize click success from the OPSD model.
OPSD_MODEL="/path/to/opsd-model"
bash scripts/rloo.sh "$OPSD_MODEL" groundgui-d2rl

Trained models are saved under rl/output/opsd_rloo/<run_name>/. By default, the SFT model also initializes the frozen OPSD teacher.

📏 Evaluation

The evaluation wrapper runs ScreenSpot-V2, ScreenSpot-Pro, MMBench-GUI L2, UI-Vision, and OSWorld-G sequentially with vLLM at temperature 0.

ENV_PREFIX="/path/to/eval-env" \
DATA_ROOT="/path/to/benchmarks" \
TP=2 \
bash scripts/eval.sh "/path/to/GroundGUI-8B" out/eval/groundgui-8b

Replace the checkpoint path with a downloaded GroundGUI-8B model directory or a locally trained model. For an SFT model, also set INSTRUCT_DIR="/path/to/Qwen3-VL-8B-Instruct"; RL models are evaluated as is. Results are saved to out/eval/groundgui-8b/<benchmark>/.

To evaluate the D2RL run above:

D2RL_MODEL="/path/to/d2rl-model"
ENV_PREFIX="/path/to/eval-env" \
DATA_ROOT="/path/to/benchmarks" \
bash scripts/eval.sh "$D2RL_MODEL" out/eval/groundgui-d2rl

📂 Repository Structure

GroundGUI/
├── assets/        # ScreenRef construction and D2RL method figures
├── scripts/       # Entry points: SFT, OPSD, RLOO, refusal SFT, evaluation
├── sft/           # Supervised fine-tuning and data configurations
├── rl/            # OPSD / RLOO training, based on GUI-SD and ms-swift
├── eval/          # Grounding benchmarks, prompts, and re-scoring
├── tools/         # Checkpoint conversion and serving utilities
├── LICENSE        # Apache-2.0 code license
└── NOTICE         # Third-party attribution and modified-file inventory

🙏 Acknowledgements

Our implementation builds on the following projects:

• Qwen3-VL for the foundation vision-language model.
• ScaleCUA for the SFT codebase.
• GUI-SD and ms-swift for the post-training infrastructure.
• GroundCUA for the evaluation framework.

See NOTICE for full attribution and the inventory of modified files.

⚖️ License

The code is licensed under Apache-2.0, except where noted in NOTICE. ScreenRef is distributed under its own license and is not covered by the repository's code license. Refer to the model and dataset pages for their respective terms.

📚 Citation

The paper is titled GroundGUI: GUI Grounding by High-Quality Data and Distillation-to-Reinforcement Learning. The arXiv link and BibTeX entry will be added when the preprint is available.

---

🤗 GroundGUI-8B  ·  🗂️ ScreenRef  ·  Back to top ↑