3
stars
13
commits
1
linked in READMEs
Aug 18, 2026
updated
README
GUIrilla-Task
Ground-truth Click & Type actions for macOS screenshots
Dataset Summary
GUIrilla-Task pairs real macOS screenshots with free-form natural-language instructions and precise GUI actions. Every sample asks an agent either to:
- Click a specific on-screen element, or
- Type a given text into an input field.
Targets are labelled with bounding-box geometry, enabling exact evaluation of visual-language grounding models. Data were gathered automatically by the GUIrilla crawler, which explores applications via the macOS Accessibility API while GPT-4 agents generate diverse, user-style tasks.
Dataset Structure
Data Fields
| Field | Type | Description |
|---|---|---|
screen_id | int | Unique screenshot index. |
app_name | string | Bundle name (e.g. com.apple.Safari). |
task | string | Final, cleaned instruction. |
raw_task | string | Raw task draft. |
action | string | Either "left click" or "type <text>". |
image | image | Full-resolution PNG. |
image_cropped | image | Tight crop of the app window. |
accessibility | string | Accessibility JSON tree for the app window. |
scaling_factor | int | UI scaling for that capture. |
element_data | string | Bounding-box / polygon / role / label in JSON. |
original_task | bool | True if task was performed in the macOS environment. |
task_category | string | High-level task category (Search & Information, Files etc.). |
element_category | string | High-level AX role (button, textfield, etc.). |
If the task is click,
element_datacontains a bounding box For type, it stores the insertion location and expected text.
Splits
| Split | Examples | Uncompressed Size | Notes |
|---|---|---|---|
train | 25 606 | 47.8 GB | Training & dev experiments |
test | 1 565 | 2.32 GB | Held-out evaluation split |
The two splits are application-level disjoint: a specific app build never appears in both.
Intended Uses
| ✔️ Recommended | ❌ Not Recommended |
|---|---|
| Training & evaluating macOS GUI agents (vision–language models, RPA, RL). | Commercial redistribution (license is NC). |
| Academic research on accessibility, automated testing, multimodal grounding. | Deployment in safety-critical automation without human oversight. |
Loading Example
from datasets import load_dataset
from PIL import Image
ds_train = load_dataset("macpaw-research/GUIrilla-Task", split="train")
row = ds_train[0]
print(row["task"], "→", row["action"])
Image.open(row["image"].filename).show()
Image.open(row["image_cropped"].filename).show()
License
CC-BY-NC-4.0 (see LICENSE).
This dataset contains screenshots and accessibility metadata captured from third-party macOS applications; the non-commercial licence reflects that provenance.
In short (not legal advice):
- ✅ Non-commercial research — training and evaluating models, publishing papers and benchmark results, redistributing for non-commercial purposes (all with attribution).
- ❌ Commercial use — e.g. training or benchmarking models for commercial products, or redistributing the data as part of a commercial offering — is not covered by this licence.
Interested in using this dataset commercially? Contact MacPaw Research to discuss options.
Citation
@article{garkot2025guirilla,
title={GUIrilla: A Scalable Framework for Automated Desktop UI Exploration},
author={Garkot, Sofiya and Shamrai, Maksym and Synytsia, Ivan and Hirna, Mariya},
journal={arXiv preprint arXiv:2510.16051},
year={2025},
url={https://arxiv.org/abs/2510.16051}
}
Contributors
Similar repos
3
stars
13
commits
1
linked in READMEs
Aug 18, 2026
updated
README
GUIrilla-Task
Ground-truth Click & Type actions for macOS screenshots
Dataset Summary
GUIrilla-Task pairs real macOS screenshots with free-form natural-language instructions and precise GUI actions. Every sample asks an agent either to:
- Click a specific on-screen element, or
- Type a given text into an input field.
Targets are labelled with bounding-box geometry, enabling exact evaluation of visual-language grounding models. Data were gathered automatically by the GUIrilla crawler, which explores applications via the macOS Accessibility API while GPT-4 agents generate diverse, user-style tasks.
Dataset Structure
Data Fields
| Field | Type | Description |
|---|---|---|
screen_id | int | Unique screenshot index. |
app_name | string | Bundle name (e.g. com.apple.Safari). |
task | string | Final, cleaned instruction. |
raw_task | string | Raw task draft. |
action | string | Either "left click" or "type <text>". |
image | image | Full-resolution PNG. |
image_cropped | image | Tight crop of the app window. |
accessibility | string | Accessibility JSON tree for the app window. |
scaling_factor | int | UI scaling for that capture. |
element_data | string | Bounding-box / polygon / role / label in JSON. |
original_task | bool | True if task was performed in the macOS environment. |
task_category | string | High-level task category (Search & Information, Files etc.). |
element_category | string | High-level AX role (button, textfield, etc.). |
If the task is click,
element_datacontains a bounding box For type, it stores the insertion location and expected text.
Splits
| Split | Examples | Uncompressed Size | Notes |
|---|---|---|---|
train | 25 606 | 47.8 GB | Training & dev experiments |
test | 1 565 | 2.32 GB | Held-out evaluation split |
The two splits are application-level disjoint: a specific app build never appears in both.
Intended Uses
| ✔️ Recommended | ❌ Not Recommended |
|---|---|
| Training & evaluating macOS GUI agents (vision–language models, RPA, RL). | Commercial redistribution (license is NC). |
| Academic research on accessibility, automated testing, multimodal grounding. | Deployment in safety-critical automation without human oversight. |
Loading Example
from datasets import load_dataset
from PIL import Image
ds_train = load_dataset("macpaw-research/GUIrilla-Task", split="train")
row = ds_train[0]
print(row["task"], "→", row["action"])
Image.open(row["image"].filename).show()
Image.open(row["image_cropped"].filename).show()
License
CC-BY-NC-4.0 (see LICENSE).
This dataset contains screenshots and accessibility metadata captured from third-party macOS applications; the non-commercial licence reflects that provenance.
In short (not legal advice):
- ✅ Non-commercial research — training and evaluating models, publishing papers and benchmark results, redistributing for non-commercial purposes (all with attribution).
- ❌ Commercial use — e.g. training or benchmarking models for commercial products, or redistributing the data as part of a commercial offering — is not covered by this licence.
Interested in using this dataset commercially? Contact MacPaw Research to discuss options.
Citation
@article{garkot2025guirilla,
title={GUIrilla: A Scalable Framework for Automated Desktop UI Exploration},
author={Garkot, Sofiya and Shamrai, Maksym and Synytsia, Ivan and Hirna, Mariya},
journal={arXiv preprint arXiv:2510.16051},
year={2025},
url={https://arxiv.org/abs/2510.16051}
}