macpaw-research/GUIrilla-Gold

Dataset

stars

commits

linked in READMEs

Aug 18, 2026

updated

Browse cluster: GUI Agents and Automation →

README

GUIrilla-Gold

Dataset Summary

GUIrilla-Gold benchmark is manually annotated test part from GUIrilla-Task.

Dataset Structure

Data Fields

Field	Type	Description
`screen_id`	`int`	Unique screenshot index.
`app_name`	`string`	Bundle name (e.g. `com.apple.Safari`).
`task`	`string`	Final, cleaned instruction.
`raw_task`	`string`	Raw task draft.
`action`	`string`	Either `"left click"` or `"type <text>"`.
`image`	`image`	Full-resolution PNG.
`image_cropped`	`image`	Tight crop of the app window.
`accessibility`	`string`	Accessibility JSON tree for the app window.
`scaling_factor`	`int`	UI scaling for that capture.
`element_data`	`string`	Bounding-box / polygon / role / label in JSON.
`original_task`	`bool`	`True` if task was performed in the macOS environment.
`task_category`	`string`	High-level task category (Search & Information, Files etc.).
`element_category`	`string`	High-level AX role (`button`, `textfield`, etc.).

Loading Example

from datasets import load_dataset
from PIL import Image

ds = load_dataset("macpaw-research/GUIrilla-Gold")

row = ds[0]
print(row["task"], "→", row["action"])
Image.open(row["image"].filename).show()
Image.open(row["image_cropped"].filename).show()

License

CC-BY-NC-4.0 (see LICENSE).

This dataset contains screenshots and accessibility metadata captured from third-party macOS applications; the non-commercial licence reflects that provenance.

In short (not legal advice):

✅ Non-commercial research — training and evaluating models, publishing papers and benchmark results, redistributing for non-commercial purposes (all with attribution).
❌ Commercial use — e.g. training or benchmarking models for commercial products, or redistributing the data as part of a commercial offering — is not covered by this licence.

Interested in using this dataset commercially? Contact MacPaw Research to discuss options.

Citation

@article{garkot2025guirilla,
  title={GUIrilla: A Scalable Framework for Automated Desktop UI Exploration},
  author={Garkot, Sofiya and Shamrai, Maksym and Synytsia, Ivan and Hirna, Mariya},
  journal={arXiv preprint arXiv:2510.16051},
  year={2025},
  url={https://arxiv.org/abs/2510.16051}
}

Contributors

Krivoblotsky

4 commits

mshamrai

4 commits

victormuryn

1 commits

Similar repos

macpaw-research/GUIrilla-Task

Dataset

rootsautomation/ScreenSpot

Dataset

macpaw-research/GUIrilla-Trees

Dataset

ServiceNow/ui-vision

Dataset

showlab/ShowUI-desktop

Dataset

Aria-UI/Aria-UI_Data

Dataset

OpenGVLab/ScaleCUA-Data

Dataset

hflqf88888/GUIOdyssey

Dataset

Linked in READMEs

MacPaw/GUIrilla

Repository for the implementation of GUIrilla

README badge

macpaw-research/GUIrilla-Gold

Dataset

stars

commits

linked in READMEs

Aug 18, 2026

updated

Browse cluster: GUI Agents and Automation →

README

GUIrilla-Gold

Dataset Summary

GUIrilla-Gold benchmark is manually annotated test part from GUIrilla-Task.

Dataset Structure

Data Fields

Field	Type	Description
`screen_id`	`int`	Unique screenshot index.
`app_name`	`string`	Bundle name (e.g. `com.apple.Safari`).
`task`	`string`	Final, cleaned instruction.
`raw_task`	`string`	Raw task draft.
`action`	`string`	Either `"left click"` or `"type <text>"`.
`image`	`image`	Full-resolution PNG.
`image_cropped`	`image`	Tight crop of the app window.
`accessibility`	`string`	Accessibility JSON tree for the app window.
`scaling_factor`	`int`	UI scaling for that capture.
`element_data`	`string`	Bounding-box / polygon / role / label in JSON.
`original_task`	`bool`	`True` if task was performed in the macOS environment.
`task_category`	`string`	High-level task category (Search & Information, Files etc.).
`element_category`	`string`	High-level AX role (`button`, `textfield`, etc.).

Loading Example

from datasets import load_dataset
from PIL import Image

ds = load_dataset("macpaw-research/GUIrilla-Gold")

row = ds[0]
print(row["task"], "→", row["action"])
Image.open(row["image"].filename).show()
Image.open(row["image_cropped"].filename).show()

License

CC-BY-NC-4.0 (see LICENSE).

This dataset contains screenshots and accessibility metadata captured from third-party macOS applications; the non-commercial licence reflects that provenance.

In short (not legal advice):

✅ Non-commercial research — training and evaluating models, publishing papers and benchmark results, redistributing for non-commercial purposes (all with attribution).
❌ Commercial use — e.g. training or benchmarking models for commercial products, or redistributing the data as part of a commercial offering — is not covered by this licence.

Interested in using this dataset commercially? Contact MacPaw Research to discuss options.

Citation

@article{garkot2025guirilla,
  title={GUIrilla: A Scalable Framework for Automated Desktop UI Exploration},
  author={Garkot, Sofiya and Shamrai, Maksym and Synytsia, Ivan and Hirna, Mariya},
  journal={arXiv preprint arXiv:2510.16051},
  year={2025},
  url={https://arxiv.org/abs/2510.16051}
}