Click-100k is a high-quality grounding dataset used to train 🍨 Gelato-30B-A3B a state-of-the-art grounding model for GUI computer-use tasks. The dataset pairs computer screen frames with low-level GUI commands and click coordinates, enabling models to learn precise UI element interaction. Click-100k was built by filtering and unifying multiple public datasets and enriching them with professional application data.
Each instance in Click-100k contains:
Example:
{
'image_path': 'showui-desktop-images/showui_desktop_004594.jpg',
'images': [<PIL.Image>],
'easyr1_prompt': 'You are an expert UI element locator. Given a GUI image and a user\'s element description, provide the coordinates of the specified element as a single (x,y) point. For elements with area, return the center point. Output the coordinate pair exactly: (x,y) <image> Tap on Located between \'Background\' and \'Notifications\' options.',
'bbox': [72, 183, 322, 232],
'image_width': 1960,
'image_height': 1092,
'normalized_bbox': [0.036734693877551024, 0.16758241758241757, 0.16428571428571428, 0.21245421245421245]
}
image_path (string): Relative path to the screenshot imageimages (list): List containing the PIL Image object(s)easyr1_prompt (string): The instruction prompt for the grounding task, including system instructions and the natural language description of the target UI elementbbox (list of 4 integers): Bounding box coordinates in [x_min, y_min, x_max, y_max] formatimage_width (int): Width of the screenshot in pixelsimage_height (int): Height of the screenshot in pixelsnormalized_bbox (list of 4 floats): Normalized bounding box coordinates in [0, 1] rangeExisting open-source GUI grounding datasets suffer from:
Click-100k addresses these issues through careful curation and filtering to create a high-quality training resource for computer-use agents.
Click-100k is built by filtering and unifying multiple public datasets:
All sources were normalized into a unified format, with a maximum of 50k instances per source to prevent dataset skewing.
The dataset underwent extensive filtering and quality control:
@misc{gelato2025,
title={Gelato — From Data Curation to Reinforcement Learning: Building a Strong Grounding Model for Computer-Use Agents},
author={Anas Awadalla, Dhruba Ghosh, Aylin Akkus, Yuhui Zhang, Marianna Nezhurina, Jenia Jitsev, Yejin Choi, Ludwig Schmidt},
year={2025},
publisher={GitHub},
howpublished={\url{https://github.com/mlfoundations/gelato}},
}
9 commits
Click-100k is a high-quality grounding dataset used to train 🍨 Gelato-30B-A3B a state-of-the-art grounding model for GUI computer-use tasks. The dataset pairs computer screen frames with low-level GUI commands and click coordinates, enabling models to learn precise UI element interaction. Click-100k was built by filtering and unifying multiple public datasets and enriching them with professional application data.
Each instance in Click-100k contains:
Example:
{
'image_path': 'showui-desktop-images/showui_desktop_004594.jpg',
'images': [<PIL.Image>],
'easyr1_prompt': 'You are an expert UI element locator. Given a GUI image and a user\'s element description, provide the coordinates of the specified element as a single (x,y) point. For elements with area, return the center point. Output the coordinate pair exactly: (x,y) <image> Tap on Located between \'Background\' and \'Notifications\' options.',
'bbox': [72, 183, 322, 232],
'image_width': 1960,
'image_height': 1092,
'normalized_bbox': [0.036734693877551024, 0.16758241758241757, 0.16428571428571428, 0.21245421245421245]
}
image_path (string): Relative path to the screenshot imageimages (list): List containing the PIL Image object(s)easyr1_prompt (string): The instruction prompt for the grounding task, including system instructions and the natural language description of the target UI elementbbox (list of 4 integers): Bounding box coordinates in [x_min, y_min, x_max, y_max] formatimage_width (int): Width of the screenshot in pixelsimage_height (int): Height of the screenshot in pixelsnormalized_bbox (list of 4 floats): Normalized bounding box coordinates in [0, 1] rangeExisting open-source GUI grounding datasets suffer from:
Click-100k addresses these issues through careful curation and filtering to create a high-quality training resource for computer-use agents.
Click-100k is built by filtering and unifying multiple public datasets:
All sources were normalized into a unified format, with a maximum of 50k instances per source to prevent dataset skewing.
The dataset underwent extensive filtering and quality control:
@misc{gelato2025,
title={Gelato — From Data Curation to Reinforcement Learning: Building a Strong Grounding Model for Computer-Use Agents},
author={Anas Awadalla, Dhruba Ghosh, Aylin Akkus, Yuhui Zhang, Marianna Nezhurina, Jenia Jitsev, Yejin Choi, Ludwig Schmidt},
year={2025},
publisher={GitHub},
howpublished={\url{https://github.com/mlfoundations/gelato}},
}
9 commits