7
stars
3
commits
3
linked in READMEs
May 8, 2025
updated
Paper | Code | Dataset | Project
MONDAY (Mobile OS Navigation Task Dataset for Agents from YouTube) is a cross-platform mobile navigation dataset for training vision-language models. This dataset contains
Please visit our project page for more details.
video_id (str): Unique identifier for the video.
title (str): Title of the video.
os (str): Operating system of the mobile device used in the video.
num_scenes (int): Number of detected scenes in the video.
scene_timestamps_in_sec (list): A list of timestamps of the detected scenes in seconds. The list has a length of num_scenes.
screen_bboxes (list): A list of bounding boxes for the detected phone screen in each scene, given as (left, top, right, bottom) pixel coordinates. The list has a length of num_scenes.
ui_element_bboxes (list): A list of bounding boxes for the detected user interface (UI) elements in each scene, given as (left, top, right, bottom) coordinates normalized to the [0, 1] range. The list has a length of num_scenes - 1.
# example
ui_element_bboxes = [
[ui_bbox1_scene1, ui_bbox2_scene1, ...], # UI elements in scene 1
[ui_bbox1_scene2, ui_bbox2_scene2, ...], # UI elements in scene 2
...
]
raw_actions (list): A list of raw actions identified from the video for each scene. The list has a length of num_scenes - 1. Multiple actions can be annotated within a single scene, and all are considered valid. Each element is a list of actions annotated in that scene, with each action represented as a dictionary containing the following keys:
ui_element_bboxes[scene_id]) associated with the action. If the action does not correspond to any UI element, the value is -1."touch", "scroll", "hardware", "typing", "long press", "multi touch" and "zoom", listed in order of frequency in the dataset.# example
raw_actions = [
[
{"box_id": 0, "details": "...", "type": "touch"}, # First action in scene 1
{"box_id": 1, "details": "...", "type": "touch"}, # Second action in scene 1
],
[
{"box_id": -1, "details": "...", "type": "typing"}, # First action in scene 2
],
...
]
Note: The box_id is -1 for actions that do not correspond to any UI element.
actions (list): A list of actions in each scene, processed for mobile navigation agent training and evaluation. The list has a length of num_scenes - 1. Multiple actions can be annotated within a single scene, and all are considered valid. Each element is a list of actions annotated in that scene, with each action represented as a dictionary containing the following keys:
"click", "scroll down", "press home", "type", "scroll up", "other hardware", "scroll left", "zoom or multi-touch", "press power", "scroll right", and "press back", listed in order of frequency in the dataset.4 * num_ui_elements per scene; otherwise, it is an empty list."type"; otherwise, this is an empty string.# example
actions = [
[
{"action_type_id": 4, "action_type_text": "click", "annot_position": annot_position, "lift": lift_point_action1, "touch": touch_point_action1, "type_text": ""}, # First action in scene 1
{"action_type_id": 4, "action_type_text": "click", "annot_position": annot_position, "lift": lift_point_action2, "touch": touch_point_action2, "type_text": ""}, # Second action in scene 1
],
[
{"action_type_id": 3, "action_type_text": "type", "annot_position": [], "lift": [-1, -1], "touch": [-1, -1], "type_text": "..."}, # First action in scene 2
],
...
]
Note: The data format of actions is derived from SeeClick and AitW.
video_fps (float): Frames per second of the video. This value must be preserved when downloading the video to ensure consistency with scene_timestamps_in_sec.
video_width (int): Width of the video in pixels. This value must be preserved when downloading the video to ensure consistency with screen_bboxes.
video_height (int): Height of the video in pixels. This value must be preserved when downloading the video to ensure consistency with screen_bboxes.
@inproceedings{jang2025_monday,
title={{Scalable Video-to-Dataset Generation for Cross-Platform Mobile Agents}},
author={Jang, Yunseok and Song, Yeda and Sohn, Sungryull and Logeswaran, Lajanugen and Luo, Tiange and Kim, Dong-Ki and Bae, Kyunghoon and Lee, Honglak},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
year={2025}
}
3 commits
7
stars
3
commits
3
linked in READMEs
May 8, 2025
updated
Paper | Code | Dataset | Project
MONDAY (Mobile OS Navigation Task Dataset for Agents from YouTube) is a cross-platform mobile navigation dataset for training vision-language models. This dataset contains
Please visit our project page for more details.
video_id (str): Unique identifier for the video.
title (str): Title of the video.
os (str): Operating system of the mobile device used in the video.
num_scenes (int): Number of detected scenes in the video.
scene_timestamps_in_sec (list): A list of timestamps of the detected scenes in seconds. The list has a length of num_scenes.
screen_bboxes (list): A list of bounding boxes for the detected phone screen in each scene, given as (left, top, right, bottom) pixel coordinates. The list has a length of num_scenes.
ui_element_bboxes (list): A list of bounding boxes for the detected user interface (UI) elements in each scene, given as (left, top, right, bottom) coordinates normalized to the [0, 1] range. The list has a length of num_scenes - 1.
# example
ui_element_bboxes = [
[ui_bbox1_scene1, ui_bbox2_scene1, ...], # UI elements in scene 1
[ui_bbox1_scene2, ui_bbox2_scene2, ...], # UI elements in scene 2
...
]
raw_actions (list): A list of raw actions identified from the video for each scene. The list has a length of num_scenes - 1. Multiple actions can be annotated within a single scene, and all are considered valid. Each element is a list of actions annotated in that scene, with each action represented as a dictionary containing the following keys:
ui_element_bboxes[scene_id]) associated with the action. If the action does not correspond to any UI element, the value is -1."touch", "scroll", "hardware", "typing", "long press", "multi touch" and "zoom", listed in order of frequency in the dataset.# example
raw_actions = [
[
{"box_id": 0, "details": "...", "type": "touch"}, # First action in scene 1
{"box_id": 1, "details": "...", "type": "touch"}, # Second action in scene 1
],
[
{"box_id": -1, "details": "...", "type": "typing"}, # First action in scene 2
],
...
]
Note: The box_id is -1 for actions that do not correspond to any UI element.
actions (list): A list of actions in each scene, processed for mobile navigation agent training and evaluation. The list has a length of num_scenes - 1. Multiple actions can be annotated within a single scene, and all are considered valid. Each element is a list of actions annotated in that scene, with each action represented as a dictionary containing the following keys:
"click", "scroll down", "press home", "type", "scroll up", "other hardware", "scroll left", "zoom or multi-touch", "press power", "scroll right", and "press back", listed in order of frequency in the dataset.4 * num_ui_elements per scene; otherwise, it is an empty list."type"; otherwise, this is an empty string.# example
actions = [
[
{"action_type_id": 4, "action_type_text": "click", "annot_position": annot_position, "lift": lift_point_action1, "touch": touch_point_action1, "type_text": ""}, # First action in scene 1
{"action_type_id": 4, "action_type_text": "click", "annot_position": annot_position, "lift": lift_point_action2, "touch": touch_point_action2, "type_text": ""}, # Second action in scene 1
],
[
{"action_type_id": 3, "action_type_text": "type", "annot_position": [], "lift": [-1, -1], "touch": [-1, -1], "type_text": "..."}, # First action in scene 2
],
...
]
Note: The data format of actions is derived from SeeClick and AitW.
video_fps (float): Frames per second of the video. This value must be preserved when downloading the video to ensure consistency with scene_timestamps_in_sec.
video_width (int): Width of the video in pixels. This value must be preserved when downloading the video to ensure consistency with screen_bboxes.
video_height (int): Height of the video in pixels. This value must be preserved when downloading the video to ensure consistency with screen_bboxes.
@inproceedings{jang2025_monday,
title={{Scalable Video-to-Dataset Generation for Cross-Platform Mobile Agents}},
author={Jang, Yunseok and Song, Yeda and Sohn, Sungryull and Logeswaran, Lajanugen and Luo, Tiange and Kim, Dong-Ki and Bae, Kyunghoon and Lee, Honglak},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
year={2025}
}
3 commits