Dataset Card for RICO SCA (SeeClick cache)

This is the SeeClick cache of a syntehtically generated dataset following RICO SCA's generation procedure. It consists of approximately 170k captions across 70k widgets and 18k screens.

Dataset Details

Dataset Description

This is a widget captioning (referring expression comprehension/generation) dataset.

• Curated by: Google Research, Nanjing University
• Language(s) (NLP): en
• License: apache-2.0

Dataset Sources

• Repository: RICO SCA Repo / SeeClick Repo
• Paper [optional]: RICO SCA paper / SeeClick paper

Uses

Direct Use

Training models that can be used to understand screens, explain mobile interaces, and act in an automated, digital context.

Dataset Structure

• screenId: Unique RICO screen ID
• image: RICO screenshot
• image_icon: Google Play Store icon for the app
• image_semantic: Semantic RICO screenshot; details are abstracted away to main visual UI elements
• file_name: Image local filename
• file_name_icon: Icon image local filename
• file_name_semantic: Screenshot Image as a semantic annotated image local filename
• captions: A list of string captions
• bbox: The bounding box for the widget being captioned, relatively scaled with the image size so that coordinates are in [0, 1]
• app_package_name: Android package name
• play_store_name: Google Play Store name
• category: Type of category of the app
• number_of_downloads: Number of downloads of the app (as a coarse range string)
• number_of_ratings: Number of ratings of the app on the Google Play store (as of collection)
• average_rating: Average rating of the app on the Google Play Store (as of collection)
• semantic_annotations: Reduced view hierarchy, to the semantically-relevant portions of the full view hierarchy. It corresponds to what is visualized in image_semantic and has a lot of details about what's on screen. It is stored as a JSON object string.

Dataset Creation

Curation Rationale

• RICO rationale: Create a broad dataset that can be used for UI automation. An explicit goal was to develop automation software that can validate an app's design and assess whether it achieves its stated goal.
• SCA rationale: Primarily to benefit efforts to create more assitive technologies for visually-impaired users

Source Data

• RICO: Mobile app screenshots, collected on Android devices.
• SCA: Generated from a trained model

Citation

RICO

BibTeX:

@inproceedings{deka2017rico,
  title={Rico: A mobile app dataset for building data-driven design applications},
  author={Deka, Biplab and Huang, Zifeng and Franzen, Chad and Hibschman, Joshua and Afergan, Daniel and Li, Yang and Nichols, Jeffrey and Kumar, Ranjitha},
  booktitle={Proceedings of the 30th annual ACM symposium on user interface software and technology},
  pages={845--854},
  year={2017}
}

APA:

Deka, B., Huang, Z., Franzen, C., Hibschman, J., Afergan, D., Li, Y., ... & Kumar, R. (2017, October). Rico: A mobile app dataset for building data-driven design applications. In Proceedings of the 30th annual ACM symposium on user interface software and technology (pp. 845-854).

RICO SCA

BibTeX:

@inproceedings{li2020mapping,
  title={Mapping Natural Language Instructions to Mobile UI Action Sequences},
  author={Li, Yang and He, Jiacong and Zhou, Xin and Zhang, Yuan and Baldridge, Jason},
  booktitle={Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  pages={8198--8210},
  year={2020}
}

APA:

Li, Y., He, J., Zhou, X., Zhang, Y., & Baldridge, J. (2020, July). Mapping Natural Language Instructions to Mobile UI Action Sequences. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (pp. 8198-8210).

SeeClick

BibTeX:

@misc{cheng2024seeclick,
      title={SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents}, 
      author={Kanzhi Cheng and Qiushi Sun and Yougang Chu and Fangzhi Xu and Yantao Li and Jianbing Zhang and Zhiyong Wu},
      year={2024},
      eprint={2401.10935},
      archivePrefix={arXiv},
      primaryClass={cs.HC}
}

APA:

Cheng, K., Sun, Q., Chu, Y., Xu, F., Li, Y., Zhang, J., & Wu, Z. (2024). Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401.10935.

Dataset Card Authors

Hunter Heidenreich, Roots Automation

Dataset Card Contact

hunter "DOT" heidenreich "AT" rootsautomation "DOT" com