This repo stores the training datasets used to train the AutoGUI model.

Autogui-625k: This is the v0.1 dataset collected by our AutoGUI annotation pipeline on 05/31/2024. For the v0.2 dataset, refer to https://huggingface.co/datasets/AutoGUI/AutoGUI-v1-702k

Cauldron: This is one of the two general datasets used to maintain the general visual understanding ability of the trained VLM. We select the Screen2Words, DocVQA, OCR-VQA, visualmrc, infovga, and Diagram image-to-text from the whole Cauldron dataset.

llava_150k: This is the 150k instruction-following dataset from LLaVA. Please refer to https://github.com/haotian-liu/LLaVA/blob/main/docs/Data.md

After downloading all the three datasets, please modify the image path of every sample in the JSON files to be its absolute path required by the Qwen-VL fine-tuning framework. Then merge the 3 JSON files. This merged file will be used as the data_path argument in the AutoGUI training script.