1️⃣ Clone Repository

git clone https://github.com/ZJULiHongxin/UIPro.git
cd UIPro

2️⃣ Install Dependencies

pip install -r requirements.txt

1. Download the MobileViews raw data from HuggingFace via:

   hf download mllmTeam/MobileViews --repo-type dataset --local-dir ./MobileViews
   

2. Unzip and organize the data as follows:

   root/
   ├── MobileViews/
   │   ├── MobileViews_0-150000/
   │   ├── MobileViews_0-150000.csv
   │   ├── MobileViews_150001-291197/
   │   ├── MobileViews_150001-291197.csv
   │   └── ...
   

3. Modify MOBILEVIEWS_DIR, ROOT, SCALE (coordinate scale), and PROB_BOX (proportion of the box-prediction samples) in utils/data_utils/make_mobileviews_data/extract_and_generate_mobilebiews_data.py.

4. Run the processing script (this may take ~48 hours due to the large number of screenshots):

   python utils/data_utils/make_mobileviews_data/extract_and_generate_mobilebiews_data.py
   

   Processed training samples will be saved in ROOT/mobileviews_processed.

5. Finally, run utils/data_utils/make_mobileviews_data/run_generate_symlinks.sh to create a unified folder for screenshots.

1. Download the WAE raw data from WAE DropBox.

2. Merge, unzip and organize the data as follows:

   root/
   ├── WAE/
   │   ├── COM.HSBFREE_25-output
   │   ├── Com.sktelecom.minit_52-output
   │   ├── Draziw.Button.Mines_71-output
   │   ├── HBVerbrauchszaehler.lite_119-output
   │   └── ...
   

3. Modify WAE_DIR, ROOT_DIR, SCALE (coordinate scale), and PROB_BOX (proportion of the box-prediction samples) in utils/data_utils/make_WAE_data/make_WAE_data.py.

4. Run the processing script (this may take ~24 hours due to the large number of screenshots):

   python utils/data_utils/make_WAE_data/make_WAE_data.py
   

   Processed training samples will be saved in ROOT/WAE_processed.

1. Download the WebUI raw data from biglab/webui-all.

2. Merge, unzip and organize the data as follows:

   root/
   ├── WebUI/
   │   ├── dh2 (GUI metadata resulting from unzipping)
   │   ├── WebUI_screenshots (A folder used to save processed GUI screenshots)
   

3. Modify WEBUI_DIR, WEBUI_PROCESSED_IMG_DIR, ROOT, SCALE (coordinate scale), and PROB_BOX (proportion of the box-prediction samples) in utils/data_utils/make_webui_data/make_webui_data.py.

4. Run the processing script (this may take ~24 hours due to the large number of screenshots):

   python utils/data_utils/make_webui_data/make_webui_data.py
   

   Processed training samples will be saved in ROOT/WebUI_processed.

1. Download the MultiUI raw data from neulab/MultiUI.

2. Merge, unzip and organize the data as follows:

   root/
   ├── MultiUI/
   │   ├── v0.6_5M
   │   ├── v0.7_exclude_v0.6
   |   ├── v0.8_exclude_v0.7
   │   ├── stage1_data.json
   │   ├── stage1_data_10k.json
   |   ├── stage2_data_to_be_combined_with_general_data.json
   

3. Modify MULTIUI_SAMPLE_FILE, IMG_DIR, SAVE_ROOT, and SCALE (coordinate scale) in utils/data_utils/make_multiui_data/make_multiui_data.py.

4. Run the processing script (this may take ~2 hours due to the large number of screenshots):

   python utils/data_utils/make_multiui_data/make_multiui_data.py
   

   Processed training samples will be saved in ROOT/MultiUI_processed.

1. Download the SeeClick-Web raw data from SeeClick-Web Annotation File and SeeClick-Web Images.

2. Unzip and organize the data as follows:

   root/
   ├── SeeClick-Web/
   │   ├── 0a5c8a5b7d73de574f2a21f27dbc9a53.png
   │   ├── 0a6dcd3f9e1907af232e2c038a866f74.png
   | ...
   

3. Modify IMG_DIR, ANNO_FILE, SAVE_ROOT, and SCALE (coordinate scale) in utils/data_utils/make_seeclickweb_data/make_seeclickweb_data.py.

4. Run the processing script (this may take ~24 hours due to the large number of screenshots):

   python utils/data_utils/make_seeclickweb_data/make_seeclickweb_data.py
   

   Processed training samples will be saved in ROOT/SeeClick-Web_processed.

1. Download the GUIEnv raw data from yiye2023/GUIEnv.

2. Unzip and organize the data as follows:

   root/
   │   └── GUICourse/
   │            └── GUIEnv/
   │                    ├── imgs
   │                    ├── ocr_grounding_train_stage2_images.parquet
   │                    ├── ocr_grounding_train_stage2_data.json
   │                    ├── ocr_grounding_train_stage1_images.parquet
   │                    ├── ocr_grounding_train_stage1_data.json
   │                    ├── ocr_grounding_test_images.parquet
   │                    └── ocr_grounding_test_data.json
   
   

3. Modify SUBTASK, CURRENT_SPLIT, DATA_ROOT, SAVE_DIR, ENABLE_TEXTLOC, ENABLE_OCR, ENABLE_INTENT_GND, and SCALE (coordinate scale) in utils/data_utils/make_refexp_data/make_refexp_data.py.

4. Run the processing script:

   python utils/data_utils/make_refexp_data/make_refexp_data.py
   

   Processed training samples will be saved in ROOT/RefExp_processed.

1. Download the RICO image data from SeeClick RICO Data.

2. Unzip and organize the data as follows:

   root/
   │   └── rico/
   │       ├── 72197.jpg
   │       ├── ...
   │       ├── 71949.json
   │       └── ...
   
   

3. Modify IMG_DIR, SAVE_ROOT, , and SCALE (coordinate scale) in utils/data_utils/make_refexp_data/make_refexp_data.py.

4. Run the processing script:

   python utils/data_utils/make_refexp_data/make_refexp_data.py
   

   Processed training samples will be saved in ROOT/RefExp_processed.

1. Download the MOTIF image data from HongxinLi/MOTIF.

2. Unzip and organize the data as follows:

   root/
   │   └── motif/
   
   

3. Modify IMG_DIR, SAVE_ROOT_DIR, and SCALE (coordinate scale) in utils/data_utils/make_motif_data/make_motif_data.py.

4. Run the processing script:

   python utils/data_utils/make_motif_data/make_motif_data.py
   

   Processed training samples will be saved in ROOT/MOTIF_processed.

1. Download the OmniAct raw data from Writer/omniact.

2. Unzip and organize the data as follows:

   root/
   ├── OmniAct/
   │   └── data/
   │       ├── tasks/
   │       ├── metadata/
   │       └── data/
   

3. Modify ROOT_DIR, SAVE_ROOT_DIR, SPLIT, and SCALE (coordinate scale) in utils/data_utils/make_omniact_data/make_omniact_data.py.

4. Run the processing script:

   python utils/data_utils/make_omniact_data/make_omniact_data.py
   

   Processed training samples will be saved in ROOT/OmniAct_processed.

1. Download AiTW screenshots from here and annotations from here.
2. Organize the data:

   root/
   ├── AITW/
   │   ├── aitw_data_test.json
   │   ├── aitw_data_train.json
   │   ├── aitw_data_val.json
   │   └── aitw_images/
   │       ├── general/
   │       ├── googleapps/
   │       ├── install/
   │       ├── single/
   │       └── webshopping/
   

3. Modify ROOT, SAVE_DIR, SPLIT, and POINT_FORMAT in utils/data_utils/make_aitw_data/make_aitw_data.py.
4. Run the script:

   python utils/data_utils/make_aitw_data/make_aitw_data.py
   

   Processed samples will be saved in SAVE_DIR/AITW_processed.

1. Download raw data following instructions in the AitZ Github Repo.
2. Organize the data:

   root/
   ├── AITZ/
   │   ├── train/
   │   │   ├── general/
   │   │   ├── googleapps/
   │   │   ├── install/
   │   │   ├── single/
   │   │   └── webshopping/
   │   └── test/
   │       ├── general/
   │       ├── googleapps/
   │       ├── install/
   │       └── webshopping/
   

3. Modify ROOT, SAVE_DIR, SCALE, SPLIT, and USE_ACTION_REFEXP in utils/data_utils/make_aitz_data/make_aitz_data.py.
4. Run the script:

   python utils/data_utils/make_aitz_data/make_aitz_data.py
   

   Processed samples will be saved in SAVE_DIR/AITZ_processed.

1. Download raw data following instructions in the AndroidControl Github Repo.
2. Organize the data:

   root/
   ├── AndroidControl/
   │   ├── raw/
   │   │   ├── android_control-00000-of-00020
   │   │   ├── android_control-00001-of-00020
   │   │   ├── ...
   │   │   ├── android_control-00019-of-00020
   │   │   └── splits.json
   

3. Modify ANDROIDCONTROL_ROOT, SAVE_DIR, SPLIT, and POINT_FORMAT in utils/data_utils/make_androidcontrol_data/make_androidcontrol_data.py.
4. Run the script:

   python utils/data_utils/make_androidcontrol_data/make_androidcontrol_data.py
   

   Processed samples will be saved in SAVE_DIR/AndroidControl_processed.

1. Download raw data from the GUIOdyssey HF Repo.
2. Organize the data:

   root/
   ├── GUIOdyssey_raw/
   │   ├── screenshots/
   │   │   ├── 2386365564178401_9.png
   │   │   ├── 5022534067657028_12.png
   │   │   ├── 7287738713744873_13.png
   │   │   └── ...
   │   ├── splits/
   │   └── annotations/
   

3. Move all images from data_* subfolders in screenshots directly to screenshots.
4. Modify DATA_ROOT, SAVE_ROOT, and SPLIT in utils/data_utils/make_guiodyssey_data/make_guiodyssey_data.py.
5. Run the script:

   python utils/data_utils/make_guiodyssey_data/make_guiodyssey_data.py
   

   Processed samples will be saved in SAVE_ROOT/GUIOdyssey_processed.

1. Download raw data from the AMEX HF Repo.
2. Organize the data:

   root/
   ├── AMEX/
   │   ├── element_anno/
   │   ├── screenshot/
   │   └── metadata/
   

3. Modify DATA_ROOT, SAVE_ROOT, and SPLIT in utils/data_utils/make_amex_data/make_amex_data.py.
4. Run the script:

   python utils/data_utils/make_amex_data/make_amex_data.py
   

   Processed samples will be saved in SAVE_ROOT/AMEX_processed.

1. Download raw data from the GUIAct HF Repo by running hf download yiye2023/GUIAct --repo-type dataset --local-dir path/to/GUICourse/GUIAct.
2. Organize the data:

   root/
   ├── GUICourse/
   │   ├── GUIAct/
   │   │   ├── smartphone_test_data.json
   │   │   ├── smartphone_test_images.parquet
   │   │   ├── smartphone_train_data.json
   │   │   └── ...
   

3. Modify DATA_ROOT, SAVE_DIR, CURRENT_SPLIT, and CURRENT_DEVICE_TYPE in the DatasetConfig class within utils/data_utils/make_guicourse_data/make_guicourse_data.py.
4. Run the script:

   python utils/data_utils/make_guicourse_data/make_guicourse_data.py
   

   Processed samples will be saved in SAVE_DIR.

{
  "mobile_actions": [
    "tap", "long_press", "drag", "input_text",
    "navigate_home", "navigate_back", "navigate_recent",
    "press_enter", "swipe", "wait", "status_complete"
  ]
}

{
  "action": "swipe",
  "start": [x, y],          // Starting coordinates
  "direction": "up",        // Movement direction  
  "distance": 200           // Swipe distance in pixels
}

What you need

• AiTW screenshots (see the Agentic SFT Data Processing → Android in the Wild (AiTW) section below). You only need the image folder for evaluation.
• The script loads the test annotations from HuggingFace: HongxinLi/AITW_test.

Expected images layout

/path/to/AITW/aitw_images/
├── general/
├── googleapps/
├── install/
├── single/
└── webshopping/

Run

python utils/eval_utils/eval_aitw.py \
  --pretrained "HongxinLi/UIPro_2stage_Mobile" \
  --imgs_dir "/path/to/AITW/aitw_images" \
  --action_refexp \
  --scale 1000

Common options

• --debug: only evaluate a few steps for quick sanity check
• --cot: enable Chain-of-Thought prompting
• --action_refexp: used to drive the UIPro model to output a short action intent before predicting the action function call.
• --max_prev_acts: how many previous actions are included in history
• --original_actspace: use the original AITW action space in prompts. Ignore this to use the unified action space provided by UIPro.

Outputs

• Saved to utils/eval_utils/eval_results/AITW/<model_postfix>[_CoT][_wActRef]/<timestamp>.json

What you need

• A local AndroidControl test JSON (e.g., the processed file produced by utils/data_utils/make_androidcontrol_data/make_androidcontrol_data.py).
• The images should be reachable by joining --data_root with the image field in the test JSON.

We also provide a preprocessed zip file of the AndroidControl test data used in the UIPro paper. First download and unzip the GoClick AndroidControl Test Data via hf download HongxinLi/AndroidControl_test --repo-type dataset --local-dir path/to/AndroidControl_test

Run

python utils/eval_utils/eval_androidcontrol.py \
  --pretrained "HongxinLi/UIPro_2stage_Mobile" \
  --testset_path "utils/eval_utils/AndroidControl-test_12685.json" \
  --data_root "/path/to/AndroidControl"

Useful options

• --action_refexp: (only used for UIPro) Please use this to prompt UIPro to output short reasoning before planning the action

Outputs

• Saved to utils/eval_utils/eval_results/androidcontrol/<model_postfix>/<timestamp>.json

What you need

• A local GUIAct processed folder that contains the test JSON and images referenced by the image field.
• You can pass --root to point to your processed GUIAct directory, and --imgs_dir to override the image base. If omitted, the script auto-detects local/server paths inside get_dataset_paths() in utils/eval_utils/eval_guiact.py.

We also provide the preprocessed GUIAct test data zip file. First download the GUIAct data from HongxinLi/GUIAct, unzip it, and organize it as follows:

root/
├── GUICourse/
│   └── GUIAct/
│   │    └── imgs
│   ├── Web_test.json
│   └── Mobile_test.json

Run

python utils/eval_utils/eval_guiact.py \
  --pretrained "HongxinLi/UIPro-7B_Stage2_Web" \
  --device_type Web \
  --scale 1000 \
  --root "/path/to/GUICourse_processed" \
  --imgs_dir "/path/to/images_base"

Common options

• --device_type: Web or Mobile
• --root: processed GUIAct directory (overrides auto-detect)
• --imgs_dir: image base directory (overrides auto-detect)
• --debug: evaluate a small random subset
• --cot: enable Chain-of-Thought prompting
• --action_refexp: (only used for UIPro) Please use this to prompt UIPro to output short reasoning before planning the action
• --original_actspace: use the original GUIAct action space

Outputs

• Saved to utils/eval_utils/eval_results/GUIAct-<device_type>/<model_postfix>/<timestamp>.json

What you need

• A local Mind2Web folder that contains:
  • mind2web_images/ (evaluation screenshots)
  • mind2web_data_test_{website|task|domain}.json (test annotations)
• You can pass --root to point to your local Mind2Web directory. If omitted, the script auto-selects the first existing path in DEFAULT_ROOTS inside utils/eval_utils/eval_mind2web.py.

Expected layout

/path/to/Mind2Web/
├── mind2web_images/
└── mind2web_data_test_website.json
    mind2web_data_test_task.json
    mind2web_data_test_domain.json

Run

python utils/eval_utils/eval_mind2web.py \
  --pretrained "HongxinLi/UIPro-7B_Stage2_Web" \
  --task website \
  --scale 1000 \
  --root "/path/to/Mind2Web"

Common options

• --task: either 'task', 'domain', or 'website' (the three splits of Mind2Web)
• --debug: only evaluate a few episodes
• --cot: enable Chain-of-Thought prompting
• --action_refexp: (only used for UIPro) Please use this to prompt UIPro to output short reasoning before planning the action
• --max_prev_acts: how many previous actions are included in history

Outputs

• Saved to utils/eval_utils/eval_results/mind2web/<model_postfix>/{website|task|domain}-<timestamp>.json