Run Grounding

GUI Actor

:rescue_worker_helmet: Installation

1. Clone this repo to your local machine:

git clone https://github.com/strawberryCheeseCake2/CF-Ground.git
cd CF-Ground

2. Create a conda environment:

conda create -n gui_actor python=3.10
conda activate gui_actor

3. Install the dependencies:

• NVIDIA GPU

pip install packaging psutil pyyaml safetensors httpx tqdm requests regex click shellingham
pip install torch==2.5.1 torchvision==0.20.1 --index-url https://download.pytorch.org/whl/cu121
pip install flash-attn==2.7.3 --no-build-isolation
pip install -r env/requirements_gui_actor.txt

:minidisc: ScreenSpot_v2 Dataset Download

cd data
wget https://huggingface.co/datasets/OS-Copilot/ScreenSpot-v2/resolve/main/screenspot_mobile_v2.json
wget https://huggingface.co/datasets/OS-Copilot/ScreenSpot-v2/resolve/main/screenspot_web_v2.json
wget https://huggingface.co/datasets/OS-Copilot/ScreenSpot-v2/resolve/main/screenspot_desktop_v2.json
wget https://huggingface.co/datasets/OS-Copilot/ScreenSpot-v2/resolve/main/screenspotv2_image.zip
unzip screenspotv2_image.zip
rm screenspotv2_image.zip
cd ..

Run

GUI-Actor 3B

cd src
python gui_actor_3B.py [gpu]

Qwen2.5-VL + Text Grounding (with Attention Fallback)

cd src
python qwen25vl+text.py [gpu]

Command Line Arguments

For gui_actor_3B.py:

• gpu (int, default: 0): Specifies the GPU number to use for computation.
• --r (float, default: 0.50): Sets the resize ratio for Stage 1 processing.
• --th (float, default: 0.12): Defines the crop threshold for Stage 1 connected region detection.
• --p (int, default: 0): Determines the crop padding for Stage 1 in pixels.
• --e (float, default: 0.7): Ensemble ratio for Stage 1 attention (0~1).
• --v (flag): If set, saves visualization images during processing.
• --mac (flag): If set, runs on Mac with MPS backend.

For qwen25vl+text.py:

• gpu (int, default: 0): Specifies the GPU number to use for computation.
• --r (float, default: 0.50): Sets the resize ratio for Stage 1 processing.
• --th (float, default: 0.11): Defines the crop threshold for Stage 1 connected region detection.
• --p (int, default: 20): Determines the crop padding for Stage 1 in pixels.
• --v (flag): If set, saves visualization images during processing.
• --mac (flag): If set, runs on Mac with MPS backend.

Usage Examples:

Qwen2.5-VL with text grounding and attention fallback:

cd src
python qwen25vl+text.py 0 --r 0.5 --th 0.11 --p 20 --v

GUI-Actor 3B:

cd src
python gui_actor_3B.py 0 --r 0.5 --th 0.12 --p 0 --e 0.7 --v