⚙️ Install Dependencies

Python

The Qwen3-VL environment is tested with Python 3.11.

Pip

python -m pip install -r requirements.txt

Note: We intentionally do not include flash-attn in requirements.txt.
Please download the wheel that matches your local PyTorch and CUDA versions from:
https://github.com/Dao-AILab/flash-attention/releases
and then install it with pip in your environment. This is usually faster and more stable than building from source during dependency installation.

🚀 Core Contributions in this repo

This repository supports two complementary directions for efficient GUI-agent inference: cross-step reuse of ViT features and training-free pruning of historical visual tokens within the LLM.

1) Cross-step ViT feature caching and reuse

During GUI navigation, the same screenshot is repeatedly processed by the model: it first appears as the current observation and then reappears as a historical observation in subsequent steps. Since the vision encoder processes each image independently, these repeated inputs produce identical ViT features.

We provide an episode-local cache-and-reuse mechanism that stores the ViT features of each screenshot after its first encoding and directly reuses them in subsequent steps. This avoids redundant vision-encoder computation while preserving the complete historical visual information. For Qwen3-VL, the cache also stores the intermediate ViT features required by DeepStack.

This finding motivates the pruning location adopted in this repository: historical visual-token pruning is performed within the subsequent LLM layers, rather than before or within the ViT. The feature cache can be enabled in the evaluation scripts with --visual_cache_enable.

2) Evaluation scripts with training-free pruning controls

The following evaluation scripts support pruning historical visual tokens:

• AITW_eval.py
• AndroidControl_eval.py
• Mind2Web_eval.py
• Odyssey_eval.py

These scripts provide CLI switches and runtime hooks for multiple training-free pruning strategies on historical screenshots, including:

• FastV (ECCV 2024)
• SparseVLM (ICML 2025)
• DART (EMNLP 2025)
• PDrop (CVPR 2025)
• DivPrune (CVPR 2025)
• Sobel-based pruning (retain foreground or background tokens)
• Random pruning

Several strategies support separate keep ratios for individual historical screenshots, ordered from the most recent observation to older observations. This makes it possible to allocate more tokens to recent visual context and fewer tokens to distant history.

To avoid conflicting pruning paths in a single run, enable only one pruning method at a time during evaluation.

3) Model-side LLM token pruning implementation

The historical visual-token pruning logic is implemented within the model internals:

• src/Qwen2_5/model_file/LLM_compression_v2_5_action/modeling_qwen2_5_vl.py
• src/Qwen3/model_file/LLM_compression_v3_action/modeling_qwen3_vl.py

These are the primary files to modify when extending an existing pruning method or implementing a new historical visual-token pruning strategy.

🎯 SFT training scripts

This repo provides supervised fine-tuning (SFT) scripts for all four datasets:

• scripts/finetune_aitw.sh
• scripts/finetune_androidcontrol.sh
• scripts/finetune_mind2web.sh
• scripts/finetune_odyssey.sh

You can launch SFT by running the corresponding script, for example:

bash scripts/finetune_aitw.sh
bash scripts/finetune_androidcontrol.sh
bash scripts/finetune_mind2web.sh
bash scripts/finetune_odyssey.sh

Note: Please update paths (e.g., MODEL_NAME, data_path, output_dir, environment-specific deepspeed/python paths) before running on your machine.

🤗 Hugging Face released SFT checkpoints

We release SFT checkpoints based on Qwen2.5-VL and Qwen3-VL on Hugging Face:

HistPrune-GUI-Checkpoints

The model repository provides the checkpoint-to-dataset mapping and additional usage details. These checkpoints can be used directly for training-free pruning evaluation without additional fine-tuning.

📦 Download datasets

You need to download and prepare the AITW, Mind2Web, AndroidControl, and Odyssey datasets on your own. The preprocessing scripts in data/ can be used to generate the local dataset JSON files; update their image paths for your local dataset layout before training or evaluation.

• AITW & Mind2Web: https://github.com/njucckevin/SeeClick/blob/main/agent_tasks/readme_agent.md

• AndroidControl: https://github.com/google-research/google-research/tree/master/android_control

• Odyssey: https://huggingface.co/datasets/OpenGVLab/GUI-Odyssey

🙏 Acknowledgement

We are deeply grateful to the EPIC Lab for the excellent insights provided in their paper "Token Pruning in Multimodal Large Language Models: Are We Solving the Right Problem?", which has greatly inspired this work.

We thank SimpAgent for providing the foundational codebase for our work.