MobileForge Benchmark Results

This dataset contains the evaluation artifacts used by MobileForge: Annotation-Free Adaptation for Mobile GUI Agents with Hierarchical Feedback-Guided Policy Optimization.

It includes AndroidWorld and MobileWorld GUI-only evaluation runs for the base agents and their MobileForge-adapted variants. The repository is intended for result verification, log inspection, and mapping the public model checkpoints to the exact benchmark artifacts reported in the paper.

What is included?

• archives/androidworld/*.tar.zst: AndroidWorld rollout artifacts for Qwen3-VL-8B, GUI-Owl-1.5-8B, ForgeQwen3-8B, and ForgeOwl-8B.
• archives/mobileworld/*.tar.zst: MobileWorld GUI-only evaluation logs for the base and MobileForge-adapted models.
• metadata/model_result_map.json: machine-readable mapping from public archive names to checkpoints, original sessions, artifact sizes, and headline results.
• metadata/archives_manifest.json: compact manifest of all archive files and sizes.

The Hugging Face Dataset Viewer is disabled because the repository mainly stores compressed rollout logs rather than row-oriented examples.

AndroidWorld results

AndroidWorld uses 116 tasks and reports Pass@1, Pass@2, and Pass@3.

MobileWorld GUI-only results

MobileWorld GUI-only uses 117 tasks. No MobileWorld rollout, task, or feedback is used for MobileForge adaptation.

Directory layout

benchmark-results/
  metadata/
    archives_manifest.json
    model_result_map.json
  archives/
    androidworld/
      qwen3-vl-base.tar.zst
      qwen3-vl-mobileforge-200tasks.tar.zst
      qwen3-vl-mobileforge-400tasks.tar.zst
      qwen3-vl-mobileforge-900tasks.tar.zst
      gui-owl-base.tar.zst
      gui-owl-mobileforge-200tasks.tar.zst
      gui-owl-mobileforge-400tasks.tar.zst
      gui-owl-mobileforge-900tasks.tar.zst
    mobileworld/
      qwen3-vl-base.tar.zst
      qwen3-vl-mobileforge-900tasks.tar.zst
      gui-owl-base.tar.zst
      gui-owl-mobileforge-900tasks.tar.zst

Each archive expands to one experiment folder. AndroidWorld archives contain per-task compressed rollout artifacts (*.pkl.gz) and a run-level *_results.json summary. MobileWorld archives contain the corresponding MobileWorld evaluation logs and screenshots.

Download examples

Download only the metadata:

from huggingface_hub import hf_hub_download

path = hf_hub_download(
    repo_id="lgy0404/mobileforge-benchmark-results",
    repo_type="dataset",
    filename="metadata/model_result_map.json",
)

Download one AndroidWorld run:

from huggingface_hub import snapshot_download

local_dir = snapshot_download(
    repo_id="lgy0404/mobileforge-benchmark-results",
    repo_type="dataset",
    allow_patterns=[
        "archives/androidworld/qwen3-vl-mobileforge-900tasks.tar.zst",
        "metadata/model_result_map.json",
        "metadata/archives_manifest.json",
    ],
)

Extract an archive:

tar --use-compress-program=unzstd -xf archives/androidworld/qwen3-vl-mobileforge-900tasks.tar.zst

Notes

• AndroidWorld summaries can be parsed with the AndroidWorld checkpoint parser released in the MobileForge codebase.
• MobileWorld logs follow the upstream MobileWorld evaluation format.
• Compressed pickle files should be treated as executable Python serialization. Load them only in a trusted research environment.

Citation

@article{liu2026mobileforge,
  title={MobileForge: Annotation-Free Adaptation for Mobile GUI Agents with Hierarchical Feedback-Guided Policy Optimization},
  author={Liu, Guangyi and Zhao, Pengxiang and Wu, Gao and Yin, Yiwen and Li, Mading and Liu, Liang and Liu, Congxiao and Qi, Zhang and Wang, Mengyan and Guo, Liang and Liu, Yong},
  journal={arXiv preprint arXiv:2606.19930},
  year={2026}
}