GEM: Generative Supervision Helps Embodied Intelligence

Ruowen Zhao¹, Bangguo Li¹, Zuyan Liu^1,2,†, Yinan Liang¹, Junliang Ye¹, Fangfu Liu¹,
Diankun Wu¹, Zhengyi Wang¹, Xumin Yu², Yongming Rao^2,✉, Han Hu², Jun Zhu^1,✉
†Project Lead.^✉Corresponding Author.
¹Tsinghua University, ²Tencent Hunyuan

https://github.com/user-attachments/assets/e978726f-d2ab-46af-8513-eb0a62a94ecc

Overview

Overview of GEM. GEM enhances semantic reasoning and physical grounding with an auxiliary depth-generation objective. Trained on the large-scale embodied data, GEM achieves strong performance across diverse embodied benchmarks. The extending GEM-VLA also attains SOTA results on simulation and real-world robot tasks.

News

• [05/28] 🔥 We release the paper on arXiv!
• [05/28] 🔥 We release the training code and dataset samples GEM-250K.
• [05/28] 🔥 We release the checkpoint of GEM-2B.

TODO

• Release of larger model (GEM-8B).
• Release of full training data.

Installation

1. Clone Repository

git clone https://github.com/zhaorw02/GEM.git 
cd GEM

2. Environment Setup

We use conda to manage the environment. Recommended versions:

• Python 3.10+
• torch>=2.6.0, torchvision, transformers>=4.57.0
• deepspeed, flash-attn, accelerate, peft, triton, torchcodec

conda create -n gem python=3.12 -y
conda activate gem
pip install -r requirements.txt
pip install flash-attn --no-build-isolation

3. Dataset Setup

Please configure the dataset paths in qwen-vl-finetune/qwenvl/data/__init__.py. Specifically, set annotation_path and data_path for GEM-250K, which can be downloaded from Hugging Face: GEM-250K.

You also need to update the depth image path and the depth image loading logic in qwen-vl-finetune/qwenvl/data/utils.py according to your local directory structure.

Model Checkpoints

The pretrained GEM-2B checkpoint is available on Hugging Face: 🤗 GEM-2B.

VLM Training

Set in qwen-vl-finetune/train.sh:

• MODEL_PATH: path to pretrained checkpoints.
• OUTPUT_DIR: where to save checkpoints.

Edit annotation_path and data_path (after downloading from GEM-250K) in qwen-vl-finetune/qwenvl/data/__init__.py.

From the qwen-vl-finetune/ directory:

cd qwen-vl-finetune
# 8 GPUs by default; set NPROC_PER_NODE or CUDA_VISIBLE_DEVICES as needed
bash scripts/train.sh

VLM Inference

Run inference with the following commands:

cd qwen-vl-finetune
python inference.py

VLA Training

Please refer to GEM-VLA for details.

BibTeX

If you find our work helpful, please consider citing:

@article{zhao2026gem,
  title={GEM: Generative Supervision Helps Embodied Intelligence},
  author={Zhao, Ruowen and Li, Bangguo and Liu, Zuyan and Liang, Yinan and Ye, Junliang and Liu, Fangfu and Wu, Diankun and Wang, Zhengyi and Yu, Xumin and Rao, Yongming and others},
  journal={arXiv preprint arXiv:2605.28548},
  year={2026}
}

Acknowledgement

Our code is based on these wonderful repos: Qwen3-VL, Sana, RDT2