Embodied-R1.5-RFT-Dataset

🌐 Project Page  |  📄 arXiv  |  💻 Code  |  🧰 EmbodiedEvalKit  |  🤗 Models & Datasets

🗓️ Update — 2026-08-20 (20260820). All 28 Stage 2 RFT JSON annotation files have been uploaded to rft_datasets_json/. The complete JSON ↔ media archive mapping is documented in the Dataset composition table below.

⚠️ Partial release. This repository currently contains only a subset of the full Stage 2 RFT data used to train Embodied-R1.5. The corresponding model checkpoints, training scripts, and full evaluation suite are available at the project repository. For the Stage 1 SFT data, see the sibling dataset Embodied-R1.5-SFT-Dataset.

📦 ModelScope mirror. Some data files that could not be uploaded to HuggingFace are hosted on ModelScope instead: modelscope.cn/datasets/iffyuan/Embodied-R1.5-RFT-Dataset. Some datasets cannot be open-sourced due to institutional policy.

📁 JSON data location. All RFT training JSON files are stored in rft_datasets_json/. Media archives referenced by these JSON files remain at the repository root or in their corresponding archive subdirectories.

Overview

This dataset is the Stage 2 reinforcement fine-tuning (RFT) corpus for Embodied-R1.5, a unified Embodied Foundation Model (EFM) built on Qwen3-VL-8B-Instruct. It is used after Stage 1 SFT to further sharpen embodied reasoning with a multi-task balanced RL recipe, and spans three core embodied capability dimensions:

• Spatial cognition & reasoning — semantic and spatial structure of the physical world, including static geometric relations and dynamic interaction possibilities.
• Task planning & correction — long-horizon decomposition, next-step planning, process detection, error localization, and correction.
• Embodied pointing & location — referring expression grounding, region-level localization, functional (affordance) grounding, and visual trace generation.

RFT training uses the EasyR1 framework; see the GitHub repository for scripts and configs.

Format

Each sample is a verifiable QA instance with a problem, multimodal references, and metadata used to compute task-specific rewards during RL:

{
  "problem_id": "...",
  "problem": "<image>\n...",
  "images": ["path/to/image.jpg"],
  "data_type": "image",
  "problem_type": "point",
  "options": [],
  "data_source": "CoSyn-Point",
  "answer": "..."
}

• data_type is one of image | video.
• problem_type is one of point | bbox | trace | multiple choice | open-ended.
• Points (point_2d) and boxes are normalized to the [0, 1000] range, regardless of original image resolution.
• For 3D traces, the depth value is in meters.
• Final answers are emitted within <answer>...</answer> tags.

Dataset composition

The corpus is organized as per-task verifiable-QA JSON files under rft_datasets_json/, paired with LFS-tracked tar.gz / zip archives that contain the referenced images and videos. A small number of large archives are split into parts (handal_images/, qa/) for upload size limits.

Additional archive er1.5_test.tar.gz at the repo root contains held-out evaluation samples and is not part of training.

Filename suffix convention

RFT files always carry a modality suffix and may carry an optional cleanup suffix:

• ER1.5_<dataset>_image_qa.json — image-level visual question answering
• ER1.5_<dataset>_image_point.json — image-level point / box grounding
• ER1.5_<dataset>_image_trace.json — image-level visual trace generation
• ER1.5_<dataset>_video_qa.json — video-level visual question answering
• ER1.5_<dataset>_image.json — image-only samples with mixed task types

Optional cleanup suffixes: _filtered.json (post-filtering), _sample3000.json (3k subsample), _50s_cleaned.json (≤50 s clips, cleaned).

To download only the RFT JSON files:

huggingface-cli download IffYuan/Embodied-R1.5-RFT-Dataset \
  --repo-type dataset \
  --include "rft_datasets_json/*.json" \
  --local-dir Embodied-R1.5-RFT-Dataset

Relationship to Stage 1 (SFT)

The Stage 1 SFT corpus (Embodied-R1.5-SFT-Dataset) is the union of (a) 33 open-source-derived ShareGPT files and (b) three automated generation pipelines. Stage 2 RFT narrows this to 28 task-specific verifiable-QA files and additionally includes:

• Self-distillation outputs (ER1.5_ER1-point_image_point.json, ER1.5_ER1-trace_image_trace.json) — ER1 model's predictions used as reward-shaped rollouts.
• RFT-only benchmarks (ERQA2, ERQA_Rush, EmbSpatial, spatialssrl) — not present in Stage 1 SFT.
• RoboCasa / PartNet 3D traces (ER1.5_robocasa_partnet_2d_image_trace.json, ER1.5_robocasa_partnet_3d_image_trace.json) — finer-grained action traces than SFT.

Status

Citation

If you find Embodied-R1.5 useful in your research, please cite our work:

@article{yuan2026embodied,
  title={Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models},
  author={Yuan, Yifu and Huang, Yaoting and Yao, Xianze and Li, Yutong and Zhang, Shuoheng and Han, Linqi and Li, Pengyi and Sun, Jiangeng and Jia, Wenting and Zhang, Zhao and Liu, Yuhao and Liao, Ruihao and Hu, Yucheng and Wu, Qiyu and Li, Yuxiao and Dong, Zibin and Ni, Fei and Zheng, Yan and Gu, Shuyang and Ma, Yi and Tang, Hongyao and Hu, Han and Hao, Jianye},
  journal={arXiv preprint arXiv:2606.11324},
  year={2026}
}

License

Released under the Apache 2.0 license.