Embodied-R1.5-SFT-Dataset

🌐 Project Page  |  📄 arXiv  |  💻 Code  |  🧰 EmbodiedEvalKit  |  🤗 Models & Datasets

🗓️ Update — 2026-08-20 (20260820). All 34 Stage 1 SFT JSON annotation files have been uploaded to sft_datasets_json/. The complete JSON ↔ image/video data mapping is documented in the Dataset composition table below.

⚠️ Partial release. This repository currently contains only a subset of the full Stage 1 SFT data used to train Embodied-R1.5. The corresponding model checkpoints, training scripts, and full evaluation suite are available at the project repository.

📁 JSON file location. All SFT JSON files are stored in sft_datasets_json/. They are not placed inside the individual image/video data folders.

📦 ModelScope mirror. Some data files that could not be uploaded to HuggingFace are hosted on ModelScope instead: modelscope.cn/datasets/iffyuan/Embodied-R1.5-SFT-Dataset. Some datasets cannot be open-sourced due to institutional policy.

This dataset is the Stage 1 supervised fine-tuning (SFT) corpus for Embodied-R1.5, a unified Embodied Foundation Model (EFM) built on Qwen3-VL-8B-Instruct. The full corpus exceeds 15B tokens and spans three core embodied capability dimensions:

• Spatial cognition & reasoning — semantic and spatial structure of the physical world, including static geometric relations and dynamic interaction possibilities.
• Task planning & correction — long-horizon decomposition, next-step planning, process detection, error localization, and correction.
• Embodied pointing & location — referring expression grounding, region-level localization, functional (affordance) grounding, and visual trace generation.

The data is built by integrating and restructuring open-source resources together with three automated data construction pipelines that target critical capability gaps.

Format

Samples follow the ShareGPT conversation format, with multimodal references to images and videos:

{
  "conversations": [
    {"from": "human", "value": "<image>\n..."},
    {"from": "gpt", "value": "...<answer>...</answer>"}
  ],
  "image": ["path/to/image.jpg"]
}

• Points (point_2d) and boxes are normalized to the [0, 1000] range, regardless of original image resolution.
• For 3D traces, the depth value is in meters.
• Final answers are emitted within <answer>...</answer> tags.

Dataset composition

All ShareGPT-format annotation files are centralized in sft_datasets_json/. The per-dataset folders contain the image/video data referenced by the JSON image / video fields.

Filename suffix convention

• ER1.5_<dataset>.json — direct use of open-source data without modification
• ER1.5_<dataset>_star.json — paper major modification or newly generated data
• ER1.5_<dataset>_trace_star.json — trajectory data with paper major modification

Status

Citation

If you find Embodied-R1.5 useful in your research, please cite our work:

@article{yuan2026embodied,
  title={Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models},
  author={Yuan, Yifu and Huang, Yaoting and Yao, Xianze and Li, Yutong and Zhang, Shuoheng and Han, Linqi and Li, Pengyi and Sun, Jiangeng and Jia, Wenting and Zhang, Zhao and others},
  journal={arXiv preprint arXiv:2606.11324},
  year={2026}
}

License

Released under the Apache 2.0 license.