🔥 Data Pyramid for Embodied Manipulation: A Survey
See the codeREADME
Awesome Embodied Data Pyramid
This repository accompanies a survey on the data pyramid for robotics and embodied AI. It curates the datasets, data-collection paradigms, simulators, and data-hungry model families reviewed in the survey — spanning real-robot data, UMI (in-the-wild) data, egocentric / ego-exo human data, simulation data, and broad general (web-scale) data — together with the VLA / world-action / embodied-VLM models that consume them. Each entry links to its paper, project page, and code where available.
News 🔥🔥🔥
This project is actively maintained and continuously updated to make embodied data resources easier for the open-source community to discover, access, and download.
- 2026-08-22: Added 20 resources from a July–August 2026 arXiv sweep, spanning contact-rich and industrial real-robot datasets (PRISM, SoftVTBench, RoboInter-Data), EgoData Engineering (7 new human-to-robot conversion works), and visuo-tactile world models used as simulators (ViTacWorld, N0-TWAM, OVTF).
- 2026-08-10: Added N0-Foundation and its OpenNeoData release to the real-robot data collection.
- 2026-08-07: Updated the data accumulation figure; introduced the EgoData Engineering collection with Ego2Robot and EgoEngine; and added the CS2-10k and RekaDaily-10k datasets from Reka AI.
- 2026-07-27: Open-sourced the project website and this GitHub project. You are welcome to use the curated links throughout this repository to access and download the collected resources. Issues and pull requests are encouraged to help keep these resources accessible and useful.
We sincerely thank all open-source dataset creators and contributors for their valuable work and continued support of the community. You can also use task-based search on the project website to find and download datasets.
If you have suggestions for new resources, improvements to methodologies, or corrections for broken links, please don't hesitate to open an issue or submit a pull request. Contributions of all kinds are welcome and greatly appreciated.
Table of Contents
Data Sources
Real-Robot Data
Egocentric & Ego-Exo Data
Simulation Data
Embodied Foundation Models
Repository
The Data Pyramid: Data Sources
Real-Robot Data
Table: Recent real-robot datasets
Table: Real-robot datasets (2024 and earlier)
UMI Data
Table: UMI-style robot-free and cross-embodiment datasets
Egocentric & Ego-Exo Data
Table: EgoData Engineering
Table: Recent egocentric and ego-exo datasets
Table: Egocentric and ego-exo datasets (2024 and earlier)
Simulation Data
Curation note. For peer-reviewed work,
YearandPublication / Typeuse the formal publication year and venue. For arXiv-only work,Yearis the first public preprint year. Software without a canonical paper is labeledSoftware. The sections below distinguish reusable 3D/scene assets, simulation backends, evaluation benchmarks, and both downloadable synthetic datasets and systems that generate such data.
3D Assets & Scene Datasets
Table: 3D assets and scene datasets
Simulation & Rendering Backends
Table: Simulation and rendering backends
Benchmarks
Table: Simulation benchmarks
Simulation/Synthetic Datasets & Data-Generation Systems
Table: Simulation/synthetic datasets and data-generation systems
World Models as Simulators
Table: World models as simulators
General Data
Visual Data
Table: Visual data
Segmentation & Localization
Table: Segmentation and localization data
Video & Temporal Data
Table: Video and temporal data
Planning / Task Decomposition
Table: Planning and task decomposition data
| Year | Acronym | Paper | Venue | Project | Repo@GitHub |
|---|---|---|---|---|---|
| 2023 | EgoPlan-IT | EgoPlan-Bench: Benchmarking Multimodal Large Language Models for Human-Level Planning | IJCV |  |  |
| 2023 | EmbodiedGPT | EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought | NeurIPS | | |
| 2023 | LLaRP | Large Language Models as Generalizable Policies for Embodied Tasks | ICLR | | |
| 2022 | EgoTaskQA | EgoTaskQA: Understanding Human Tasks in Egocentric Videos | NeurIPS | | |
| 2025 | WAP | World-aware Planning Narratives Enhance Large Vision-Language Model Planner | NeurIPS | | |
Grasp Data
Table: Grasp data
| Year | Acronym | Paper | Venue | Project | Repo@GitHub |
|---|---|---|---|---|---|
| 2026 | MapleGrasp | MapleGrasp: Mask-guided Feature Pooling for Language-driven Efficient Robotic Grasping | WACV | | |
| 2024 | Language-Driven 6-DoF Grasp Detection Using Negative Prompt Guidance | ECCV | | | |
| 2024 | DexGraspNet 2.0 | DexGraspNet 2.0: Learning Generative Dexterous Grasping in Large-scale Synthetic Cluttered Scenes | CoRL | | |
| 2024 | Grasp-Anything | Grasp-Anything: Large-scale Grasp Dataset from Foundation Models | ICRA | | |
| 2021 | SuctionNet-1Billion | SuctionNet-1Billion: A Large-Scale Benchmark for Suction Grasping | RAL | | |
| 2018 | Jacquard | Jacquard: A Large Scale Dataset for Robotic Grasp Detection | IROS | | |
| 2020 | GraspNet-1Billion | GraspNet-1Billion: A Large-Scale Benchmark for General Object Grasping | CVPR | | |
Embodied Foundation Models
Representative VLA and WAM Methods
Representative VLA and WAM methods by release time, model type, institution, and data source.
Data sources: 
Table: Representative VLA and WAM methods (74 entries)
| Time | Method | Institution | Project | Model | Data |
|---|---|---|---|---|---|
| 2023.3 | PaLM-E | | VLA | | |
| 2023.7 | RT-2 | DeepMind | | VLA | |
| 2024.5 | Octo | UC Berkeley | | VLA | |
| 2024.6 | OpenVLA | Stanford University | | VLA | |
| 2024.10 | GR-2 | ByteDance | | VLA | |
| 2024.10 | π0 | Physical Intelligence | | VLA | |
| 2024.10 | RDT-1B | THU | | VLA | |
| 2024.11 | CogACT | THU | | VLA | |
| 2025.1 | SpatialVLA | Shanghai AI Lab | | VLA | |
| 2025.1 | UP-VLA | THU | | VLA | |
| 2025.3 | HybridVLA | PKU | | VLA | |
| 2025.3 | GR00T N1 | NVIDIA | | VLA | |
| 2025.3 | GO-1 | Shanghai AI Lab & AgiBot | | VLA | |
| 2025.3 | CoT-VLA | NVIDIA | | VLA | |
| 2025.4 | UWM | University of Washington | | WAM | |
| 2025.4 | π0.5 | Physical Intelligence | | VLA | |
| 2025.5 | UniVLA | HKU | | VLA | |
| 2025.6 | SmolVLA | Hugging Face | | VLA | |
| 2025.6 | GR00T N1.5 | NVIDIA | | VLA | |
| 2025.7 | DreamVLA | SJTU | | VLA | |
| 2025.7 | EgoVLA | UC San Diego | | VLA | |
| 2025.7 | Being-H0 | PKU | | VLA | |
| 2025.7 | GR-3 | ByteDance Seed | | VLA | |
| 2025.7 | H-RDT | THU | | VLA | |
| 2025.8 | GalaxeaVLA(G0) | Galaxea | | VLA | |
| 2025.8 | ReconVLA | HKUST | | VLA | |
| 2025.8 | MolmoAct | AI2 & UW | | VLA | |
| 2025.8 | EO-1 | Shanghai AI Lab | | VLA | |
| 2025.9 | RynnVLA-001 | DAMO | | VLA | |
| 2025.10 | X-VLA | THU | | VLA | |
| 2025.10 | InternVLA-M1 | Shanghai AI Lab | | VLA | |
| 2025.10 | VITRA | THU | | VLA | |
| 2025.11 | METIS | PKU | | VLA | |
| 2025.11 | iFlyBot-VLA | iFLYTEK & LindenBot | | VLA | |
| 2025.12 | VideoVLA | XJTU | | VLA | |
| 2025.12 | Motus | THU | | WAM | |
| 2025.12 | GR00T N1.6 | NVIDIA | | VLA | |
| 2026.1 | InternVLA-A1 | Shanghai AI Lab | | VLA | |
| 2026.1 | LaST_0 | PKU | | VLA | |
| 2026.1 | Being-H0.5 | BeingBeyond | | VLA | |
| 2026.1 | LingbotVLA | Ant Group | | VLA | |
| 2026.1 | LingbotVA | Ant Group | | WAM | |
| 2026.2 | RDT2 | THU | | VLA | |
| 2026.2 | Xiaomi-Robotics-0 | Xiaomi | | VLA | |
| 2026.2 | ABot-M0 | AMAP CV Lab | | VLA | |
| 2026.2 | DM0 | Dexmal | | VLA | |
| 2026.2 | LDA-1B | PKU | | WAM | |
| 2026.2 | DreamZero | NVIDIA | | WAM | |
| 2026.2 | EgoScale | NVIDIA | | VLA | |
| 2026.3 | GigaWorld-Policy | GigaAI | | WAM | |
| 2026.3 | MolmoB0T | AI2 | | VLA | |
| 2026.3 | UniDex | THU | | VLA | |
| 2026.4 | JoyAI-RA 0.1 | JD | | VLA | |
| 2026.4 | π0.7 | Physical Intelligence | | VLA | |
| 2026.4 | GR00T N1.7 | NVIDIA | | VLA | |
| 2026.4 | Being-H0.7 | BeingBeyond | | WAM | |
| 2026.4 | MotuBrain | Shengshu | | WAM | |
| 2026.5 | Qwen-VLA | Qwen | | VLA | |
| 2026.5 | Wall-OSS-0.5 | X Square Robot | | VLA | |
| 2026.5 | MolmoAct2 | UW & AI2 | | VLA | |
| 2026.6 | Wall-WM | X Square Robot | | WAM | |
| 2026.6 | LaST-HD | PKU | | VLA | |
| 2026.6 | Hy-Embodied-0.5-VLA | Tencent Robotics X | | VLA | |
| 2026.6 | Kairos | ACE Robotics | | WAM | |
| 2026.6 | Qwen-RobotManip | Qwen | | VLA | |
| 2026.7 | ABot-M0.5 | AMAP CV Lab | | WAM | |
| 2026.7 | ACE-Brain-0.5 | ACE-Robotics | | VLA | |
| 2026.7 | InternVLA-A1.5 | Shanghai AI Lab | | VLA | |
| 2026.7 | LingbotVLA 2.0 | Ant Group | | VLA | |
| 2026.7 | LingbotVA 2.0 | Ant Group | | WAM | |
| 2026.7 | HumanScale | PKU | | WAM | |
| 2026.7 | GigaWorld-Policy-0.5 | GigaAI | | WAM | |
| 2026.7 | Xiaomi-Robotics-1 | Xiaomi | | VLA | |
| 2026.7 | Xiaomi-Robotics-U0 | Xiaomi Robotics | | WAM | |
Contributing
Contributions are welcome! Please open an issue or pull request to add resources, fix links, update metadata, or suggest improvements. For new resources, please include the paper title, project page, code link if available, release date, scale, and the most relevant data category.
Acknowledgement
If you find this repository useful, please consider citing our paper:
@misc{ye2026datapyramid,
title={{Data Pyramid for Embodied Manipulation}},
author={Yifan Ye and Yankai Fu and Yaoxu Lv and Bohan Hou and Jun Cen and Lingdong Kong and Duo Zheng and Tianxing Chen and Jiaming Liu and Ziang Cao and Yunfan Lou and Wei Chow and Xian Sun and Yingshuo Wang and Kuangzhi Ge and Xiaowei Chi and Xidong Zhang and Zhibo Pang and Yiwu Zhong and Sirui Han and Zhihe Lu and Weihao Yuan and Qifeng Chen and Michael Yu Wang and Yao Mu and Ziwei Liu and Jianfei Yang and Ping Luo and Shanghang Zhang},
year={2026},
eprint={2607.24744},
archivePrefix={arXiv},
primaryClass={cs.RO},
url={https://arxiv.org/abs/2607.24744}
}
Contributors
Similar repos
Languages
HTML
100.0%
🔥 Data Pyramid for Embodied Manipulation: A Survey
See the codeREADME
Awesome Embodied Data Pyramid
This repository accompanies a survey on the data pyramid for robotics and embodied AI. It curates the datasets, data-collection paradigms, simulators, and data-hungry model families reviewed in the survey — spanning real-robot data, UMI (in-the-wild) data, egocentric / ego-exo human data, simulation data, and broad general (web-scale) data — together with the VLA / world-action / embodied-VLM models that consume them. Each entry links to its paper, project page, and code where available.
News 🔥🔥🔥
This project is actively maintained and continuously updated to make embodied data resources easier for the open-source community to discover, access, and download.
- 2026-08-22: Added 20 resources from a July–August 2026 arXiv sweep, spanning contact-rich and industrial real-robot datasets (PRISM, SoftVTBench, RoboInter-Data), EgoData Engineering (7 new human-to-robot conversion works), and visuo-tactile world models used as simulators (ViTacWorld, N0-TWAM, OVTF).
- 2026-08-10: Added N0-Foundation and its OpenNeoData release to the real-robot data collection.
- 2026-08-07: Updated the data accumulation figure; introduced the EgoData Engineering collection with Ego2Robot and EgoEngine; and added the CS2-10k and RekaDaily-10k datasets from Reka AI.
- 2026-07-27: Open-sourced the project website and this GitHub project. You are welcome to use the curated links throughout this repository to access and download the collected resources. Issues and pull requests are encouraged to help keep these resources accessible and useful.
We sincerely thank all open-source dataset creators and contributors for their valuable work and continued support of the community. You can also use task-based search on the project website to find and download datasets.
If you have suggestions for new resources, improvements to methodologies, or corrections for broken links, please don't hesitate to open an issue or submit a pull request. Contributions of all kinds are welcome and greatly appreciated.
Table of Contents
Data Sources
Real-Robot Data
Egocentric & Ego-Exo Data
Simulation Data
Embodied Foundation Models
Repository
The Data Pyramid: Data Sources
Real-Robot Data
Table: Recent real-robot datasets
Table: Real-robot datasets (2024 and earlier)
UMI Data
Table: UMI-style robot-free and cross-embodiment datasets
Egocentric & Ego-Exo Data
Table: EgoData Engineering
Table: Recent egocentric and ego-exo datasets
Table: Egocentric and ego-exo datasets (2024 and earlier)
Simulation Data
Curation note. For peer-reviewed work,
YearandPublication / Typeuse the formal publication year and venue. For arXiv-only work,Yearis the first public preprint year. Software without a canonical paper is labeledSoftware. The sections below distinguish reusable 3D/scene assets, simulation backends, evaluation benchmarks, and both downloadable synthetic datasets and systems that generate such data.
3D Assets & Scene Datasets
Table: 3D assets and scene datasets
Simulation & Rendering Backends
Table: Simulation and rendering backends
Benchmarks
Table: Simulation benchmarks
Simulation/Synthetic Datasets & Data-Generation Systems
Table: Simulation/synthetic datasets and data-generation systems
World Models as Simulators
Table: World models as simulators
General Data
Visual Data
Table: Visual data
Segmentation & Localization
Table: Segmentation and localization data
Video & Temporal Data
Table: Video and temporal data
Planning / Task Decomposition
Table: Planning and task decomposition data
| Year | Acronym | Paper | Venue | Project | Repo@GitHub |
|---|---|---|---|---|---|
| 2023 | EgoPlan-IT | EgoPlan-Bench: Benchmarking Multimodal Large Language Models for Human-Level Planning | IJCV | | |
| 2023 | EmbodiedGPT | EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought | NeurIPS | | |
| 2023 | LLaRP | Large Language Models as Generalizable Policies for Embodied Tasks | ICLR | | |
| 2022 | EgoTaskQA | EgoTaskQA: Understanding Human Tasks in Egocentric Videos | NeurIPS | | |
| 2025 | WAP | World-aware Planning Narratives Enhance Large Vision-Language Model Planner | NeurIPS | | |
Grasp Data
Table: Grasp data
| Year | Acronym | Paper | Venue | Project | Repo@GitHub |
|---|---|---|---|---|---|
| 2026 | MapleGrasp | MapleGrasp: Mask-guided Feature Pooling for Language-driven Efficient Robotic Grasping | WACV | | |
| 2024 | Language-Driven 6-DoF Grasp Detection Using Negative Prompt Guidance | ECCV | | | |
| 2024 | DexGraspNet 2.0 | DexGraspNet 2.0: Learning Generative Dexterous Grasping in Large-scale Synthetic Cluttered Scenes | CoRL | | |
| 2024 | Grasp-Anything | Grasp-Anything: Large-scale Grasp Dataset from Foundation Models | ICRA | | |
| 2021 | SuctionNet-1Billion | SuctionNet-1Billion: A Large-Scale Benchmark for Suction Grasping | RAL | | |
| 2018 | Jacquard | Jacquard: A Large Scale Dataset for Robotic Grasp Detection | IROS | | |
| 2020 | GraspNet-1Billion | GraspNet-1Billion: A Large-Scale Benchmark for General Object Grasping | CVPR | | |
Embodied Foundation Models
Representative VLA and WAM Methods
Representative VLA and WAM methods by release time, model type, institution, and data source.
Data sources:
Table: Representative VLA and WAM methods (74 entries)
| Time | Method | Institution | Project | Model | Data |
|---|---|---|---|---|---|
| 2023.3 | PaLM-E | | VLA | | |
| 2023.7 | RT-2 | DeepMind | | VLA | |
| 2024.5 | Octo | UC Berkeley | | VLA | |
| 2024.6 | OpenVLA | Stanford University | | VLA | |
| 2024.10 | GR-2 | ByteDance | | VLA | |
| 2024.10 | π0 | Physical Intelligence | | VLA | |
| 2024.10 | RDT-1B | THU | | VLA | |
| 2024.11 | CogACT | THU | | VLA | |
| 2025.1 | SpatialVLA | Shanghai AI Lab | | VLA | |
| 2025.1 | UP-VLA | THU | | VLA | |
| 2025.3 | HybridVLA | PKU | | VLA | |
| 2025.3 | GR00T N1 | NVIDIA | | VLA | |
| 2025.3 | GO-1 | Shanghai AI Lab & AgiBot | | VLA | |
| 2025.3 | CoT-VLA | NVIDIA | | VLA | |
| 2025.4 | UWM | University of Washington | | WAM | |
| 2025.4 | π0.5 | Physical Intelligence | | VLA | |
| 2025.5 | UniVLA | HKU | | VLA | |
| 2025.6 | SmolVLA | Hugging Face | | VLA | |
| 2025.6 | GR00T N1.5 | NVIDIA | | VLA | |
| 2025.7 | DreamVLA | SJTU | | VLA | |
| 2025.7 | EgoVLA | UC San Diego | | VLA | |
| 2025.7 | Being-H0 | PKU | | VLA | |
| 2025.7 | GR-3 | ByteDance Seed | | VLA | |
| 2025.7 | H-RDT | THU | | VLA | |
| 2025.8 | GalaxeaVLA(G0) | Galaxea | | VLA | |
| 2025.8 | ReconVLA | HKUST | | VLA | |
| 2025.8 | MolmoAct | AI2 & UW | | VLA | |
| 2025.8 | EO-1 | Shanghai AI Lab | | VLA | |
| 2025.9 | RynnVLA-001 | DAMO | | VLA | |
| 2025.10 | X-VLA | THU | | VLA | |
| 2025.10 | InternVLA-M1 | Shanghai AI Lab | | VLA | |
| 2025.10 | VITRA | THU | | VLA | |
| 2025.11 | METIS | PKU | | VLA | |
| 2025.11 | iFlyBot-VLA | iFLYTEK & LindenBot | | VLA | |
| 2025.12 | VideoVLA | XJTU | | VLA | |
| 2025.12 | Motus | THU | | WAM | |
| 2025.12 | GR00T N1.6 | NVIDIA | | VLA | |
| 2026.1 | InternVLA-A1 | Shanghai AI Lab | | VLA | |
| 2026.1 | LaST_0 | PKU | | VLA | |
| 2026.1 | Being-H0.5 | BeingBeyond | | VLA | |
| 2026.1 | LingbotVLA | Ant Group | | VLA | |
| 2026.1 | LingbotVA | Ant Group | | WAM | |
| 2026.2 | RDT2 | THU | | VLA | |
| 2026.2 | Xiaomi-Robotics-0 | Xiaomi | | VLA | |
| 2026.2 | ABot-M0 | AMAP CV Lab | | VLA | |
| 2026.2 | DM0 | Dexmal | | VLA | |
| 2026.2 | LDA-1B | PKU | | WAM | |
| 2026.2 | DreamZero | NVIDIA | | WAM | |
| 2026.2 | EgoScale | NVIDIA | | VLA | |
| 2026.3 | GigaWorld-Policy | GigaAI | | WAM | |
| 2026.3 | MolmoB0T | AI2 | | VLA | |
| 2026.3 | UniDex | THU | | VLA | |
| 2026.4 | JoyAI-RA 0.1 | JD | | VLA | |
| 2026.4 | π0.7 | Physical Intelligence | | VLA | |
| 2026.4 | GR00T N1.7 | NVIDIA | | VLA | |
| 2026.4 | Being-H0.7 | BeingBeyond | | WAM | |
| 2026.4 | MotuBrain | Shengshu | | WAM | |
| 2026.5 | Qwen-VLA | Qwen | | VLA | |
| 2026.5 | Wall-OSS-0.5 | X Square Robot | | VLA | |
| 2026.5 | MolmoAct2 | UW & AI2 | | VLA | |
| 2026.6 | Wall-WM | X Square Robot | | WAM | |
| 2026.6 | LaST-HD | PKU | | VLA | |
| 2026.6 | Hy-Embodied-0.5-VLA | Tencent Robotics X | | VLA | |
| 2026.6 | Kairos | ACE Robotics | | WAM | |
| 2026.6 | Qwen-RobotManip | Qwen | | VLA | |
| 2026.7 | ABot-M0.5 | AMAP CV Lab | | WAM | |
| 2026.7 | ACE-Brain-0.5 | ACE-Robotics | | VLA | |
| 2026.7 | InternVLA-A1.5 | Shanghai AI Lab | | VLA | |
| 2026.7 | LingbotVLA 2.0 | Ant Group | | VLA | |
| 2026.7 | LingbotVA 2.0 | Ant Group | | WAM | |
| 2026.7 | HumanScale | PKU | | WAM | |
| 2026.7 | GigaWorld-Policy-0.5 | GigaAI | | WAM | |
| 2026.7 | Xiaomi-Robotics-1 | Xiaomi | | VLA | |
| 2026.7 | Xiaomi-Robotics-U0 | Xiaomi Robotics | | WAM | |
Contributing
Contributions are welcome! Please open an issue or pull request to add resources, fix links, update metadata, or suggest improvements. For new resources, please include the paper title, project page, code link if available, release date, scale, and the most relevant data category.
Acknowledgement
If you find this repository useful, please consider citing our paper:
@misc{ye2026datapyramid,
title={{Data Pyramid for Embodied Manipulation}},
author={Yifan Ye and Yankai Fu and Yaoxu Lv and Bohan Hou and Jun Cen and Lingdong Kong and Duo Zheng and Tianxing Chen and Jiaming Liu and Ziang Cao and Yunfan Lou and Wei Chow and Xian Sun and Yingshuo Wang and Kuangzhi Ge and Xiaowei Chi and Xidong Zhang and Zhibo Pang and Yiwu Zhong and Sirui Han and Zhihe Lu and Weihao Yuan and Qifeng Chen and Michael Yu Wang and Yao Mu and Ziwei Liu and Jianfei Yang and Ping Luo and Shanghang Zhang},
year={2026},
eprint={2607.24744},
archivePrefix={arXiv},
primaryClass={cs.RO},
url={https://arxiv.org/abs/2607.24744}
}
Similar repos
Contributors
Languages
HTML
100.0%