Open-AoE

2,000 Hours of Smartphone-Collected Egocentric Manipulation Data
with a Complete Data-to-Model Toolchain

English | 简体中文

[!TIP] New to Open-AoE? Start with the data specification, render one segment with AoE-Visualization, then choose a model recipe from AoE-Training-Ready. Use AiWork to start your experiment.

Open-AoE is a large-scale, real-world egocentric manipulation dataset collected entirely with consumer smartphones. It provides approximately 2,000 hours of first-person video with synchronized hand motion, camera motion, and bilingual atomic-action annotations. The repository connects those signals to visualization, human-to-robot retargeting, robot replay, and model-specific training recipes.

Get the Release

The technical report is available on arXiv.

Choose Your Starting Point

Explore Open-AoE with AiWork

AiWork is a desktop AI workspace where you can select a project, choose a Skill or Expert, and describe a task in a conversation. For Open-AoE, start with the data you need or the robot/model you want to work with; review the returned files, charts, videos, and commands as the work progresses.

AiWork's desktop interface: projects and “Experts · Skills” on the left, with a task prompt and selected Expert in the conversation area. Compose a request, select the Expert, and start the conversation.

Choose a guide

The Human-to-Robot Data Conversion Expert prepares robot-specific trajectories and data. The Embodied Model Training Expert checks compatibility and generates commands; you run training and evaluation on your compute and return the results for analysis.

Find openaoe数据助手 in the Skill Center Marketplace, or 人类数据转换专家 and 具身模型训练专家 in the Expert Community. For manual upload, download the AiWork installation packages.

Quick Start: Inspect One Segment

1. Clone the repository

git clone https://github.com/ant-research/Open-AoE.git
cd Open-AoE

2. Download the data

Choose either Hugging Face or ModelScope, then locate one extracted segment directory. Its expected layout is documented in the data specification.

3. Render the segment

Some visualization and retargeting workflows require MANO model files. Register and download MANO_RIGHT.pkl and MANO_LEFT.pkl from the MANO website, then install them into the repository's shared asset directory:

bash assets/mano/download_mano.sh \
  ~/Downloads/MANO_RIGHT.pkl \
  ~/Downloads/MANO_LEFT.pkl

cd aoe-visualization
pip install -r requirements.txt
python visualize.py --sample /path/to/open_aoe_segment

The output is output/<segment-name>/AoE_output_vis.mp4, a synchronized review of the ego video, reconstructed hands, wrist trajectories, atomic actions, world-frame motion, and timeline. See the visualization guide for EGL/OpenGL requirements and batch rendering.

From Raw Data to Model Input

Each segment is a synchronized multimodal record rather than a standalone video:

The toolchain then maps those synchronized signals into task-specific representations:

[!IMPORTANT] Training conversion is an action-semantics adaptation, not just file-format conversion. Start with the shared action specification, then follow the README for your target model.

Training Recipe Map

Each recipe is self-contained and documents its upstream repository and verified commit, data conversion, environment variables, training command, outputs, and any required patch. Upstream projects and checkpoints are not vendored into this repository.

Reconstruction and Retargeting Map

Third-party repositories, model weights, robot assets, and generated videos are not included. Follow each subproject's setup guide to obtain its external dependencies.

Data Processing and Quality Control

1. On-device capture control checks hand visibility, wearing conditions, lighting, motion quality, and device health before upload.
2. Offline quality control and scene labeling filters invalid or sensitive content, standardizes frame rate, slices videos, and assigns scene/task metadata.
3. Reconstruction and annotation estimates camera trajectories, reconstructs MANO hands, and produces atomic-action segments.
4. Quality inspection and delivery applies completeness, correctness, and temporal-consistency gates followed by human review.

Repository Layout

Open-AoE/
├── Open-AoE-tech-report.pdf  # Current technical report
├── open-aoe-2000h/           # Dataset format and field-level documentation
├── aoe-visualization/        # Synchronized data review and rendering
├── aoe-reconstruct-retarget/ # Reconstruction, retargeting, replay, and overlays
├── aoe-training-ready/       # Model adapters, converters, launchers, and patches
├── assets/mano/              # Shared MANO setup helper; model files are not tracked
├── docs/                     # Overview figures and AiWork best-practice guides
├── CONTRIBUTING.md
├── LEGAL.md
└── LICENSE

Why Open-AoE?

• Smartphone-first collection: consumer devices make real-world egocentric capture accessible and scalable.
• Manipulation-aligned signals: calibrated RGB is synchronized with camera motion, MANO hand reconstruction, validity masks, and atomic actions.
• Complete data-to-model path: the repository covers inspection, retargeting, representation conversion, and model-specific training integration.
• Modular and extensible: components can be used independently, and new robot embodiments or model recipes can be added as self-contained integrations.

Citation

The technical report is available on arXiv.

Contact

• Data processing pipeline applications: kaile.zk@antgroup.com
• Industry collaboration: kaile.zk@antgroup.com

Contributing

Contributions of robot embodiments, reconstruction or retargeting backends, visualization features, data converters, and training recipes are welcome. Read CONTRIBUTING.md before opening a pull request.

License and Third-Party Components

Original source code in this repository is released under the Apache License 2.0. Dataset distribution terms, model weights, robot assets, MANO files, and third-party components may use different licenses. Read LEGAL.md and the relevant subproject README before redistribution or commercial use.

Acknowledgements

Open-AoE is built by the AoE community. We gratefully acknowledge the following contributors from industry, universities, and research institutions.

Dataset and Visualization

• National University of Singapore: Qingze Guan
• University of Chinese Academy of Sciences: Zhengxing Wu
• Ant Digital Technology, Ant Group: Zishuo Li, Wanke Zhan, Yang Sun, Zhiyi Huang, Zitong Shan

Open-Source Code, Toolchain, and Experiments

• Zhejiang University: Jiadong Hong
• The University of Hong Kong: Zhenchao Jin, Yushi Feng
• The Hong Kong University of Science and Technology (Guangzhou): Taowen Wang
• Beijing Academy of Artificial Intelligence: You Liu, Yibo Wang
• University of Chinese Academy of Sciences: Yifan Yang
• The Hong Kong University of Science and Technology: Hao Cheng
• Ant Digital Technology, Ant Group: Bowen Yang, Changtao Miao, Zhaowen Zhou, Man Luo

We also thank all data contributors, maintainers, and upstream open-source and research communities whose work made this release possible.