What this is

Learning to act in the physical world requires more than observing what an action looks like: models must capture how first-person perception, whole-body motion, dexterous manipulation, object state, sound, and touch evolve together as humans pursue goals over time. Existing datasets fragment this experience across viewpoints, modalities, or spatial scales.

ACE-Data-0 records all of it in one pass, on one clock, in one world frame. Participants receive goal-level instructions ("prepare a cup of tea and serve it at the table") rather than step-by-step scripts, so planning, hesitation, and improvisation enter the data by themselves. Human states, object states, and contact are metrically tracked or directly sensed, so the annotations are measured rather than estimated: they stay correct under furniture occlusion, extreme viewpoints, and motion blur, where image-based detectors fail.

These values describe the planned release and will be verified in the final release manifest.

What each take contains

Every take shares one timeline and one world coordinate frame, and ships its own calibration and sync tables as data. Any tracked 3D point can therefore be projected onto any pixel of any view, and any two streams paired at any instant, without rerunning any part of the capture pipeline.

Takes come in three families: atomic HOI (1-3 household tasks, ~3 min), chains of HOI (one continuous activity of ~20-30 min ending with the scene tidied back into order), and human-scene interaction (whole-body motion and furniture contact, almost no objects, ~5 min).

Room-scale directory structure

Room-scale/
├── Body_shape/
├── calibration/
└── Take-xxxxxx/
    ├── motions/
    ├── raw/
    ├── timeline/
    ├── videos_and_annotations/
    └── caption/

• Body_shape/ contains the body-shape beta parameters for each participant.
• calibration/ contains calibration data for the OptiTrack system and ZED cameras, together with the scripts used to calculate the Homie calibrations.
• Each Take-xxxxxx/ directory contains the data for one take:
  • motions/ contains human and object motion data.
  • raw/ contains the raw egocentric, exocentric, and motion-capture data.
  • timeline/ contains the synchronized timeline for all modalities. Its annotation.json file provides video_keep_interval_s, which specifies the video intervals to retain after removing footage captured only for synchronization.
  • videos_and_annotations/ contains the captured egocentric and exocentric videos at their original resolution, along with annotated videos for the human body, hands, and objects.
  • caption/ contains a text description of the take. Caption timestamps refer to the filtered timeline after applying video_keep_interval_s.

Table-scale directory structure

Table-scale/
├── HOI/
│   ├── data/
│   │   └── chunk-000/
│   ├── meta/
│   │   └── takes/
│   ├── objs/
│   ├── raw/
│   │   ├── calibration/
│   │   ├── camera_array_aniposelib/
│   │   ├── captions/
│   │   ├── hand_poses/
│   │   ├── imu/
│   │   ├── obj_file/
│   │   ├── obj_poses/
│   │   ├── optitrack_to_camera_world_alignment/
│   │   └── poses/
│   └── videos/
│       ├── egocap/
│       └── gopro/
└── Tactile/
    └── ACE-Data-Tactile-{0..10}/
        ├── data/
        │   └── chunk-000/
        ├── meta/
        │   └── takes/
        ├── raw/
        │   ├── calibration/
        │   ├── camera_array_aniposelib/
        │   ├── captions/
        │   ├── imu/
        │   └── tactile/
        └── videos/
            ├── egocap/
            └── gopro/

• HOI/ contains table-scale human-object interaction recordings focused on fine-grained hand-object manipulation.

  • data/ contains LeRobot-compatible parquet chunks. Rows are canonical reference samples on a fixed-rate timeline; frame_index is an episode-local reference-sample index, while the raw-row ranges point to records in raw/.
  • meta/ contains take-level metadata, including the takes/ index and task information.
  • objs/ contains the object mesh assets referenced by the object-tracking data.
  • raw/ contains the captured sensor records and sidecars, including camera calibration, camera-array processing data, captions, human and object poses, IMU data, object files, and OptiTrack-to-camera alignment data.
  • videos/ contains the source video streams, separated into egocap/ for egocentric capture and gopro/ for exocentric GoPro cameras.

• Tactile/ contains the tactile table-scale release split into eleven shards, ACE-Data-Tactile-0/ through ACE-Data-Tactile-10/. Each shard follows the same layout:

  • data/ contains the canonical timeline samples and parquet chunks for that shard.
  • meta/ contains take-level metadata under takes/.
  • raw/ contains calibration, camera-array processing data, captions, IMU records, and raw tactile measurements under tactile/.
  • videos/ contains the corresponding egocap/ and gopro/ video streams.

The data/, raw/, and videos/ directories are complementary: the parquet data provides timeline indices and raw-row ranges, sensor records remain in raw/, and the video bytes remain in videos/.

How it was captured

Participants wear an ACE-Ego-Head-V02 Lite headset (4 fisheye cameras, IMU, 5 tracked markers), a 41-marker mocap suit, and full-palm tactile gloves.

Two numbers carry the credibility of everything above. Temporal: all devices are registered to the OptiTrack 60 Hz clock by photographing a nanosecond-resolution clock displayed on the mocap host, giving millisecond-level residuals, within a single mocap frame. Spatial: an ArUco board with retroreflective corners bridges exocentric cameras that share no field of view (median reprojection error < 3 px), while the headset is solved by hand-eye calibration against its tracked rig (~2 px), so egocentric camera poses are measured rather than estimated and do not drift.

The blog and the technical report cover the capture protocol, calibration, and annotation pipeline in full.

Benchmark

We hold out 10 hours as a test set and evaluate 30+ published methods across three levels: tactile inference from video, human motion recovery, and hand motion from egocentric and exocentric views. Existing methods degrade sharply under contact, occlusion, egomotion, and long horizons. In particular, strong per-frame pose accuracy does not imply an accurate world-frame trajectory, and egomotion, not finger articulation, dominates the error in egocentric hand reconstruction. Full tables are in the technical report.

Access

This repository is gated. Access is granted to named individuals for non-commercial academic research only, and takes effect once you accept the terms on the access form above.

• Use your institutional email and describe your intended research. What you submit is recorded as part of your licence; inaccurate information is a breach of the agreement.
• Access is personal and non-transferable. Collaborators and students must each submit their own request. Redistributing the data terminates your licence and revokes your access.
• For commercial licensing or industrial collaboration, open a thread on the Community tab instead of submitting the form.

Data files are not published yet. Repository layout, storage requirements, checksums, and loading examples will be added at release time. Because the streams are large, the release will use sharded archives rather than direct browser downloads. Approved users keep their access.

Intended and prohibited uses

Intended for non-commercial academic research on embodied perception, human and hand motion recovery, human-object and human-scene interaction, egocentric and multi-view video understanding, cross-modal learning across vision, motion, audio, and touch, and imitation learning, world models, and vision-language-action systems.

Not for identifying, re-identifying, or profiling participants; biometric recognition or surveillance; inferring sensitive personal attributes; any commercial purpose; or representing all homes, cultures, bodies, abilities, or household practices without further validation. The LICENSE is binding and defines the full set of restrictions.

All participants volunteered and signed informed consent covering data collection and research release, including the appearance of their faces. The dataset contains identifiable individuals. If you are a participant and want your recordings withdrawn, reach us through the Community tab and the affected takes will be removed from subsequent releases.

Limitations

Two sites only, so limited variation in layouts, furnishings, and lighting. Tracked objects must be scanned and marked in advance, and state changes of articulated mechanisms, fluids, and deformable materials are not annotated. The mocap suit, gloves, headset, and markers are visible in the recordings and may introduce dataset-specific visual cues.

License

ACE-Data-0 Research License Agreement: non-commercial academic research only, no redistribution, no re-identification. Read it in full before requesting access.

Citation

@article{cao2026acedata0,
  title   = {ACE-Data-0: Human-Centric Ambient Capture as Embodied Data Engine},
  author  = {Cao, Yukang and Xie, Haozhe and Wen, Beichen and Yao, Runmao and
             Liu, Yinghao and Huang, Yue and Liao, Zhichao and Wang, Yunxiang and
             Liu, Haiheng and Tian, Xingshun and Su, Dawei and Zhuo, Long and
             Tao, Dacheng and Wang, Xiaogang and Pan, Liang and Liu, Ziwei},
  journal = {arXiv preprint arXiv:2607.28625},
  year    = {2026}
}

Contact

Project updates on the blog. Questions about access, licensing, or annotations belong on the Community tab of this repository.