HiFi-UMI-2K: High-Fidelity Robot-Free Manipulation Data

2,000 hours released · 6 synchronized camera views · 480+ scenes · 3 mm pose accuracy · <40 µs synchronization

🌐 Project Website | 📦 Dataset | 📄 Paper: arXiv:2607.25895

_{Examples from the HiFi-UMI corpus. Click the image to play the video.}

📚 Introduction

HiFi-UMI is a portable, high-fidelity bimanual capture system for collecting robot-free manipulation demonstrations. HiFi-UMI-2K is the large-scale dataset produced by this system and is designed to provide action-grounded supervision for manipulation policy pre-training and post-training without requiring a robot during data collection.

The release contains a curated 2,000-hour subset of a source corpus exceeding 20,000 hours and 4.32 million episodes across more than 480 scenes. Each episode includes synchronized multi-view video, calibrated bimanual end-effector trajectories, gripper states, language annotations, task metadata, and quality-control information.

The HiFi-UMI system and dataset accompany the technical report:

HiFi-UMI: Learning Deployable Manipulation Policies from High-Fidelity UMI Data Alone

The work asks whether increasing the fidelity of robot-free data can remove the remaining real-robot teleoperation "anchor" normally used during deployment-oriented post-training. The paper reports that policies trained only on HiFi-UMI task demonstrations can be deployed directly on a real bimanual robot, with no teleoperated robot demonstrations in the training loop.

💡 Highlights

• Robot-free collection: demonstrations are captured directly from natural human manipulation without requiring the target robot, a teleoperation rig, or an instrumented environment.
• High-fidelity trajectories: head-mounted offline stereo-inertial SLAM and fiducial-marker localization provide approximately 3 mm local end-effector accuracy.
• Native bimanual pose: both hands are localized in the same head-camera coordinate frame, providing accurate inter-hand relative pose without post-hoc cross-camera reconstruction.
• Hardware synchronization: all cameras, IMUs, encoders, and gripper signals share a GPIO hardware trigger, yielding cross-sensor offsets below 40 µs.
• Ultra-wide six-view sensing: every episode contains stereo head views and two non-parallel fisheye views per hand, covering approximately 200° horizontally and vertically around each gripper.
• Replay-validated trajectories: the processing pipeline achieves approximately 98% trajectory reconstruction success and 98% WBC replay validation success.
• Training-ready export: data is distributed in a LeRobot v3-style format with frame-level Parquet tables, MP4 videos, task text, episode metadata, validity masks, and normalization statistics.
• Deployment-oriented supervision: the accompanying paper evaluates HiFi-UMI-only post-training on both vision-language-action and world-action-model policy families.

📊 Dataset at a Glance

The 2,000-hour release is a curated subset of the larger source corpus. Statistics explicitly labeled as source-corpus statistics describe the full processed collection rather than the released subset alone.

🧤 HiFi-UMI Capture System

The capture hardware is co-designed around four fidelity requirements:

1. Accurate pose acquisition: a head-mounted stereo rig and IMU are processed with offline stereo-inertial SLAM. Marker cubes on both hands are localized in the same head-camera frame.
2. Natural bimanual interaction: a full-palm, glove-like asymmetric gripper preserves direct contact, natural force distribution, and support for both small and larger objects.
3. Wide visual coverage: two head cameras and four hand cameras reduce occlusion and blind spots during contact-rich manipulation.
4. Online quality control: the device detects underexposure, motion blur, overly fast motion, and tracking-risk events during collection and provides immediate operator feedback.

_{Native bimanual capture hardware: two hands and four hand-camera views.}

🔄 High-Fidelity Data Engine

Raw captures pass through a closed-loop data-production pipeline:

1. Collection and upload with hardware synchronization, online quality monitoring, and operator-provided temporal markers.
2. Trajectory reconstruction and automatic cleaning using offline stereo-inertial SLAM and automatic anomaly detection.
3. WBC replay validation using whole-body retargeting in simulation to reject kinematically or dynamically infeasible trajectories.
4. AI-assisted annotation for task descriptions, subtask boundaries, manipulated objects, abnormal events, and annotation confidence.
5. Human verification focused on quality-control flags and low-confidence annotations.
6. Analysis and export with task, scene, object, action-pattern, trajectory-quality, and replay-success statistics.

Training and evaluation results feed back into later collection plans, allowing the corpus to be rebalanced toward missing tasks, objects, interaction dynamics, and recovery behaviors.

_{Trajectory reconstruction and six-view replay. Click the image to play the video.}

✍️ Trajectory Fidelity

The handwriting demonstration below provides a qualitative view of the local trajectory accuracy. The reconstructed bimanual trajectory preserves millimeter-scale pen motion and remains aligned with all six camera streams.

_{Click the image to play the handwriting reconstruction video.}

📈 Results Reported in the Paper

The paper compares HiFi-UMI-only post-training with conventional in-domain real-robot teleoperation post-training. Architecture, initialization, optimization, action representation, and deployment protocol are held fixed within each model family.

Zero-robot post-training

Additional findings:

• Pooling the two VLA families gives 64.4% (206/320) for HiFi-UMI post-training and 64.1% (205/320) for teleoperation post-training.
• OpenPI-π_0.5 post-trained only on HiFi-UMI reaches 85% success on Remote Insertion.
• The comparisons are evaluated under scene-level distribution shift for the HiFi-UMI condition: UMI demonstrations are collected across different sites, while teleoperation data is collected in the robot evaluation environment.
• The comparison measures practical data-production pipelines rather than equal-sample data efficiency: the VLA experiments use approximately 3,200 HiFi-UMI trajectories or 300 teleoperation trajectories per task.

Large-scale pre-training

The paper also pre-trains StarVLA-QwenPI on a 4,000-hour HiFi-UMI mixture:

• Held-out action-prediction error decreases by 61% over one pass through the corpus.
• Pre-decay scaling follows a power law with exponent α = 0.268 and R² = 0.993.
• Mean action error on ten unseen tasks decreases by 41%.
• HiFi-UMI pre-training raises aggregate real-robot post-training success by 18.1 percentage points under a controlled initialization comparison.

These figures describe the research setting reported in the technical report. They are not universal performance guarantees for every policy, robot embodiment, or downstream task.

📂 Repository Layout

The repository is sharded at the top level:

chunk-XXXX/
└── part-0000/
    ├── data/
    │   └── chunk-000/
    │       └── file-000.parquet
    ├── videos/
    │   ├── observation.images.head_main/
    │   │   └── chunk-000/file-000.mp4
    │   ├── observation.images.head_main_stereo_right/
    │   │   └── chunk-000/file-000.mp4
    │   ├── observation.images.left_hand_up/
    │   │   └── chunk-000/file-000.mp4
    │   ├── observation.images.left_hand_down/
    │   │   └── chunk-000/file-000.mp4
    │   ├── observation.images.right_hand_up/
    │   │   └── chunk-000/file-000.mp4
    │   └── observation.images.right_hand_down/
    │       └── chunk-000/file-000.mp4
    └── meta/
        ├── info.json
        ├── modality.json
        ├── stats.json
        ├── tasks.parquet
        └── episodes/
            └── chunk-000/file-000.parquet

Path templates are recorded in meta/info.json:

{
  "data_path": "data/chunk-{chunk_index:03d}/file-{file_index:03d}.parquet",
  "video_path": "videos/{video_key}/chunk-{chunk_index:03d}/file-{file_index:03d}.mp4"
}

📊 Frame-Level Data

data/chunk-000/file-000.parquet is the frame-level training table. Each row corresponds to one synchronized frame.

Frames with valid.frame == false are intentionally retained in both Parquet and video files. This preserves strict alignment between data rows, video frames, and timestamps. Most training pipelines should filter to valid.frame == true.

🎮 State and Action Representation

The state and action vectors contain 20 dimensions:

observation.state = [right_10d, left_10d]
action            = [right_action_10d, left_action_10d]

Each hand uses the following 10-dimensional layout:

[x, y, z, rot6d_0, rot6d_1, rot6d_2, rot6d_3, rot6d_4, rot6d_5, gripper]

The stored action uses the same layout and units and represents an absolute next-state target. The final frame normally repeats the final available target so that action rows and video frames remain aligned.

The policy implementations in the paper may convert these stored targets into model-specific robot-centric relative actions during training. Users should not assume that the exported action is already expressed in the relative action convention used by a particular VLA, WAM, or robot controller.

meta/modality.json records the semantic slice of every state and action block, including right and left end-effector pose, gripper state, and the mapping between video keys and original camera streams.

🧭 Coordinate Frames

_{Coordinate-frame conventions for the HiFi-UMI capture system.}

The dataset uses the following coordinate-frame conventions:

Both hand coordinate frames use the same axis convention. The head-camera frame follows the standard optical-camera convention, with +Z pointing into the observed scene.

The definitions above describe the local coordinate frames attached to the two hands and the head camera. Their time-varying trajectories are all expressed in the same shared world coordinate frame.

Shared World Coordinate Frame

• Shared frame: the left-hand, right-hand, and head trajectories of a recording are represented in one common world coordinate system.
• Origin: the world-frame origin is initialized at an arbitrary position and has no fixed semantic location in the physical scene.
• +Z axis: the positive Z axis is aligned with the direction of gravity.
• XY plane: the X and Y axes lie in the plane perpendicular to gravity.

Because the world origin is arbitrary, absolute world positions should not be compared directly across different recordings without additional alignment. Within a recording, the head and both hand trajectories are mutually aligned and can be transformed or compared in the shared world frame.

🎥 Video Streams

Videos are stored as:

videos/{video_key}/chunk-{chunk_index:03d}/file-{file_index:03d}.mp4

Standard video keys:

observation.images.head_main
observation.images.head_main_stereo_right
observation.images.left_hand_up
observation.images.left_hand_down
observation.images.right_hand_up
observation.images.right_hand_down

Resolution, FPS, codec, pixel format, and channel count are defined in meta/info.json["features"].

Within each video shard, episode frames are concatenated in episode_index order. The position of each episode inside every video file is recorded in meta/episodes/chunk-000/file-000.parquet:

videos/{video_key}/chunk_index
videos/{video_key}/file_index
videos/{video_key}/from_timestamp
videos/{video_key}/to_timestamp

To locate the video frame corresponding to a data row:

1. Select the row's episode_index.
2. Look up the episode's video file and from_timestamp.
3. Add the row's episode-relative timestamp.
4. Decode the corresponding frame using the FPS from meta/info.json.

🏷️ Task Text

meta/tasks.parquet maps natural-language task descriptions to task_index:

index.name = "task"
columns    = ["task_index"]

The frame table references this mapping through data.task_index. Task strings associated with each episode are also stored in the tasks field of the episode metadata.

🧾 Episode Metadata

meta/episodes/chunk-000/file-000.parquet contains one row per episode.

Expected invariants:

length = dataset_to_index - dataset_from_index
timestamp = frame_index / fps

🗂️ Dataset Metadata

meta/info.json is the primary entry point:

meta/stats.json contains part-level feature statistics:

min
max
mean
std
count

These values can be used for normalization. If a training job uses only valid.frame == true, we recommend recomputing normalization statistics after applying the same filtering rule.

📖 Citation

If you use HiFi-UMI, please cite the technical report:

@article{simpleai2026hifiumi,
  title   = {HiFi-UMI: Learning Deployable Manipulation Policies from High-Fidelity UMI Data Alone},
  author  = {{Simple AI} and Wei, Yuteng and Ma, Jinming and Wang, Jiawei and Zhou, Weitao and Zuo, Yushen and Rui, Ke and Li, Minglei and Zhang, Jinhao and Pan, Zhikang and Wang, Xiang and Jia, Haoran and Du, Huan and Zeng, Zicheng and Ma, Jun and Qin, Guiyu and Zhang, Di and Li, Xiaofei},
  journal = {arXiv preprint arXiv:2607.25895},
  year    = {2026}
}

📜 License

The dataset is released under the Creative Commons Attribution 4.0 International License (CC BY 4.0).

You may share and adapt the data, including for commercial use, provided that appropriate attribution is given, a link to the license is included, and modifications are indicated.

❤️ Acknowledgements

HiFi-UMI-2K is produced by Simple AI with contributions from the capture-system, data-engine, annotation, policy-learning, and real-robot evaluation teams, together with the operators and reviewers who collected and verified the demonstrations.

For project updates, paper release information, and additional videos, visit the HiFi-UMI project website.