HOI-Retarget — Humanoid Human–Object Interaction Motion

Project page · Code · 3D viewer · Paper · arXiv

Learning from demonstration (LfD) has enabled humanoid robots to acquire diverse whole-body skills, but extending this paradigm to human-object interaction (HOI) is limited by the availability of robot-compatible interaction references. We present HOI-Retarget, a contact-centric retargeting method that transfers HOI onto a humanoid robot for large-scale motion-data generation. Its windowed trajectory optimization uses every labeled contact as a target in the object frame, balancing body tracking, foot support and smoothness under the robot's kinematic limits. The method can augment a single demonstration across object sizes, absorb contacts reconstructed from monocular video, and extend to several robots manipulating one object.

This is the corpus that method produces: 13,904 humanoid trajectories from five human–object interaction capture sets. Each row is one (motion, robot) pair — joint angles, a floating base, the object's 6-DoF pose, and the per-link contact flags that say which palms and feet touch the object at every frame.

---

Contents

Per dataset (G1, object_scale 0.83)

Per robot

The H2 is taller and keeps less: 91.8 % on OMOMO, 97.6 % on ParaHome, 93.2 % on NeuralDome, 77.9 % on CoRoleHOI, 90.0 % on IMHD².

Object meshes

The 13 OMOMO objects ship under assets/objects/ (44 MB of URDF, mesh and surface samples), so the 4,421 OMOMO motions are usable as downloaded. Point the code at this directory:

export HOI_RETARGET_OBJECT_ROOTS=<this download>

The variable adds search roots; do not use HOI_RETARGET_ASSETS, which replaces the whole asset root and hides the robot models.

The other four datasets' meshes reach you through InterAct under CC BY-NC-SA 4.0 with written authorisation and are not redistributed here. Place them under the same root in the layout object_model_path names:

<root>/assets/objects/<object>.urdf       OMOMO — included above
<root>/<dataset>/_assets/<object>.urdf    ParaHome, NeuralDome, CoRoleHOI, IMHD²

The code repository's docs/DATA.md covers obtaining them, and hoi-retarget-stage-object writes the URDF and surface samples for a mesh of your own.

Object names repeat across datasets but the meshes do not, so objects are keyed <dataset>__<object>. docs/objects.jpg is a contact sheet of all 75.

---

Quick start

pip install datasets

from datasets import load_dataset
import numpy as np

ds = load_dataset("leggedrobotics/hoi-retarget", split="train")           # everything
ds = load_dataset("leggedrobotics/hoi-retarget", "omomo", split="train")  # one source dataset

clean = ds.filter(lambda r: r["qc_pass"])          # the curated subset
g1    = ds.filter(lambda r: r["robot"] == "unitree_g1")

r = clean[0]
dof = np.array([np.asarray(x) for x in r["dof_pos"]])    # (T, 29) joint angles, radians
obj = np.array([np.asarray(x) for x in r["object_pos"]]) # (T, 3) object position, metres
print(r["clip_id"], dof.shape, r["object"], r["qc_flags"])

Every column is described in docs/SCHEMA.md.

---

Quality control

qc_pass is evaluated per (motion, robot): the two robots have different joint limits and fail differently.

qc_pass = False  if  wrist_runfrac > 0.50            # wrist_sustained: a wrist joint pinned beyond
                                                     # 80 % of its own half-range for >50 % of the clip
                 or  foldx_p80 >= 50                 # body_folded: the trunk folds 50 deg further
                                                     # from vertical than the human's did
                 or (limit_sat_pct >= 15 and foldx_p80 >= 15)
                                                     # body_folded: joints against their stops,
                                                     # corroborated by a real fold

foldx_p80 is source-relative — the robot's trunk fold minus the human's — so a person who genuinely squats or sits scores near zero. The flags catch physically implausible motion, not unusual interaction: a clip where the robot kicks a box or never uses its hands passes. Failing rows ship flagged, and qc_flags names the rule that fired (wrist_sustained, body_folded).

One flag is set by hand: object_floating marks the 18 IMHD² skateboard rows (9 motions × 2 robots). In the source data the board sits 20–30 cm above the floor throughout, with the feet on the floor, so no retarget can put the robot on the board.

---

Seeing the motions

Every row carries a render of that trajectory in the video column, so the table preview plays each clip in place at 320 × 320 (~50 kB). CoRoleHOI rows show both robots of the pair (this row's robot in the normal colour, the partner tinted).

HOI-Retarget Contact Playback plays any clip in 3D in the browser, with search and filters by dataset, robot and QC result.

---

Licensing

CC BY-NC-SA 4.0: non-commercial, attribution, and derivatives carry the same licence. That is the most restrictive term among the sources, and it propagates.

These trajectories are derivative works of the source motion capture, and the contact annotations they were optimised against originate with InterAct under CC BY-NC-SA 4.0. Cite the source dataset for any clip you use; the dataset column names it and NOTICE.md gives the reference.

The OMOMO object meshes are redistributed from InterMimic under MIT (geometry © 2023 Jiaman Li, assets © 2025 Sirui Xu), notice in assets/objects/LICENSE-OBJECTS. Robot models are Unitree's, BSD-3-Clause (docs/LICENSE-unitree.txt). The retargeting code is BSD-3-Clause.

---

Citation

@article{shin2026hoiretarget,
  title   = {HOI-Retarget: Contact-Centric Retargeting for Human-Object Interaction},
  author  = {Shin, Jihwan and L\'opez Escoriza, Adri\`a and He, Junzhe and
             Heyrman, Matthias and Hutter, Marco},
  journal = {arXiv preprint arXiv:2609.34674},
  year    = {2026},
  url     = {https://arxiv.org/abs/2609.34674}
}

Cite the source dataset your clips come from as well — the dataset column names it, and NOTICE.md lists every reference.