HOI-Retarget — Contact Playback

A 3D viewer for the HOI-Retarget corpus: human–object interaction capture retargeted onto humanoid robots, with the per-link contact flags the pipeline treats as a constraint rather than an output.

Pick a clip, scrub it, orbit it. When a contact link's flag is set at that frame, the link tints and its lane lights up — orange for hands, cyan for feet. Contact is the only saturated colour on the page, so you can read frame by frame exactly which palm or foot is on the object.

What is in it

13,448 clips covering all 13,904 corpus trajectories, streamed from the dataset repository rather than bundled here.

Two robots, every clip retargeted to both: Unitree G1 (42 bodies, objects scaled 0.83) and Unitree H2 (36 bodies, objects at 1.0). Four contact links each — two palm pads, two ankles.

The 456 collaborative clips are two people carrying one object, retargeted to two robots sharing one scene. The second robot is tinted steel blue-violet and its contact lanes are striped, so you can tell at a glance whose hand is on the object. Every CoRoleHOI motion is half of a pair, so none appears on its own; the two independently solved object tracks agree to within 4.5 cm (median 0.3 cm) once each agent's z_offset_m is added to its z coordinates.

Finding a clip

13,448 is too many to scroll, so:

• Search matches object, subject, dataset, robot and clip id; terms are AND-ed.
• Filters for dataset, robot and QC result.
• QC defaults to the 12,586 clips that pass; a collaborative clip passes when both of its agents do, so this counts clips, not the dataset's 13,904 rows (13,018 pass). The 862 that fail are one click away: a wrist held near its joint stop for more than half the clip (wrist_sustained), a trunk folded far beyond the human's (body_folded), or the IMHD² skateboard, which floats above the floor in the source data (object_floating).

Keyboard: / focuses search, l the list, arrows move the cursor, Enter loads.

Objects

Object meshes are keyed <dataset>__<object>, never by bare name. chair is a different mesh in NeuralDome and IMHD², and so is pan in ParaHome and IMHD² — keying on the name alone draws the wrong object. All 75 ship with this Space; only the trajectories stream.

Light and dark

Follows your system, with a toggle in the header that overrides it. The contact hues are identical in both themes; what changes is what they are drawn on, so orange and cyan stay legible against paper as well as against slate.

Data format

The viewer reads the dataset's viewer/ folder: index.json (every clip and its agents), shards/<ds>__<robot>.json, and one little-endian float16 .bin per (motion, robot), block-major:

body_pos (T, NB, 3) · body_quat (T, NB, 4) wxyz · object_pos (T, 3) · object_rot (T, 4) wxyz · contact (T, NC)

NB is 42 for the G1 and 36 for the H2; NC is 4. ?data=<relative dir> points the viewer at a local copy for development; remote origins are refused.

Licence

CC BY-NC-SA 4.0, propagated from the most restrictive source: three of the five datasets are themselves NonCommercial-ShareAlike.

Object geometry keeps its own terms — MIT for the OMOMO meshes (via InterMimic), the source dataset's licence for the rest. The per-dataset breakdown is in licenses/OBJECTS.md. Robot models are Unitree Robotics, BSD-3-Clause. Contact annotations originate with InterAct. Full attribution in the corpus's NOTICE.md.

The retargeting code is separate and permissive (BSD-3-Clause): github.com/leggedrobotics/hoi-retarget.