KinDER Demonstration Datasets

Human-collected robot demonstration datasets for the KinDER physical-reasoning benchmark (RSS 2026).

These datasets are used to train the imitation learning baselines (kinder-imitation-learning) and the model-based RL baseline (kinder-mbrl) shipped with KinDER.

---

Dataset summary

---

Data collection

2D environments (motion2d, stickbutton2d, dynobstruction2d, dynpushpullhook2d): Demonstrations were collected using a PS5 controller.

3D environments (BaseMotion3D, shelf3d, sweep3d, transport3D): Demonstrations were collected using an iPhone web app (XR Browser) or a VR headset, streaming 6-DoF pose commands to the TidyBot robot in simulation.

---

File format

Every .hdf5 file follows the same schema:

data/
  demo_0/
    actions       (T, action_dim)   — control commands applied at each step
    obs/
      robot_state (T, robot_dim)    — proprioceptive robot state
      env_state   (T, env_dim)      — environment / object state
      image       (T, 224, 224, 3)  — top-down / scene RGB image  [2D envs]
      base_image  (T, 224, 224, 3)  — robot base camera           [3D envs]
      wrist_image (T, 224, 224, 3)  — robot wrist camera          [3D envs]
      overview_image (T, 224, 224, 3) — static overview camera    [3D envs]
  demo_1/
    ...

All images are uint8 RGB arrays of shape (T, 224, 224, 3).
State and action arrays are float32.

Per-dataset dimensions

2D environments

3D environments (TidyBot)

3D action space (11-dim): [base_x, base_y, base_yaw, joint_1, …, joint_7, gripper] — delta joint-position commands.

3D robot state (19–22-dim): base pose + joint angles + joint velocities + gripper state.

---

Loading the data

With h5py (any baseline)

import h5py
import numpy as np

with h5py.File("sweep3d_100.hdf5", "r") as f:
    episodes = list(f["data"].keys())           # ["demo_0", "demo_1", ...]
    ep = f["data"][episodes[0]]
    actions     = ep["actions"][:]              # (T, 11)
    robot_state = ep["obs"]["robot_state"][:]   # (T, 22)
    env_state   = ep["obs"]["env_state"][:]     # (T, 131)
    base_image  = ep["obs"]["base_image"][:]    # (T, 224, 224, 3)

With kinder-mbrl (world model training)

python experiments/train_world_model.py \
    --mode train \
    --hdf5_path sweep3d_100.hdf5 \
    --output_dir output \
    --epochs 1000

The training script reads robot_state and env_state from obs/ and computes per-step (state, action, delta) transitions automatically.

With kinder-imitation-learning (Diffusion Policy)

Convert raw demos to the diffusion policy format:

cd kinder-baselines/kinder-models/scripts
python demos_to_hdf5.py \
    --teleop_data_dir $YOUR_DATA_DIR \
    --output_path $OUTPUT_HDF5_PATH \
    --render_images

Then train:

cd ~/kinder-diffusion-policy
mamba activate robodiff
python train.py --config-name=train_sweep3d_image

---

Baselines trained on this data

---

Citation

If you use these datasets, please cite the paper: KinDER: A Physical Reasoning Benchmark for Robot Learning and Planning:

@inproceedings{huang2026kinder,
  title     = {KinDER: A Physical Reasoning Benchmark for Robot Learning and Planning},
  author    = {Huang, Yixuan and Li, Bowen and Saxena, Vaibhav and Liang, Yichao and Mishra, Utkarsh and Ji, Liang and Zha, Lihan and Wu, Jimmy and Kumar, Nishanth and Scherer, Sebastian and Xu, Danfei and Silver, Tom},
  booktitle = {Robotics: Science and Systems (RSS)},
  year      = {2026}
}