Dataset Card

This is the fine-tuning dataset used in the paper RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation.

Source

• Project Page: https://rdt-robotics.github.io/rdt-robotics/
• Paper: https://arxiv.org/pdf/2410.07864
• Code: https://github.com/thu-ml/RoboticsDiffusionTransformer
• Model: https://huggingface.co/robotics-diffusion-transformer/rdt-1b

Uses

Download all archive files and use the following command to extract:

cat rdt_data.tar.gz.* | tar -xzvf -

In the resulting rdt_data folder, each sub-folder is a task. In each task sub-folder, we have HDF5 files and the instruction JSON file, as illustrated below:

rdt_data/---task_1/---episode_1.hdf5
          |         |
          |-task_2/ |-episode_2.hdf5
          |         |
          |-task_3/ |-...
          |         |
          |-...     |-expanded_instruction_gpt-4-turbo.json

Each HDF5 file corresponds to a trajectory/episode of the task, which has the following keys:

• observations:
  • qpos: joint positions of the two robot arms, (TRAJ_LEN, 14); the seventh and fourteenth joints are gripper joint angles
  • images:
    • cam_high: RGB images from the exterior camera at each time step, (TRAJ_LEN, 480, 640, 3)
    • cam_left_wrist: RGB images from the left-wrist camera at each time step, (TRAJ_LEN, 480, 640, 3)
    • cam_right_wrist: RGB images from the right-wrist camera at each time step, (TRAJ_LEN, 480, 640, 3)
• action: desired joint positions of the two robot arms at the next time step, (TRAJ_LEN, 14); Note that this is slightly different from the actual joint positions at the next time step

Note: The number in episode_<NUMBER>.hdf5 is not necessarily consecutive. TRAJ_LEN may vary from episode to episode.

Each JSON file corresponds to the annotated language instructions of the task, which has the following keys:

• instruction: original human-annotated instructions; its value is a string
• expanded_instruction: instructions generated by GPT-4-Turbo by expanding the original one; its value is a list of strings
• simplified_instruction: instructions generated by GPT-4-Turbo by simplifying the original one; its value is a list of strings

Citation

If you find our work helpful, please cite us:

@article{liu2024rdt,
  title={RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation},
  author={Liu, Songming and Wu, Lingxuan and Li, Bangguo and Tan, Hengkai and Chen, Huayu and Wang, Zhengyi and Xu, Ke and Su, Hang and Zhu, Jun},
  journal={arXiv preprint arXiv:2410.07864},
  year={2024}
}

Thank you!