EgoSPT

EgoSPT is an egocentric manipulation trajectory dataset collected for vision-conditioned trajectory prediction. Each episode contains an RGB video, time-aligned end-effector poses, gripper widths, and valid-frame masks.

Dataset Summary

• Scenes: scene1, scene2, scene3
• Tasks: 112 task folders
• Episodes: 11,515 processed episode folders
• Size: about 37 GB
• Main modality: egocentric RGB video
• Action target: future end-effector trajectory with gripper width

Directory Structure

EgoSPT/
├── scene1/
├── scene2/
└── scene3/
    └── <task_name>/
        └── recording_output_processed/
            └── episode_<id>/
                ├── camera_1.mp4
                ├── pose_interp
                ├── gripper_widths
                └── valid_indices

Each task name follows the pattern:

put_fork<id>_to_<target><id>

where targets include bowls, cups, and plates.

Episode Contents

Each processed episode contains:

The pose trajectory is represented as homogeneous SE(3) transforms. Downstream code converts these poses into relative actions:

[dx, dy, dz, rot6d_0..5, gripper_width]

Usage With umi_day.vision_traj

Place the dataset under:

umi_day/EgoSPT

Then train with:

python -m umi_day.vision_traj.train \
  data.root=umi_day/EgoSPT \
  data.annotations_json=annotations_merged.json

The vision_traj loader expects an annotation JSON that provides object and target bounding boxes on the first frame of each episode.

Notes

• This dataset is intended for robotics research on egocentric perception, object-conditioned manipulation, and trajectory prediction.
• The processed episode folders are directly readable by umi_day/vision_traj/dataset.py.
• See umi_day/vision_traj/README.md for model training and evaluation commands.