[T-PAMI'26] Uni-Hand: Universal Hand Motion Forecasting in Egocentric Views

Junyi Ma¹, Wentao Bao², Jingyi Xu¹, Guanzhong Sun³, Yu Zheng¹, Erhang Zhang¹, Xieyuanli Chen⁴, Hesheng Wang^1*

¹ Shanghai Jiao Tong University
² Meta Reality Labs
³ China University of Mining and Technology
⁴ National University of Defense Technology

[Paper][Project Page][Code][Preliminary Version]

Human Videos are All You Need!

In this repository, we demonstrate how to train Uni-Hand using only human demonstration videos and leverage the trained model to generate end-effector trajectories for robotic manipulation. To evaluate our method on other public datasets (e.g., EgoPAT3D), please refer to our preliminary work.

If any bugs are spotted or any download links are broken, please do not hesitate to make a PR or open an issue. We would be glad if Uni-Hand could be useful as a codebase for your own research and development.

📣 News

• [2026-05] Add V-JEPA 2 support to Uni-Hand.
• [2026-05] Uni-Hand is accepted by T-PAMI! 🎉
• [2026-03] Add a DiT-based trajectory diffusion variant for Uni-Hand.
• [2026-03] Release Uni-Hand's depth input from Depth Anything 3.
• [2026-02] Add SAM-3D-Body support to Uni-Hand.
• [2025-12] Release the initial version of Uni-Hand.

📚 Contents

• Install Uni-Hand
  • Prepare a Conda Environment
  • Clone HaMeR/SAM-3D-Body and DINOv2/V-JEPA 2 (Optional)
• Prepare Human Video Data
  • Recommended Data Structure
  • Extract and Refine 3D Hand Trajectories
  • Label Hand-Object Contact and Separation Timestamps
  • Extract Vision Features
• Run Uni-Hand
  • Train and Evaluate
  • Visualize Predicted End-Effector Trajectories
• Download Human Videos and Pretrained Models
  • Datasets
  • Uni-Hand Pretrained Checkpoints
• Deploy on Your Own Robot
• Cite Our Work
• License

🛠️ Install Uni-Hand

Prepare a Conda Environment

First, clone Uni-Hand:

git clone https://github.com/IRMVLab/UniHand
cd UniHand

Create and activate a new conda environment:

conda create -n unihand python=3.10
conda activate unihand

Install dependencies:

pip install -r requirements.txt

Clone HaMeR/SAM-3D-Body and DINOv2/V-JEPA 2 (Optional)

Since we use HaMeR for hand motion extraction and DINOv2 for vision feature extraction, we need to clone them in this project. We also recommend using SAM-3D-Body and V-JEPA 2.

git clone https://github.com/geopavlakos/hamer.git
# install HaMeR following its instruction
# replace hamer/datasets/vitdet_dataset.py with preprocess_human_video/vitdet_dataset.py in our repo

# Optional
git clone https://github.com/facebookresearch/sam-3d-body.git
# install sam-3d-body following its instruction
# replace sam-3d-body/sam_3d_body_estimator.py with preprocess_human_video/sam_3d_body_estimator.py in our repo

git clone https://github.com/facebookresearch/dinov2.git
# install DINOv2 following its instruction
# Model: facebook/dinov2-base

# Optional
git clone https://github.com/facebookresearch/vjepa2.git
# install V-JEPA 2 following its instruction
# Model: ./weights/vitg-384.pt

• ⚡ Alternatively, you can directly download our preprocessed data (hand trajectories + vision features) for our toy dataset here.
• Uni-Hand supports vision features generated by any other visual foundation models. Please update the input_dims of glip_encoder in model.yaml if the feature vector dimension is different from DINOv2's. In this repo, we have released pretrained models with DINOv2 and V-JEPA 2.

🎥 Prepare Human Video Data

Recommended Data Structure

./UniHand
    |-- human_video_data
        |-- 2025-0723-07-17-46
        |-- 2025-0723-07-17-52
        |-- 2025-0723-07-17-59
            |-- depth
                |-- 000000.npy
                |-- 000001.npy
                |-- ...
            |-- rgb
                |-- 000000.npy
                |-- 000001.npy
                |-- ...
            |-- 2025-0723-07-17-59_point_cloud.ply
    |-- hand_keypoints
    |-- hand_trajs
    |-- vision_features

• We have provided the toy dataset (100 human pick-and-place videos) here, which was recorded by a RealSense LiDAR Camera L515. We plan to release more human manipulation videos in the future.
• The .ply file will be generated by the following scripts automatically.
• You can collect your own dataset with one RGBD camera following the data structure. Just sit in front of your robot and hit record—it's that easy!

Extract and Refine 3D Hand Trajectories

First, we extract raw 3D hand trajectories from human videos, with the help of HaMeR or SAM-3D-Body.

# with HaMeR
cp preprocess_human_video/extract_hand_keypoints_hamer.py ./hamer
cd hamer
python extract_hand_keypoints_hamer.py \
    --img_folder ../human_video_data \
    --out_folder ../hand_keypoints 

# or using SAM-3D-Body
cp preprocess_human_video/extract_hand_keypoints_sam3d_body.py ./sam-3d-body
cd sam-3d-body
python extract_hand_keypoints_sam3d_body.py \
    --img_folder ../human_video_data \
    --out_folder ../hand_keypoints_sam3d_crop \
    --sam3d_checkpoint ../sam-3d-body/checkpoints/sam-3d-body-dinov3/model.ckpt \
    --sam3d_mhr_path ../sam-3d-body/checkpoints/sam-3d-body-dinov3/assets/mhr_model.pt \
    --draw_on_crop \
    --save_bbox \
    --seq_order name \
    --draw_on_full

# extract trajs
cd ../preprocess_human_video
python generate_hand_trajs.py \
    --input_root ../human_video_data \
    --keypoint_root ../hand_keypoints \
    --output_root ../hand_trajs

The point cloud files have been generated automatically. Then, we clean the hand trajectories for better training performance:

python clean_hand_data.py \
    --gt_paths ../hand_trajs \
    --joint_idx 0

You can indicate the joint index you want to visualize.

⚡ Alternatively, you can directly use our preprocessed hand motion data here.

Label Hand-Object Contact and Separation Timestamps

• Contact/separation timestamps are crucial for training Uni-Hand, since trajectory data alone is insufficient to generate the gripper’s grasping actions. Please refer to our paper for more details.
• ⚡ We have provided the contact/separation labels of the toy dataset under the unihand/data_utils folder. You can manually label the timings of your own videos and organize them as date_folders.csv.
• You can also try our recent work, EgoLoc, for autonomous temporal interaction localization.

Extract Vision Features

Here we use DINOv2 to extract vision features, while you can also replace it with any other visual foundation models.

cp preprocess_human_video/extract_visual_features_dinov2.py ./dinov2
cd dinov2
python extract_visual_features_dinov2.py \
    --input_root ../human_video_data  \
    --output_root ../vision_features \
    --interval 1

We also provide the script for V-JEPA 2 feature extraction. Please set input_dims of glip_encoder in model.yaml to 1408.

cp preprocess_human_video/extract_visual_features_vjepa2.py ./vjepa2
cd vjepa2
python extract_visual_features_vjepa2.py \
    --input_root ../human_video_data  \
    --output_root ../vision_features \
    --interval 1

⚡ Alternatively, you can directly use our pre-extracted features: DINOv2, V-JEPA 2 for quick reproduction.

🚀 Run Uni-Hand

Train and Evaluate

To train Uni-Hand, set evaluate to false in unihand/configs/traineval.yaml, and run:

cd unihand
bash run_unihand.sh

• After training, you can evaluate the trained model by setting evaluate to true in unihand/configs/traineval.yaml and run the same command. You can set gap to 1 for better results.
• You can resume training from or evaluate a checkpoint by setting resume in unihand/configs/traineval.yaml. If you evaluate your model trained from scratch, set use_os_weights to false and set resume to a non-existent path.
• ⚡ We provide the pretrained model of Uni-Hand here. Please set use_os_weights to true and test it.
• Uni-Hand is robust to background variations thanks to the use of depth information.

Visualize Predicted End-Effector Trajectories

After evaluation, you can visualize end-effector trajectories. As in this demo, we predict hand wrist trajectories for robotic manipulation, and a heuristic grasp offset is used to convert the predicted wrist waypoints to end-effector trajectories.

cd unihand
python viz_predicted_trajs.py

⚡ We have provided the predicted trajectories for the toy dataset here.

📦 Download Human Videos and Pretrained Models

Datasets

Uni-Hand Pretrained Checkpoints

• We have put the train/test splits and contact/separation labels of the pick-and-place task under the unihand/data_utils folder. For the open-door task, please refer to this link.
• We empirically found that SAM-3D-Body produces jittered trajectory labels with its hand branch in our cases, leading to worse predictions. We are figuring out a smooth mechanism to deal with this issue.
• The pseudo depth estimated by Depth Anything 3 is also provided. Feel free to use it as alternative depth input of Uni-Hand.
• We sincerely appreciate Yutong Dong for debugging Uni-Hand and developing the SAM-3D-Body version.
• We would also like to thank Yunji Feng for his efforts in developing a DiT-based trajectory diffusion variant for Uni-Hand. Please refer to this branch for more details.

🤖 Deploy on Your Own Robot

We have demonstrated the deployment of Uni-Hand on real robots. Please refer to our project page and paper for more details.

Let your robot’s end-effector follow the trajectories predicted by Uni-Hand! :robot:

This is the initial version of Uni-Hand, which has already provided an out-of-the-box paradigm for human-video-based imitation learning. We will provide a more comprehensive version with multimodal inputs and additional downstream tasks, and relax the heuristic grasp assumption in four months.

🙏🏻 Cite Our Work

If you find our work helpful to your research, we would appreciate it if you could cite our paper:

@article{ma2026unihand,
    title={Uni-Hand: Universal Hand Motion Forecasting in Egocentric Views},
    author={Junyi Ma and Wentao Bao and Jingyi Xu and Guanzhong Sun and Yu Zheng and Erhang Zhang and Xieyuanli Chen and Hesheng Wang},
    journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
    year={2026},
}

@INPROCEEDINGS{ma2025mmtwin,
    author={Ma, Junyi and Bao, Wentao and Xu, Jingyi and Sun, Guanzhong and Chen, Xieyuanli and Wang, Hesheng},
    booktitle={2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
    title={Novel Diffusion Models for Multimodal 3D Hand Trajectory Prediction},
    year={2025},
    pages={2408-2415},
    doi={10.1109/IROS60139.2025.11246803}}

@article{ma2025madiff,
    title={MADiff: Motion-Aware Mamba Diffusion Models for Hand Trajectory Prediction on Egocentric Videos},
    author={Junyi Ma and Xieyuanli Chen and Wentao Bao and Jingyi Xu and Hesheng Wang},
    journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
    year={2025},
}

📄 License

This project is free software made available under the MIT License. For details see the LICENSE file.