Physical AI Bench - Conditional Generation

Paper | Code

This dataset (Phsical AI benchmark, PAI-Bench) consisting of 600 examples across three key scenarios: robotic arm operations, driving, and ego-centric everyday life scenes, each representing a critical aspect of Physical AI. This dataset is constructed by sampling a number of videos from three different datasets. The specific details are provided below.

Dataset Summary

• Dataset Size: 600 video samples
• Video Format: MP4 files with various processing variants
• Annotations: Text captions for each video
• Processing Variants: Blur, Canny edge detection, Depth estimation, SAM2 segmentation

File Organization

physical-ai-bench-transfer/
├── videos/          # Original video files
├── blur/            # Blur-processed videos
├── canny/           # Edge detection videos
├── depth_vids/      # Depth estimation videos
├── depth_npzs/      # Depth estimation numpy arrays
├── sam2_vids/       # SAM2 segmentation videos
├── sam2_pkls/       # SAM2 segmentation pickle files
└── captions/        # JSON files with video descriptions

Citation

If you use Physical AI Bench in your research, please cite:

@misc{zhou2025paibenchcomprehensivebenchmarkphysical,
      title={PAI-Bench: A Comprehensive Benchmark For Physical AI}, 
      author={Fengzhe Zhou and Jiannan Huang and Jialuo Li and Deva Ramanan and Humphrey Shi},
      year={2025},
      eprint={2512.01989},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2512.01989}, 
}