PhysInOne is a large-scale synthetic dataset for visual physics learning and reasoning, containing 153,810 dynamic 3D scenes and 2 million annotated videos across 71 physical phenomena in mechanics, optics, fluid dynamics, and magnetism. Each scene features complex multi-object and multi-physics interactions with rich annotations, including RGB videos, depth maps, object masks, 3D trajectories, camera poses, object meshes, material properties, and textual descriptions. The dataset supports research on physics-aware video generation, future frame prediction, physical property estimation, motion transfer, physical reasoning, and world models.

📦 Dataset Repositories & Downloads

Due to the large scale of PhysInOne, the rendered data and annotations are split across 16 Hugging Face repositories. Each entry shows the shard size, release status, live all-time downloads, live downloads in the last 30 days, and its repository link.

Combined snapshot (Sep 5, 2026): P01–P16 have 986,449 all-time downloads and 615,484 downloads in the last 30 days. Including the main repository, the per-repository sums are 999,927 and 616,472.

Main repositoryMainOpen ↗

PhysInOneP01Train · 01/16CompleteOpen ↗

4.52 TB

PhysInOneP02Train · 02/16CompleteOpen ↗

7.09 TB

PhysInOneP03Train · 03/16CompleteOpen ↗

7.59 TB

PhysInOneP04Train · 04/16CompleteOpen ↗

7.58 TB

PhysInOneP05Train · 05/16CompleteOpen ↗

7.16 TB

PhysInOneP06Train · 06/16CompleteOpen ↗

7.19 TB

PhysInOneP07Train · 07/16CompleteOpen ↗

7.21 TB

PhysInOneP08Train · 08/16CompleteOpen ↗

7.21 TB

PhysInOneP09Train · 09/16CompleteOpen ↗

7.20 TB

PhysInOneP10Train · 10/16CompleteOpen ↗

7.25 TB

PhysInOneP11Train · 11/16CompleteOpen ↗

7.48 TB

PhysInOneP12Train · 12/16CompleteOpen ↗

6.68 TB

PhysInOneP13Train · 13/16CompleteOpen ↗

6.66 TB

PhysInOneP14Train · 14/16CompleteOpen ↗

6.72 TB

PhysInOneP15Train · 15/16CompleteOpen ↗

7.93 TB

PhysInOneP16Train · 16/16CompleteOpen ↗

1.57 TB

Download badges query the official Hugging Face API and update automatically. Counts are repository-level download events, not deduplicated users; accessing multiple shards can produce one event in each shard.

For large-scale downloading and filtering, please refer to the How to Download section below.

PhysInOne is divided into Train, Val, and Test splits, containing 122,988, 15,411, and 15,411 scenes, respectively. The three splits are generated with completely distinct 3D meshes and backgrounds, ensuring that no scene or visual content is shared across splits. This separation provides a robust benchmark for evaluating generalization and physical reasoning.

The first public 3D asset release is now available and contains Unreal Engine project resources for 1,000 validation scenes. The current release includes 4,299 files (approximately 22.25 GiB), including eight archives that must be extracted in place while preserving their project-relative paths. Full training asset archives are not part of this validation release.

• Browse the PhysicBenchmark project
• Download and setup guide
• Automated downloader
• Exact validation release list

Install Unreal Engine 5.5.4. Windows is recommended for the simplest setup; Linux is also supported with additional platform configuration. The automated downloader retrieves the exact release, safely extracts all eight ZIP files into their correct parent directories, preserves the Unreal Engine package hierarchy, supports interrupted-run recovery, and records detailed logs.

pip install -U huggingface_hub tqdm

hf download vLAR/PhysInOne \
  "PhysInOne Utils/scripts/download_3d_assets.py" \
  --repo-type dataset \
  --local-dir .

python "PhysInOne Utils/scripts/download_3d_assets.py" \
  --output-dir ./PhysInOne-3D-Assets

After download and extraction, open:

PhysInOne-3D-Assets/3D Assets/PhysicBenchmark/PhysInOne.uproject

Do not rename or flatten directories such as Content/PhysInOne/, Content/__ExternalActors__/, or Content/__ExternalObjects__/; Unreal Engine asset references depend on the exact project-relative structure. See the full setup guide for manual extraction paths, disk-space guidance, resume behavior, and troubleshooting.

All user-facing evaluation inputs required for the four PhysInOne Leaderboard tasks have been released. Ground-truth targets remain private and are not included in the public release. Exact task-specific file lists and a resumable Python downloader allow users to download one task, multiple tasks, or individual scenes without scanning unrelated dataset shards.

Install the downloader dependency and download any task as follows:

pip install -U huggingface_hub tqdm

hf download vLAR/PhysInOne \
  "PhysInOne Utils/scripts/download_leaderboard.py" \
  "PhysInOne Utils/leaderboard_lists/video-generation.txt" \
  "PhysInOne Utils/leaderboard_lists/future-prediction.txt" \
  "PhysInOne Utils/leaderboard_lists/physical-properties-estimation.txt" \
  "PhysInOne Utils/leaderboard_lists/motion-transfer.txt" \
  "PhysInOne Utils/LEADERBOARD_DOWNLOAD.md" \
  --repo-type dataset \
  --local-dir .

python "PhysInOne Utils/scripts/download_leaderboard.py" \
  --task future-prediction \
  --output-dir ./PhysInOne-Leaderboard

Use --task all to download all four tasks, --scene SCENE_ID to select individual scenes, or --extract to validate and extract scene ZIPs. Downloads are resumable, and every run records successes and failures for review.

See the full Leaderboard download guide and the task file lists for details.

Please download the PhysInOne Utils folder from the main dataset repository. It contains:

PhysInOne Utils/
├── metadata/
│   ├── repo_assignment.txt
│   └── repo_map.json
├── 3d_assets_lists/
│   ├── validation.txt
│   ├── validation_archives.txt
│   └── summary.json
├── leaderboard_lists/
│   ├── video-generation.txt
│   ├── future-prediction.txt
│   ├── physical-properties-estimation.txt
│   └── motion-transfer.txt
├── 3D_ASSETS_DOWNLOAD.md
├── LEADERBOARD_DOWNLOAD.md
└── scripts/
    ├── filter_cases.py
    ├── download.py
    ├── download_3d_assets.py
    └── download_leaderboard.py

Run the following commands from inside the downloaded folder so the scripts can find metadata/ using their default paths:

cd "PhysInOne Utils"

PhysInOne is distributed across multiple Hugging Face dataset repositories because of its large scale. We provide utilities for selecting and downloading full-dataset cases and Leaderboard evaluation data:

• filter_cases.py: exports selected cases to JSON.
• download.py: downloads the selected case zip files from the corresponding repositories.
• download_leaderboard.py: downloads public evaluation data for the four Leaderboard tasks; see LEADERBOARD_DOWNLOAD.md.
• download_3d_assets.py: downloads and safely assembles the validation Unreal Engine project; see 3D_ASSETS_DOWNLOAD.md.

pip install huggingface_hub tqdm

Export a JSON file containing case information matching your filter criteria, for use in subsequent download scripts. The filter_cases.py script supports selection by:

• Split: train, val, test
• Activity complexity: single, double, triple
• Physical phenomenon abbreviation: MovingHitsFixed, FrictionStop, LiquidTension, ..., GranularFall
• Number of cases: globally sample num cases after filtering

The following examples show common filtering workflows.

python scripts/filter_cases.py \
  --split train \
  --output selected_cases.json

python scripts/filter_cases.py \
  --split train \
  --activity_type double \
  --output selected_cases.json

By default, phenomenon matching uses contains mode. For example, the following command selects all double-physics cases that contain FrictionStop(For all physical phenomena and their abbreviations, please refer to the Data Fields section below.):

python scripts/filter_cases.py \
  --split train \
  --activity_type double \
  --phenomena FrictionStop \
  --match_mode contains \
  --output selected_cases.json

For exact matching, use --match_mode exact. Order does not matter. For example, this command selects cases whose phenomenon set is exactly {AccelConcaveSpin, AccelSurfaceSpin}:

python scripts/filter_cases.py \
  --split train \
  --activity_type double \
  --phenomena AccelConcaveSpin AccelSurfaceSpin \
  --match_mode exact \
  --output selected_cases.json

--num is the global number of cases sampled after filtering (This is because the number of cases meeting the filtering criteria may be much larger than the number you need.). The default random seed is 42.

python scripts/filter_cases.py \
  --split train \
  --activity_type double \
  --phenomena FrictionStop \
  --num 3000 \
  --seed 42 \
  --output selected_cases.json

Each selected case is downloaded as a zip file from the corresponding Hugging Face shard repository.

python scripts/download.py \
  --selection selected_cases.json \
  --output_dir ./PhysInOne

If the dataset repositories are gated or private, pass a Hugging Face token:

python scripts/download.py \
  --selection selected_cases.json \
  --output_dir ./PhysInOne \
  --token YOUR_HF_TOKEN

By default, downloaded zip files are saved into a flat output folder. To preserve shard/split/activity structure:

python scripts/download.py \
  --selection selected_cases.json \
  --output_dir ./PhysInOne \
  --keep_shard_structure

The preserved structure is:

PhysInOne/
  physinone_part1/
    Train/
      DoublePhysics/
        AccelConcaveSpin_AccelSurfaceSpin__bg070__K5ER39_trajectory.zip

AccelConcaveSpin__bg010__m3Zrtz_trajectory/
├── AccelConcaveSpin__bg010__m3Zrtz_trajectory.json
├── blender_CineCamera_0.json
├── blender_CineCamera_1.json
├── blender_CineCamera_2.json
├── blender_CineCamera_3.json
├── blender_CineCamera_5.json
├── blender_CineCamera_6.json
├── blender_CineCamera_7.json
├── blender_CineCamera_8.json
├── blender_CineCamera_9.json
├── blender_CineCamera_10.json
├── blender_CineCamera_11.json
├── blender_CineCamera_12.json
├── blender_CineCamera_Moving.json
├── cameras.ply
├── caption.txt
├── points3d.ply
├── recorder_stats.json
├── static_camera_list.txt
├── CineCamera_0/
│   ├── depth/  (0000.npz ~ 0089.npz, 90 files)
│   ├── rgb/    (0000.jpg ~ 0089.jpg, 90 files)
│   └── seg/    (0000.npz ~ 0089.npz, 90 files)
├── CineCamera_1/
│   ...
├── CineCamera_2/
│   ...
├── CineCamera_3/
│   ...
├── CineCamera_5/
│   ...
├── CineCamera_6/
│   ...
├── CineCamera_7/
│   ...
├── CineCamera_8/
│   ...
├── CineCamera_9/
│   ...
├── CineCamera_10/
│   ...
├── CineCamera_11/
│   ...
├── CineCamera_12/
│   ...
└── CineCamera_Moving/
    ├── depth/  (0000.npz ~ 0089.npz, 90 files)
    ├── rgb/    (0000.jpg ~ 0089.jpg, 90 files)
    └── seg/    (0000.npz ~ 0089.npz, 90 files)

PhysInOne provides synchronized visual and physical annotations for each dynamic 3D scene.

•Depth

•Segmentation

•Captions

•Trajectories

•Cameras

•Point Clouds

•Physical Phenomenon and Abbreviations

All 3D assets and materials included in PhysInOne have been sourced from publicly available platforms and verified to carry licenses compatible with non-commercial use. These include:

• SketchFab: assets under various licenses, verified that AI-related usage is allowed.
• Fab: assets under CC BY or Unreal Engine Standard License, explicitly permitting AI-related usage.
• BlenderKit: distributed under Royalty-Free (RF) license.
• ShareTextures: textures under CC0 license.

In total, assets comply with licenses including CC BY-NC, CC BY-SA, CC BY-NC-SA, CC0, CC BY, and RF, ensuring all files can be legally used for building a non-commercial dataset. Users must adhere to the original licenses for any redistribution or derivative work.

If you use PhysInOne in your research, please cite:

@misc{zhou2026physinonevisualphysicslearning,
      title={PhysInOne: Visual Physics Learning and Reasoning in One Suite}, 
      author={Siyuan Zhou and Hejun Wang and Hu Cheng and Jinxi Li and Dongsheng Wang and Junwei Jiang and Yixiao Jin and Jiayue Huang and Shiwei Mao and Shangjia Liu and Yafei Yang and Hongkang Song and Shenxing Wei and Zihui Zhang and Peng Huang and Shijie Liu and Zhengli Hao and Hao Li and Yitian Li and Wenqi Zhou and Zhihan Zhao and Zongqi He and Hongtao Wen and Shouwang Huang and Peng Yun and Bowen Cheng and Pok Kazaf Fu and Wai Kit Lai and Jiahao Chen and Kaiyuan Wang and Zhixuan Sun and Ziqi Li and Haochen Hu and Di Zhang and Chun Ho Yuen and Bing Wang and Zhihua Wang and Chuhang Zou and Bo Yang},
      year={2026},
      eprint={2604.09415},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2604.09415}, 
}

For questions about the dataset, please contact:

• {siyuan.zhou, hejun.wang, hu123.cheng, jinxi.li}@connect.polyu.hk, bo.yang@polyu.edu.hk