DynamicVerse: A Physically-Aware Multimodal Framework for 4D World Modeling

Overview

DynamicVerse is an integrated framework for dynamic scene understanding and 4D reconstruction. It combines advanced visual models such as Sa2VA, Qwen-VL, DAM, CameraBench, CoTracker, and UniDepth to achieve end-to-end processing from video to 4D scenes.

This repository hosts the processed datasets used in the DynamicVerse project. These data cover multiple mainstream dynamic scene datasets and have undergone keyframe extraction, multimodal analysis, dense segmentation, and 4D reconstruction processing.

Dataset Statistics

Dataset Structure

The data storage structure of this repository is shown below. Data is classified according to the original dataset source and packaged into ZIP files.

For larger datasets (such as dynpose-100k), the data is distributed across multiple independent ZIP files (e.g., dynpose-0000.zip, dynpose-0001.zip, etc.). Each compressed package contains a specific number of independent Scenes.

└── DynamicVerse
    ├── DAVIS/                  # Processed results for DAVIS dataset
    │   └── DAVIS.zip
    ├── dynamic_replica/        # Processed results for Dynamic Replica dataset
    │   ├── dynamic_replica-0000.zip
    │   ├── dynamic_replica-0001.zip
    │   ├── ...
    │   └── dynamic_replica-0009.zip
    ├── dynpose-100k/           # Processed results for DynPose-100k dataset
    │   ├── dynpose-0000.zip
    │   ├── dynpose-0001.zip
    │   ├── ...
    │   └── dynpose-0089.zip
    ├── MOSE/                   # Processed results for MOSE dataset
    │   ├── MOSE-0000.zip
    │   └── MOSE-0001.zip
    ├── MVS-Synth/              # Processed results for MVS-Synth dataset
    │   └── MVS-Synth.zip
    ├── point_odyssey/          # Processed results for Point Odyssey dataset
    │   ├── point_odyssey-0000.zip
    │   └── point_odyssey-0001.zip
    ├── SAV/                    # Processed results for SAV dataset
    │   └── sav.zip
    ├── spring/                 # Processed results for Spring dataset
    │   └── spring.zip
    ├── uvo/                    # Processed results for UVO dataset
    │   └── uvo.zip
    ├── VOST/                   # Processed results for VOST dataset
    │   └── VOST.zip
    ├── youtube_vis/            # Processed results for YouTube-VIS dataset
    │   └── youtube_vis.zip
    └── README.md

Data Content

After decompressing the above ZIP files, each scene directory contains standardized data as follows:

<scene_id>/
├── camera.npz                  # Camera parameters (poses, intrinsics)
├── captions/                   # Multimodal description files
│   ├── camera_caption.json     # Description of camera motion
│   ├── object_caption.json     # Description of objects
│   └── scene_caption.json      # Overall scene description
├── category/                   # Category information
│   └── category.json
├── depths/                     # Depth map sequence (16-bit PNG, Metric Scale)
│   ├── 00001.png
│   ├── 00002.png
│   └── ... (n png files)
├── mask/                       # Segmentation mask sequence
│   ├── 00001.png
│   ├── 00002.png
│   └── ... (n png files)
└── rgb/                        # RGB image sequence
    ├── 00001.jpg
    ├── 00002.jpg
    └── ... (n jpg files)

Usage

Download and Decompress

You can use the Hugging Face CLI or directly download the required ZIP files. Since the files are independent, you can download parts of the data as needed.

Sample Usage (Processing Pipeline)

The official repository provides the pipeline code for data generation. Here's a quick start guide to set up the environment and run the DynamicGen demo for processing a complete geometric scene pipeline.

1. DynamicVerse Environment

git clone --recurse-submodules https://github.com/Dynamics-X/DynamicVerse.git 
cd DynamicVerse
conda create -n dynamicverse python=3.10
conda activate dynamicverse
bash scripts/install.sh

2. Download Pre-trained Models

bash scripts/download_weights.sh

This script will automatically download the following models:

• CoTracker3 (for motion tracking)
• UniDepth (for depth estimation)
• Sa2VA-8B (multimodal understanding model)
• Qwen2.5-VL-72B-Instruct (vision-language model)(optional)

3. Run DynamicGen Demo

Process a complete geometric scene pipeline:

cd dynamicgen
bash scripts/run_pipeline_demo.sh '' -all

This script executes the following steps:

1. Keyframe Extraction: Motion-aware video keyframe extraction
2. Scene Analysis: Multimodal analysis using Qwen and Sa2VA
3. Segmentation Processing: Generate object masks and organize output
4. 4D Reconstruction (Optional): Complete 4D scene reconstruction using dynamicBA

For more detailed usage and configuration, including local Qwen2.5-VL deployment, refer to the GitHub repository.

Citation

If you find our dataset useful in your research, please citing the following paper:

@misc{wen2025dynamicverse,
        title={DynamicVerse: A Physically-Aware Multimodal Framework for 4D World Modeling}, 
        author={Kairun Wen and Yuzhi Huang and Runyu Chen and Hui Zheng and Yunlong Lin and Panwang Pan and Chenxin Li and Wenyan Cong and Jian Zhang and Junbin Lu and Chenguo Lin and Dilin Wang and Zhicheng Yan and Hongyu Xu and Justin Theiss and Yue Huang and Xinghao Ding and Rakesh Ranjan and Zhiwen Fan},
        year={2025},
        eprint={2512.03000},
        archivePrefix={arXiv},
        primaryClass={cs.CV},
        url={https://arxiv.org/abs/2512.03000}, 
    }

License

Apache-2.0