4DThinker Training Data

This repository contains the training data for 4DThinker, a framework that enables VLMs to "think with 4D" through dynamic latent mental imagery, built upon SpatialVID and DSR_Suite-Data.

Data Structure

data/
├── dift_data.jsonl              # DIFT training data (~38K samples)
├── 4drl_data_filtered.jsonl     # 4DRL training data (~37K samples)
└── processed_data/              # Video frames & mask overlays
    ├── <video_id>/
    │   ├── frames/              # Extracted video frames (1 FPS)
    │   │   ├── frame_0000.jpg
    │   │   ├── frame_0001.jpg
    │   │   └── ...
    │   ├── masked_static_overlays/   # Static object mask overlays
    │   │   ├── static_f0000.jpg
    │   │   └── ...
    │   └── masked_dynamic_overlays/  # Dynamic object mask overlays
    │       ├── dyn_f0000.jpg
    │       └── ...
    └── ...

Data Formats

DIFT Data (dift_data.jsonl)

Each line is a JSON object with structured CoT reasoning interleaved with mental imagery:

{
  "id": "video_id_question_type",
  "text_input": "Question text with options A/B/C/D",
  "text_output": "<think>reasoning with <output_image> placeholders</think><answer>A. answer</answer>",
  "image_input": ["./data/processed_data/<video_id>/frames/frame_0000.jpg", ...],
  "image_output": ["./data/processed_data/<video_id>/masked_dynamic_overlays/dyn_f0003.jpg", ...]
}

• text_input: Multiple-choice question about camera or object motion
• text_output: Chain-of-thought with <output_image> placeholders representing mental imagery steps
• image_input: Video frames (sampled at 1 FPS) serving as visual context
• image_output: Mask overlay images corresponding to each <output_image> placeholder

4DRL Data (4drl_data_filtered.jsonl)

Each line is a QA-only JSON object (no CoT) for reinforcement learning:

{
  "Type": "abs_dir",
  "Question": "Between 14.1s and 17.7s, ...",
  "A": "option A text",
  "B": "option B text",
  "C": "option C text",
  "D": "option D text",
  "Correct": "A",
  "video_path": "./raw_data/DSR-data/train_video/<video>.mp4"
}

Statistics

Bibtex

If you find 4DThinker helpful for your work, please cite

@article{chen20264dthinker,
  title={4DThinker: Thinking with 4D Imagery for Dynamic Spatial Understanding},
  author={Chen, Zhangquan and Zhang, Manyuan and Yu, Xinlei and An, Xiang and Li, Bo and Xie, Xin and Wang, ZiDong and Sun, Mingze and Chen, Shuang and Li, Hongyu and others},
  journal={arXiv preprint arXiv:2605.05997},
  year={2026}
}

License

Apache License 2.0