Thinking with Camera: A Unified Multimodal Model for Camera-Centric Understanding and Generation

   📖 Project Page  ｜    🖥️ GitHub    |   🤗 Hugging Face   |    📑 Paper   

Dataset Details

Datasets and benchmarks that span vision, language, and camera modalities remain scarce in the domain of spatial multimodal intelligence. To address this gap, we introduce Puffin-4M, a large-scale, high-quality dataset comprising 4 million vision-language-camera triplets. Puffin-4M includes single-view images with precise camera parameters, descriptive captions, pixel-wise camera maps, and spatial reasoning annotations across diverse indoor and outdoor scenarios. Beyond single views, it also incorporates cross-view and aesthetic images, making it a versatile benchmark for both understanding and generation tasks.

---

Dataset Samples

We show the samples of our Puffin-4M for each task (camera-centric generation and understanding, world exploration, spatial imagination, and photographic guidance) as follows.

Directory Structure

DATA_PATH/
├─ training data/
│  ├─ cap_folder/ # captions, including scene descriptions and camera parameters
│  │  ├─ 000000.tar.gz
│  │  └─ ...
│  ├─ cap_folder_cot/ # captions with thinking, including spatial reasoning descriptions and camera parameters
│  │  ├─ 000000.tar.gz
│  │  └─ ...
│  ├─ local_folder/ # images
│  │  ├─ 000000.tar.gz
│  │  └─ ...
│  ├─ summary.json
│  ├─ cross_view/ # instruction tuning data for world exploration and spatial imagination
│  │  ├─ cap_folder/  # captions, including text descriptions and camera parameters
│  │  │  ├─ 000000.tar.gz
│  │  │  └─ ...
│  │  ├─ cap_folder_cam/  # captions, only including camera parameters
│  │  │  ├─ 000000.tar.gz
│  │  │  └─ ...
│  │  ├─ cap_folder_scene/  # captions, only including scene descriptions
│  │  │  ├─ 000000.tar.gz
│  │  │  └─ ...
│  │  ├─ local_folder/ # target views
│  │  │  ├─ 000000.tar.gz
│  │  │  └─ ...
│  │  ├─ local_folder_init/ # initial views
│  │  │  ├─ 000000.tar.gz
│  │  │  └─ ...
│  │  ├─ summary.json
│  ├─ photography/ # instruction tuning data for photographic guidance
│  │  ├─ cap_folder/  # captions, only including camera parameters
│  │  │  ├─ 000000.tar.gz
│  │  ├─ local_folder/  # images
│  │  │  ├─ 000000.tar.gz
│  │  ├─ summary.json
├─ benchmark/
│  ├─ Puffin-Und/
│  │  ├─ images/
│  │  │  ├─ 0000001.jpg
│  │  │  ├─ ...
│  │  ├─ cameras.csv
│  ├─ Puffin-Gen/
│  │  ├─ caption/
│  │  │  ├─ caption_src/
│  │  │  │  ├─ 0000001.json
│  │  │  │  ├─ ...
│  │  │  ├─ caption_degree/
│  │  │  │  ├─ 0000001.json
│  │  │  │  ├─ ...
│  │  │  ├─ caption_photographic_term/
│  │  │  │  ├─ 0000001.json
│  │  │  │  ├─ ...
│  │  ├─ camera/
│  │  │  ├─ 0000001.pt
│  │  │  ├─ ...
│  │  ├─ cameras.csv
└─ README.md

Dataset Download

You can download the entire Puffin-4M dataset using the following command:

hf download KangLiao/Puffin-4M --repo-type dataset

The whole dataset (training data and benchmark) is approximately 449GB in size. Note that we omit the camera maps from the uploaded training data due to their large total size (~3 MB each, amounting to ~11.4 TB in total). However, these maps can be easily generated using the provided script scripts/camera/cam_dataset.py available on our GitHub repository.

Citation

If you find Puffin useful for your research or applications, please cite our paper using the following BibTeX:

  @article{liao2025puffin,
    title={Thinking with Camera: A Unified Multimodal Model for Camera-Centric Understanding and Generation},
    author={Liao, Kang and Wu, Size and Wu, Zhonghua and Jin, Linyi and Wang, Chao and Wang, Yikai and Wang, Fei and Li, Wei and Loy, Chen Change},
    journal={arXiv preprint arXiv:2510.08673},
    year={2025}
  }

License

This project is licensed under NTU S-Lab License 1.0.