Puffin-World: Scaling a Unified Multimodal Model with Native 3D World States

📖 Project Page   |   💻 GitHub   |   🤗 Models   |   📄 HF Paper   |   🤗 HF Blog

Dataset Details

Datasets and benchmarks that span vision, language, and camera modalities remain scarce in the domain of spatial multimodal intelligence. Puffin-16M is a large-scale, camera-centric dataset that substantially scales up Puffin-4M, comprising ~16.5 million samples across two complementary parts:

• Puffin-Cam-15M — 15,338,221 single-view image–caption–camera triplets. Each perspective image (rendered from panoramas across diverse indoor and outdoor scenes) is paired with a caption that combines a scene description, precise camera parameters (roll, pitch, field-of-view, and radial distortion), and a camera-height category.
• Puffin-Traj-1M — 1,136,696 camera trajectories. Each trajectory is a 90-frame sequence with dense per-frame camera annotations (Euler angles, intrinsics, and camera pose), supporting camera-controlled world exploration and trajectory-aware generation and understanding.

Two small held-out benchmarks accompany the training corpus for evaluation, rendered from panoramas disjoint from the training scenes and packaged in the same format as their training counterparts:

• Puffin-Cam-15M-Bench — 600 single-view image–caption–camera pairs (<hash>.jpg ↔ <hash>.json, one shard each). Captions keep the scene-and-camera description; instead of camera_height, each JSON records the image width / height.
• Puffin-Traj-1M-Bench — 100 camera trajectories (90 frames + cameras.json per scene, one shard) with the same per-frame annotation schema as Puffin-Traj-1M.

---

Data Format

Puffin-Cam-15M — captions

Each caption is a JSON file, named by a random hash that matches its paired image (<hash>.jpg ↔ <hash>.json). It keeps the original scene-and-camera caption and adds a camera_height field:

{
  "caption": "The image shows a rocky foreground with a clear blue sky and scattered clouds above ... The camera parameters (roll, pitch, field-of-view, and radial distortion) are: 0.0977, 0.2316, 1.0756, 0.0000.",
  "camera_height": "Eye-level shot"
}

camera_height is one of: Eye-level shot, Low-position shot, High-position shot, Aerial shot, Underwater shot.

Puffin-Traj-1M — trajectories

Each trajectory scene is a folder (named by a random hash) containing its frames and a single cameras.json with per-frame camera annotations:

{
  "motion_type": "pitch", "image_height": 640, "image_width": 640,
  "vfov_deg": 84.47, "focal_px": 352.49, "images_per_segment": 90,
  "frames": [
    {
      "file_path": "000001.jpg",
      "roll_deg": 43.27, "pitch_deg": 44.5, "yaw_deg": 3.38,
      "camera_intrinsics": [[fx, 0, cx], [0, fy, cy], [0, 0, 1]],
      "camera_pose": [[ ... 4x4 camera-to-world ... ]]
    }
  ]
}

---

Camera Parameters and Ranges

Puffin-Cam-15M (single-view)

Each perspective view is rendered from a panorama with camera parameters sampled as:

In each caption, the four numbers following "The camera parameters (roll, pitch, field-of-view, and radial distortion) are:" are given in radians; the radial-distortion term is always 0 under the pinhole model.

The paired camera_height field further categorizes the shooting height into five levels ordered by increasing altitude, spanning underwater scenes up to aerial and space-like views:

The altitude figures in the illustration (e.g. ~10 km, ~50 km, 100 km+) are indicative only and do not define hard numeric cut-offs for the labels.

Puffin-Traj-1M (trajectory)

Each trajectory is a continuous single-axis camera motion rendered from a panorama:

Per frame, cameras.json stores roll_deg / pitch_deg / yaw_deg (degrees), the 3×3 camera_intrinsics, and the 4×4 camera_pose (camera-to-world).

---

Directory Structure

DATA_PATH/
├─ Puffin-Cam-15M/            # 15,338,221 single-view image-caption-camera samples
│  ├─ image_folder/           # perspective images (rendered from panoramas)
│  │  ├─ 000000.zip           # 10,000 images per shard, named by random hash
│  │  └─ ...                  # 000000.zip ... 001533.zip
│  ├─ cap_folder/             # captions: scene description + camera parameters + camera height
│  │  ├─ 000000.zip           # <hash>.json, paired with the same hash in image_folder
│  │  └─ ...
│  ├─ summary.json            # index mapping each image to its caption
├─ Puffin-Cam-15M-Bench/      # held-out benchmark: 600 image-caption-camera pairs
│  ├─ image_folder/000000.zip # 600 images, named by hash
│  └─ cap_folder/000000.zip   # 600 captions, <hash>.json paired with image_folder
├─ Puffin-Traj-1M/            # 1,136,696 camera trajectories (90-frame sequences)
│  ├─ data/
│  │  ├─ 000000.zip           # 500 scenes per shard; each scene: <hash>/000001.jpg ... 000090.jpg + cameras.json
│  │  └─ ...
│  ├─ summary.json            # index of trajectory scenes
├─ Puffin-Traj-1M-Bench/      # held-out benchmark: 100 trajectories
│  └─ data/000001.zip         # 100 scenes; each scene: <hash>/000001.jpg ... 000090.jpg + cameras.json
└─ README.md

Both summary.json files index the (hashed) samples so that images and their annotations can be located within the shard archives.

Dataset Download

You can download the entire Puffin-16M dataset using the following command:

hf download KangLiao/Puffin-16M --repo-type dataset

Puffin-Cam-15M is approximately 2.4 TB, and Puffin-Traj-1M is approximately 9.2 TB (~11.6 TB in total); the two held-out benchmarks are small (~100 MB and ~1.1 GB). You may also download a single part, e.g.:

hf download KangLiao/Puffin-16M --repo-type dataset --include "Puffin-Cam-15M/*"
# benchmarks only
hf download KangLiao/Puffin-16M --repo-type dataset --include "Puffin-Cam-15M-Bench/*" "Puffin-Traj-1M-Bench/*"

Pixel-wise camera maps are omitted due to their large total size, but can be generated from the provided camera parameters using scripts/camera/cam_dataset.py in our GitHub repository.

Citation

If Puffin-16M or Puffin-World supports your research, please cite our paper:

@article{liao2026puffinworld,
  title   = {Puffin-World: Scaling a Unified Multimodal Model with Native 3D World States},
  author  = {Liao, Kang and Luo, Yihang and Wu, Xiao-Ming and Jin, Linyi and Wu, Size and Lin, Chunyu and Zhao, Yao and Wang, Fei and Li, Wei and Loy, Chen Change},
  journal = {arXiv preprint arXiv:2609.04196},
  year    = {2026}
}

License

This project is licensed under NTU S-Lab License 1.0.