SpatialMosaic: A Multi-View VLM Dataset for Partial Visibility

Description

SpatialMosaic is a multi-view visual question answering dataset for evaluating spatial reasoning under partial visibility, occlusion, and low-overlap views. It pairs indoor ScanNet++ and outdoor Waymo scene references with multi-frame VQA annotations. Questions require models to combine fragmented evidence across 2-5 views, rather than answering from a single image. The tasks cover object counting, object presence, localization, best-view selection, object-object spatial relations, and view-specific position reasoning.

This Hugging Face repository contains annotation files only. It does not redistribute ScanNet++, Waymo, image, video, depth, or scene data. Users must obtain any required source images from the official dataset providers and comply with their licenses, access requirements, and usage terms.

The current configuration contains 2,200 test examples: 1,100 indoor ScanNet++ examples and 1,100 outdoor Waymo examples. Across the two JSON files, the annotations include 17 question_type labels, 2-5 referenced frames per example, 169 indoor ScanNet++ scenes, and 9 outdoor Waymo scenes. Evaluation is performed by exact multiple-choice answer matching against mc_answer.

Available Files and Splits

This card is configured for two annotation-only test splits: one indoor split based on ScanNet++ scene references and one outdoor split based on Waymo scene references.

Loading

from datasets import load_dataset

ds = load_dataset("jmkey/spatial_mosaic_vqa", "default")
print(ds)
print(ds["indoor_test"][0])
print(ds["outdoor_test"][0])

For direct local use of the raw JSON files:

import json

with open("merged_indoor_test.json", "r") as f:
    indoor_rows = json.load(f)

with open("merged_outdoor_test.json", "r") as f:
    outdoor_rows = json.load(f)

Data Format

Each entry contains the referenced source dataset, scene name, frame identifiers, question type, question text, answer options, and the correct multiple-choice answer in mc_answer.

Common fields include:

• dataset: source dataset identifier, such as scannetpp or waymo
• scene_name: source scene or segment identifier
• frames: list of 2-5 frame identifiers used by the question
• question_type: task subtype label
• question: natural-language question
• options: multiple-choice answer candidates
• mc_answer: correct option label

Some entries include additional metadata such as ground_truth, bbox_2d, bbox_2d_diag, GT Scenario, overlap_avg, occlusion_avg, occ_level, overlap_level, and vis_level.

Intended Use

SpatialMosaic is intended for non-commercial research on:

• multi-view visual question answering
• spatial reasoning under partial visibility
• occlusion-aware and low-overlap view understanding
• VLM evaluation and instruction-tuning on multi-frame inputs

Out-of-scope uses include commercial use under this annotation license, redistribution of restricted source datasets, and treating these annotations as a substitute for the original ScanNet++ or Waymo data.

Evaluation

SpatialMosaic uses multiple-choice evaluation. The primary metric is exact-match accuracy against mc_answer.

accuracy = sum(pred == row["mc_answer"] for pred, row in zip(predictions, rows)) / len(rows)

Citation

If you use SpatialMosaic, cite the paper:

@article{lee2025spatialmosaic,
  title={SpatialMosaic: A Multiview VLM Dataset for Partial Visibility},
  author={Lee, Kanghee and Lee, Injae and Kwak, Minseok and Hong, Jungi and Ryu, Kwonyoung and Park, Jaesik},
  journal={arXiv preprint arXiv:2512.23365},
  year={2025}
}

License

The released VQA annotations are licensed under CC BY-NC 4.0. This license applies only to the annotations. Source images, videos, depth, and scene data remain governed by the original ScanNet++ and Waymo licenses and terms.