VQASynth 🎹

Spatial Reasoning is fundamental to interacting within and navigating physical environments for embodied AI applications like robotics. However, data samples suitable for learning these capabilities are rare in AI pretraining datasets. Don't be limited by what your model can do out-of-the-box, curate any image dataset from the Huggingface Hub for Spatial VQA with tools for 3D scene understanding.

VLMs trained using VQASynth 🎹

• estimate 3D distances between objects in an image
• describe distances colloquially, convert between common units of measurement
• answer queries about the orientation and spatial relationships between objects
• base responses on consistent references like floors and surfaces
• apply CoT "thinking" for more robust reasoning and better estimates

Run a Pipeline on Your Image Dataset

Environment

Before running the demo scripts, ensure you have the following installed:

• Python 3.10 or later
• Docker, Docker Compose V2
• NVIDIA Container Toolkit
• At least 24GB VRAM (A10 or larger) and 16GB RAM

Use Docker Compose to transform Image datasets from Huggingface Hub into VQA datasets describing spatial relations between objects. You can process different datasets after updating the config.yaml.

Then run the spatial VQA pipeline locally with Docker:

# Authenticate to push to hub
huggingface-cli login

# Run the pipeline
cd /path/to/VQASynth
bash run.sh

You can run the colab notebook (requires A100 runtime) or compose the pipeline modules in-process to turn one image into grounded spatial VQAs:

from PIL import Image
from vqasynth.localize import Localizer
from vqasynth.scene_fusion import SpatialSceneConstructor
from vqasynth.prompts import PromptGenerator

image = Image.open("warehouse.jpg").convert("RGB")

# Detect + segment task-relevant objects
masks, _, captions = Localizer(captioner_type="florence").run(image)

# Lift to 3D — VGGT emits per-object point clouds, depth, and intrinsics in one pass
pcd_filepaths, canonicalized, _, _ = SpatialSceneConstructor().run(
    "warehouse_0", image, masks, output_dir="./scenes"
)

# Generate spatial VQAs from the reconstructed 3D scene
qa_pairs = PromptGenerator().run(captions, pcd_filepaths, canonicalized)

# Example output:
#   "How close is the man in red hat walking from the wooden pallet with boxes?"
#   → "Approximately 60.13 centimeters."

The resulting Huggingface dataset is in the cache directory and you can push to hub with:

from vqasynth.datasets import Dataloader
Dataloader(cache_dir).push_to_hub(final_dataset, target_repo_name)

Description

Fusing semantic and metric data into templated VQA chat, Vision Language Models can be instruction-tuned with low-rank adapters to enhance their baseline spatial reasoning capabilities. VQASynth 🎹 provides an open-source reproduction of SpatialVLM, which describes a 3D scene reconstruction pipeline and prompt templates for enhancing the spatial reasoning abilities of VLMs including:

🪄 Object-grounded captions by point prompting with Molmo

📐 Improves metric depth estimation speed & accuracy by replacing DepthPro with VGGT.

🪢 SAM2 replaces SAM in the localization refinement stage.

🧠 Multimodal "Thinking" by CoT Reasoning

Datasets made using VQASynth 🎹

• SpaceOm
• SpaceThinker
• OpenSpaces_MC_R1
• OpenSpaces
• vqasynth_spacellava

Models trained using VQASynth 🎹

• SpaceOm - the best overall
• SpaceThinker-Qwen2.5VL-3B - the most accurate distance estimates
• SpaceQwen2.5-VL-3B-Instruct - 3B - the most popular
• SpaceLLaVA - 13B - the original

Examples from SpaceThinker:

Synthetic spatial reasoning traces suffice to train VLMs capable of grounded, quantitative spatial reasoning—inferring accurate distances, interpreting 3D scene context, and formatting open-ended answers precisely by integrating visual cues, real-world object priors, and human-centric spatial logic.

Model Evaluation

Report VLM spatial reasoning on benchmarks: SpatialScore, OmniSpatial, SpaCE-10, MindCube

Notebooks

We've hosted some notebooks visualizing and experimenting with the techniques included in this repo.

Agent with VQASynth Tools

The VQASynth tool inventory — Florence-2 detection + segmentation, DepthPro/VGGT/FoundationGeo metric depth, Orient-Anything object orientation, NVIDIA DAM regional captioning, MediaPipe pose keypoints, open3d 3D bounding boxes, and OpenCV multi-view correspondence — is exposed as an NOOA-based agent (experiments/nooa_agent/) that composes tool calls dynamically per prompt rather than following a pre-templated pipeline. Useful when the input question isn't known at pipeline-design time.

• Dynamic tool composition — the LLM decides which tools to call and in what order per question. detect_objects → metric_depth → distance_3d for a "how far apart" query; detect_objects → segment → orient → describe_region for camera-facing object descriptions; find_correspondences → depth → detect_3d_boxes for cross-view scene diffs. No pre-coded question types.
• Dense reward annotation — plug into a robot-learning pipeline as a per-frame reward or CoT source. The agent satisfies lerobot's VlmClient protocol via SpatialAnnotatorVlmClient, so it drops into any annotation module (VQA, plan, ECoT) as a tool-grounded alternative to a raw VLM call.
• Traces grounded on tool calls — every annotate() streams a JSONL row in OpenAI-messages format (ready for Qwen2.5-VL / Qwen3-VL fine-tuning) preserving the full tool-call chain, not just the final answer. Reasoning stays auditable.

Quick example:

from PIL import Image
from experiments.nooa_agent.spatial_annotator import SpatialAnnotator, SceneContext
from experiments.nooa_agent.trace import TraceWriter
from nooa.unifiedllm.registry import get_llm_client

agent = SpatialAnnotator(llm=get_llm_client("gemini/gemini-2.5-pro"))
img = Image.open("warehouse.jpg").convert("RGB")

with TraceWriter("traces.jsonl") as writer:
    scene = SceneContext(agent, img, trace_writer=writer, image_ref="warehouse.jpg")
    result = await scene.annotate("How far apart are the two workers in the foreground?")

print(result.answer)            # "The two workers in the foreground are 2.04 meters apart."
print(result.confidence)        # "high"
print(result.tool_calls_used)   # 4

See experiments/nooa_agent/README.md for install, resource tiers (CPU with DepthPro / GPU with VGGT), the full tool inventory, and the lerobot integration path.

References

This project was inspired by or utilizes concepts discussed in the following research paper(s):

@article{chen2024spatialvlm,
  title = {SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities},
  author = {Chen, Boyuan and Xu, Zhuo and Kirmani, Sean and Ichter, Brian and Driess, Danny and Florence, Pete and Sadigh, Dorsa and Guibas, Leonidas and Xia, Fei},
  journal = {arXiv preprint arXiv:2401.12168},
  year = {2024},
  url = {https://arxiv.org/abs/2401.12168},
}
@inproceedings{wang2025vggt,
  title={VGGT: Visual Geometry Grounded Transformer},
  author={Wang, Jianyuan and Chen, Minghao and Karaev, Nikita and Vedaldi, Andrea and Rupprecht, Christian and Novotny, David},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year={2025}
}
@article{ravi2024sam2,
  title={SAM 2: Segment Anything in Images and Videos},
  author={Ravi, Nikhila and Gabeur, Valentin and Hu, Yuan-Ting and Hu, Ronghang and Ryali, Chaitanya and Ma, Tengyu and Khedr, Haitham and R{\"a}dle, Roman and Rolland, Chloe and Gustafson, Laura and Mintun, Eric and Pan, Junting and Alwala, Kalyan Vasudev and Carion, Nicolas and Wu, Chao-Yuan and Girshick, Ross and Doll{\'a}r, Piotr and Feichtenhofer, Christoph},
  journal={arXiv preprint arXiv:2408.00714},
  url={https://arxiv.org/abs/2408.00714},
  year={2024}
}