README
SpaceThinker Dataset
Usage
Try training a LLaVA-style VLM using the SpaceThinker Dataset
Enhanced Quantitative Spatial Reasoning with Test-Time Compute
The SpaceThinker dataset is created using VQASynth to synthesize spatial reasoning traces from a subset of images in the localized narratives split of the cauldron.
Data Samples
 |  |  |
|---|---|---|
| Prompt: How far is the man in the red hat from the pallet of boxes in feet? | Prompt: How far is the Goal from the penalty taker in this image? | Prompt: Approximately how many meters apart are the chair and bookshelf? |
| Reasoning: To determine how far the man in the red hat is from the pallet of boxes, I need to consider the spatial arrangement in the warehouse. The man is walking on the floor, and there's a visible gap between him and the pallet. First, I'll estimate the size of the pallet. A standard pallet is typically around 4 feet wide and 8 feet long. Given that the man is standing near the edge of the pallet, he might be about 2–3 feet away from the front edge of the pallet. Next, I'll consider the distance from the pallet to the man. The man appears to be walking along the aisle, which seems to be about 6–7 feet wide. If he is standing at the edge of the pallet, he would likely be about 2–3 feet from the front edge of the pallet. Combining these observations, the man in the red hat is approximately 2–3 feet from the pallet of boxes. | Reasoning: First, I observe the scene where a player in a red uniform stands near the goal. The goal itself is a standard soccer goalpost, which typically extends 2.5 meters beyond the line of the goalposts. The player appears to be standing close to the goal, but not directly at the goal line. To estimate the distance, I consider that the player's position relative to the goal suggests he is likely standing closer than 2.5 meters away. Given the typical setup of a penalty kick scenario, the player would be positioned further back from the goal than the goal itself. By visual estimation, the player seems to be standing approximately 10–15 meters from the goal. This range aligns with the common distance for a penalty kick, considering the player's stance and the goal's structure. Thus, the distance from the goal to the penalty taker is likely around 10–15 meters. | Reasoning: To determine the distance between the chair and the bookshelf, I'll consider the typical dimensions of these objects. A standard office chair is around 60–70 cm tall, and a bookshelf can be anywhere from 1.2 to 1.8 meters tall. Given that the chair is likely placed on a desk or floor level, the height difference between the chair and the top of the bookshelf would be approximately 1 meter. Assuming the bookshelf is at least 1 meter high, and the chair is at ground level, the vertical distance between them is indeed 1 meter. The horizontal distance might vary depending on the layout, but given the room's dimensions and the placement of the chair relative to the bookshelf, the total distance between the chair and the top of the bookshelf is likely around 1 meter. Therefore, the chair and bookshelf are approximately 1 meter apart. |
| Answer: 2–3 feet | Answer: 10–15 meters | Answer: 1 meter |
Data Creation
The VQASynth reasoning trace synthesis pipeline relies on a 3D scene graph extracted for each image.
- Describe and localize objects with Molmo point prompting SAM2.
- VGGT is applied to the image for 3D scene reconstruction.
- Synthesize consistent spatial reasoning traces with templated QA pairs as context for LLMs.
The SpaceThinker dataset includes over 12K samples synthesized using VQASynth on a subset of images in the localized narratives split of the cauldron. SpaceThinker is formatted similar to the Llama-Nemotron-Post-Training-Dataset-v1 to toggle reasoning for compatibility with the Llama-3.1-Nemotron-Nano-8B-v1 base LLM.
The model builds upon the ideas from SpatialVLM (Chen et al., 2024), introducing synthetic reasoning traces grounded on a 3D scene reconstruction pipeline using Molmo, VGGT, SAM2.
Dataset Summary
- ~12K synthetic spatial reasoning traces
- Question types: spatial relations (distances (units), above, left-of, contains, closest to)
- Format: image (RGB) + question + answer with reasoning traces
- Dataset: remyxai/SpaceThinker
- Code: Synthetize Spatial Reasoning Traces with VQASynth
🧠 Check out the latest VLM trained using the SpaceThinker dataset: SpaceThinker-Qwen2.5-VL-3B 📖 Read more about thinking VLM
Limitations
- Performance may degrade in cluttered environments or camera perspective.
- This dataset was generated using synthetic reasoning over an internet image dataset.
- Multimodal biases inherent to the underlying dataset (local narratives split of the Cauldron) may persist.
Users are encouraged to evaluate outputs critically and consider adapting the synthesis pipeline for domain-specific safety and performance.
Citation
@misc{VQASynth,
author = {remyxai},
title = {VQASynth},
year = {2024},
note = {GitHub repository},
url = {https://github.com/remyxai/VQASynth/tree/main}
}
Contributors
17 commits
Similar repos
Linked in READMEs
README
SpaceThinker Dataset
Usage
Try training a LLaVA-style VLM using the SpaceThinker Dataset
Enhanced Quantitative Spatial Reasoning with Test-Time Compute
The SpaceThinker dataset is created using VQASynth to synthesize spatial reasoning traces from a subset of images in the localized narratives split of the cauldron.
Data Samples
| | |
|---|---|---|
| Prompt: How far is the man in the red hat from the pallet of boxes in feet? | Prompt: How far is the Goal from the penalty taker in this image? | Prompt: Approximately how many meters apart are the chair and bookshelf? |
| Reasoning: To determine how far the man in the red hat is from the pallet of boxes, I need to consider the spatial arrangement in the warehouse. The man is walking on the floor, and there's a visible gap between him and the pallet. First, I'll estimate the size of the pallet. A standard pallet is typically around 4 feet wide and 8 feet long. Given that the man is standing near the edge of the pallet, he might be about 2–3 feet away from the front edge of the pallet. Next, I'll consider the distance from the pallet to the man. The man appears to be walking along the aisle, which seems to be about 6–7 feet wide. If he is standing at the edge of the pallet, he would likely be about 2–3 feet from the front edge of the pallet. Combining these observations, the man in the red hat is approximately 2–3 feet from the pallet of boxes. | Reasoning: First, I observe the scene where a player in a red uniform stands near the goal. The goal itself is a standard soccer goalpost, which typically extends 2.5 meters beyond the line of the goalposts. The player appears to be standing close to the goal, but not directly at the goal line. To estimate the distance, I consider that the player's position relative to the goal suggests he is likely standing closer than 2.5 meters away. Given the typical setup of a penalty kick scenario, the player would be positioned further back from the goal than the goal itself. By visual estimation, the player seems to be standing approximately 10–15 meters from the goal. This range aligns with the common distance for a penalty kick, considering the player's stance and the goal's structure. Thus, the distance from the goal to the penalty taker is likely around 10–15 meters. | Reasoning: To determine the distance between the chair and the bookshelf, I'll consider the typical dimensions of these objects. A standard office chair is around 60–70 cm tall, and a bookshelf can be anywhere from 1.2 to 1.8 meters tall. Given that the chair is likely placed on a desk or floor level, the height difference between the chair and the top of the bookshelf would be approximately 1 meter. Assuming the bookshelf is at least 1 meter high, and the chair is at ground level, the vertical distance between them is indeed 1 meter. The horizontal distance might vary depending on the layout, but given the room's dimensions and the placement of the chair relative to the bookshelf, the total distance between the chair and the top of the bookshelf is likely around 1 meter. Therefore, the chair and bookshelf are approximately 1 meter apart. |
| Answer: 2–3 feet | Answer: 10–15 meters | Answer: 1 meter |
Data Creation
The VQASynth reasoning trace synthesis pipeline relies on a 3D scene graph extracted for each image.
- Describe and localize objects with Molmo point prompting SAM2.
- VGGT is applied to the image for 3D scene reconstruction.
- Synthesize consistent spatial reasoning traces with templated QA pairs as context for LLMs.
The SpaceThinker dataset includes over 12K samples synthesized using VQASynth on a subset of images in the localized narratives split of the cauldron. SpaceThinker is formatted similar to the Llama-Nemotron-Post-Training-Dataset-v1 to toggle reasoning for compatibility with the Llama-3.1-Nemotron-Nano-8B-v1 base LLM.
The model builds upon the ideas from SpatialVLM (Chen et al., 2024), introducing synthetic reasoning traces grounded on a 3D scene reconstruction pipeline using Molmo, VGGT, SAM2.
Dataset Summary
- ~12K synthetic spatial reasoning traces
- Question types: spatial relations (distances (units), above, left-of, contains, closest to)
- Format: image (RGB) + question + answer with reasoning traces
- Dataset: remyxai/SpaceThinker
- Code: Synthetize Spatial Reasoning Traces with VQASynth
🧠 Check out the latest VLM trained using the SpaceThinker dataset: SpaceThinker-Qwen2.5-VL-3B 📖 Read more about thinking VLM
Limitations
- Performance may degrade in cluttered environments or camera perspective.
- This dataset was generated using synthetic reasoning over an internet image dataset.
- Multimodal biases inherent to the underlying dataset (local narratives split of the Cauldron) may persist.
Users are encouraged to evaluate outputs critically and consider adapting the synthesis pipeline for domain-specific safety and performance.
Citation
@misc{VQASynth,
author = {remyxai},
title = {VQASynth},
year = {2024},
note = {GitHub repository},
url = {https://github.com/remyxai/VQASynth/tree/main}
}
Similar repos
Linked in READMEs
Contributors
17 commits