⚠️ Warning: The Dataset Viewer and Data Studio above are for display only. They show just 800 samples from the full RefSpatial dataset, taken from the "SubsetVisualization" folder in Hugging Face ".parquet" format.
24
280 commits
2 linked in READMEs
updated Feb 4, 2026
RefSpatial: A Large-scale Dataset for teaching a general VLM to achieve spatial referring with reasoning
RefSpatial is a comprehensive dataset combining 2D images from OpenImages, 3D videos from CA-1M, and simulated data generated from Blender (Fig. 1 (a)). Its key features include:
Figure 1: RefSpatial dataset pipeline and statistics of 31 spatial relations.
In Fig. 1, we present the dataset recipe that progressively integrates 2D, 3D, and simulated data to enable general VLMs to adapt to spatial referring tasks in a bottom-up manner. For more details of our dataset pipeline, please refer to our paper RoboRefer.
After downloading all the data from Hugging Face, your RefSpatial folder will look like this. Large files (like 2D images) are split into parts (.part_a, etc.), while others are single .tar.gz files.
RefSpatial/
├── 2D/
│ ├── image/
│ │ ├── image.tar.gz.part_a
│ │ └── ... (more split files)
│ ├── depth/
│ │ ├── depth.tar.gz.part_a
│ │ └── ... (more split files)
│ ├── choice_qa.json
│ └── reasoning_template_qa.json
├── 3D/
│ ├── image/
│ │ └── image.tar.gz
│ ├── depth/
│ │ └── depth.tar.gz
│ ├── image_multi_view/
│ │ └── image_multi_view.tar.gz
│ ├── depth_multi_view/
│ │ └── depth_multi_view.tar.gz
│ ├── image_visual_choice/
│ │ ├── image_visual_choice.tar.gz.part_a
│ │ └── ... (more split files)
│ ├── choice_qa.json
│ ├── multi_view_qa.json
│ ├── reasoning_template_qa.json
│ ├── vacant_qa.json
│ └── visual_choice_qa.json
├── Simulator/
│ ├── image/
│ │ └── image.tar.gz
│ ├── depth/
│ │ └── depth.tar.gz
│ └── metadata.json
├── SubsetVisualization/
│ ├── 2D-00000-of-00001.parquet
│ └── ... (more .parquet files)
├── unzip_dataset.sh
└── delete_tar_gz.sh
2D/: Contains all data related to 2D spatial reasoning tasks.
image/, depth/: Contain the split archives for 2D images and their corresponding depth maps.*.json: The relevant question-answering and reasoning annotations.3D/: Contains data for more complex 3D scene understanding tasks.
image/, depth/: Standard single-view 3D scene images and depth maps.image_multi_view/, depth_multi_view/: Data for multi-view tasks.image_visual_choice/: Data for visual choice question tasks; its larger size requires it to be split into archives.*.json: Annotations for the various 3D tasks.Simulator/: Contains data generated from a simulation environment.
image/, depth/: Images and depth maps generated by the simulator, which are perfectly aligned and annotated.metadata.json: Metadata for the scenes.SubsetVisualization/: Contains subset samples for quick visualization and data inspection.
*.parquet: These files preview a small part of the dataset without needing to unzip everything.The provided unzip_dataset.sh script could decompress all of the *.tar.gz files. Please run it from the RefSpatial root directory.
cd RefSpatial
bash unzip_dataset.sh
This script will automatically perform the following actions:
.part_a, .part_b, etc., the script will use the cat command to combine them into a single, complete .tar.gz file. For example, image.tar.gz.part_a, ... will be merged into image.tar.gz.tar command to extract all .tar.gz archives into their current directories.If you wish to delete all .tar.gz and .part_* files after successful decompression to save disk space, you can run:
bash delete_tar_gz.sh
For details on how to use the RefSpatial dataset with the RoboRefer series of models, please refer to the official implementation repository: Zhoues/RoboRefer".
After successfully running the decompression script, all archives will be replaced by the actual image (e.g., .jpg, .png) and depth map files. The final directory structure will be as follows:
RefSpatial/
├── 2D/
│ ├── image/
│ │ ├── 000002b97e5471a0.jpg
│ │ └── ... (all 2D image files)
│ ├── depth/
│ │ ├── 000002b97e5471a0.png
│ │ └── ... (all 2D depth map files)
│ ├── choice_qa.json
│ └── reasoning_template_qa.json
├── 3D/
│ ├── image/
│ │ ├── 42444499_2458914221666_wide_image.png
│ │ └── ... (all 3D single-view images)
│ ├── depth/
│ │ ├── 42444499_2458914221666_wide_depth.png
│ │ └── ... (all 3D single-view depth maps)
│ ├── image_multi_view/
│ │ ├── 42444499_2460713483458_wide_image.png
│ │ └── ... (all 3D multi-view images)
│ ├── depth_multi_view/
│ │ ├── 42444499_2460713483458_wide_depth.png
│ │ └── ... (all 3D multi-view depth maps)
│ ├── image_visual_choice/
│ │ ├── 42444499_2458914221666_image_with_bbox_0.png
│ │ └── ... (all 3D visual choice images)
│ ├── choice_qa.json
│ └── ... (other 3D json files)
├── Simulator/
│ │ └── 00020ec1a2dbc971.png
│ │ └── ... (all simulator images)
│ ├── depth/
│ │ └── 00020ec1a2dbc971.png
│ │ └── ... (all simulator depth maps)
│ └── metadata.json
└── ... (scripts and visualization folder)
To use this dataset for model training, you need to match the entries of the image and depth path in the JSON files with the decompressed image and depth map files. Below is the mapping of each JSON file to its corresponding image and depth folders.
{
"2D": {
"folder": "RefSpatial/2D",
"jsons": {
"choice_qa.json": {
"image_root": "RefSpatial/2D/image",
"depth_root": "RefSpatial/2D/depth"
},
"reasoning_template_qa.json": {
"image_root": "RefSpatial/2D/image",
"depth_root": "RefSpatial/2D/depth"
}
}
},
"3D": {
"folder": "RefSpatial/3D",
"jsons": {
"choice_qa.json": {
"depth_root": "RefSpatial/3D/depth",
"image_root": "RefSpatial/3D/image"
},
"multi_view_qa.json": {
"depth_root": "RefSpatial/3D/depth_multi_view",
"image_root": "RefSpatial/3D/image_multi_view"
},
"reasoning_template_qa.json": {
"depth_root": "RefSpatial/3D/depth",
"image_root": "RefSpatial/3D/image"
},
"vacant_qa.json": {
"depth_root": "RefSpatial/3D/depth",
"image_root": "RefSpatial/3D/image"
},
"visual_choice_qa.json": {
"depth_root": "RefSpatial/3D/depth",
"image_root": "RefSpatial/3D/image_visual_choice"
},
}
},
"Simulator": {
"folder": "RefSpatial/Simulator",
"jsons": {
"metadata.json": {
"image_root": "RefSpatial/Simulator/image",
"depth_root": "RefSpatial/Simulator/depth"
}
}
}
}
If you have any questions about the dataset, feel free to email Jingkun An (anjingkun02@gmail.com) Yi Han (hany01@buaa.edu.cn), and Enshen Zhou(zhouenshen@buaa.edu.cn).
Please consider citing our work if this dataset is useful for your research.
@article{zhou2025roborefer,
title={RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Robotics},
author={Zhou, Enshen and An, Jingkun and Chi, Cheng and Han, Yi and Rong, Shanyu and Zhang, Chi and Wang, Pengwei and Wang, Zhongyuan and Huang, Tiejun and Sheng, Lu and others},
journal={arXiv preprint arXiv:2506.04308},
year={2025}
}
⚠️ Warning: The Dataset Viewer and Data Studio above are for display only. They show just 800 samples from the full RefSpatial dataset, taken from the "SubsetVisualization" folder in Hugging Face ".parquet" format.
24
280 commits
2 linked in READMEs
updated Feb 4, 2026
RefSpatial: A Large-scale Dataset for teaching a general VLM to achieve spatial referring with reasoning
RefSpatial is a comprehensive dataset combining 2D images from OpenImages, 3D videos from CA-1M, and simulated data generated from Blender (Fig. 1 (a)). Its key features include:
Figure 1: RefSpatial dataset pipeline and statistics of 31 spatial relations.
In Fig. 1, we present the dataset recipe that progressively integrates 2D, 3D, and simulated data to enable general VLMs to adapt to spatial referring tasks in a bottom-up manner. For more details of our dataset pipeline, please refer to our paper RoboRefer.
After downloading all the data from Hugging Face, your RefSpatial folder will look like this. Large files (like 2D images) are split into parts (.part_a, etc.), while others are single .tar.gz files.
RefSpatial/
├── 2D/
│ ├── image/
│ │ ├── image.tar.gz.part_a
│ │ └── ... (more split files)
│ ├── depth/
│ │ ├── depth.tar.gz.part_a
│ │ └── ... (more split files)
│ ├── choice_qa.json
│ └── reasoning_template_qa.json
├── 3D/
│ ├── image/
│ │ └── image.tar.gz
│ ├── depth/
│ │ └── depth.tar.gz
│ ├── image_multi_view/
│ │ └── image_multi_view.tar.gz
│ ├── depth_multi_view/
│ │ └── depth_multi_view.tar.gz
│ ├── image_visual_choice/
│ │ ├── image_visual_choice.tar.gz.part_a
│ │ └── ... (more split files)
│ ├── choice_qa.json
│ ├── multi_view_qa.json
│ ├── reasoning_template_qa.json
│ ├── vacant_qa.json
│ └── visual_choice_qa.json
├── Simulator/
│ ├── image/
│ │ └── image.tar.gz
│ ├── depth/
│ │ └── depth.tar.gz
│ └── metadata.json
├── SubsetVisualization/
│ ├── 2D-00000-of-00001.parquet
│ └── ... (more .parquet files)
├── unzip_dataset.sh
└── delete_tar_gz.sh
2D/: Contains all data related to 2D spatial reasoning tasks.
image/, depth/: Contain the split archives for 2D images and their corresponding depth maps.*.json: The relevant question-answering and reasoning annotations.3D/: Contains data for more complex 3D scene understanding tasks.
image/, depth/: Standard single-view 3D scene images and depth maps.image_multi_view/, depth_multi_view/: Data for multi-view tasks.image_visual_choice/: Data for visual choice question tasks; its larger size requires it to be split into archives.*.json: Annotations for the various 3D tasks.Simulator/: Contains data generated from a simulation environment.
image/, depth/: Images and depth maps generated by the simulator, which are perfectly aligned and annotated.metadata.json: Metadata for the scenes.SubsetVisualization/: Contains subset samples for quick visualization and data inspection.
*.parquet: These files preview a small part of the dataset without needing to unzip everything.The provided unzip_dataset.sh script could decompress all of the *.tar.gz files. Please run it from the RefSpatial root directory.
cd RefSpatial
bash unzip_dataset.sh
This script will automatically perform the following actions:
.part_a, .part_b, etc., the script will use the cat command to combine them into a single, complete .tar.gz file. For example, image.tar.gz.part_a, ... will be merged into image.tar.gz.tar command to extract all .tar.gz archives into their current directories.If you wish to delete all .tar.gz and .part_* files after successful decompression to save disk space, you can run:
bash delete_tar_gz.sh
For details on how to use the RefSpatial dataset with the RoboRefer series of models, please refer to the official implementation repository: Zhoues/RoboRefer".
After successfully running the decompression script, all archives will be replaced by the actual image (e.g., .jpg, .png) and depth map files. The final directory structure will be as follows:
RefSpatial/
├── 2D/
│ ├── image/
│ │ ├── 000002b97e5471a0.jpg
│ │ └── ... (all 2D image files)
│ ├── depth/
│ │ ├── 000002b97e5471a0.png
│ │ └── ... (all 2D depth map files)
│ ├── choice_qa.json
│ └── reasoning_template_qa.json
├── 3D/
│ ├── image/
│ │ ├── 42444499_2458914221666_wide_image.png
│ │ └── ... (all 3D single-view images)
│ ├── depth/
│ │ ├── 42444499_2458914221666_wide_depth.png
│ │ └── ... (all 3D single-view depth maps)
│ ├── image_multi_view/
│ │ ├── 42444499_2460713483458_wide_image.png
│ │ └── ... (all 3D multi-view images)
│ ├── depth_multi_view/
│ │ ├── 42444499_2460713483458_wide_depth.png
│ │ └── ... (all 3D multi-view depth maps)
│ ├── image_visual_choice/
│ │ ├── 42444499_2458914221666_image_with_bbox_0.png
│ │ └── ... (all 3D visual choice images)
│ ├── choice_qa.json
│ └── ... (other 3D json files)
├── Simulator/
│ │ └── 00020ec1a2dbc971.png
│ │ └── ... (all simulator images)
│ ├── depth/
│ │ └── 00020ec1a2dbc971.png
│ │ └── ... (all simulator depth maps)
│ └── metadata.json
└── ... (scripts and visualization folder)
To use this dataset for model training, you need to match the entries of the image and depth path in the JSON files with the decompressed image and depth map files. Below is the mapping of each JSON file to its corresponding image and depth folders.
{
"2D": {
"folder": "RefSpatial/2D",
"jsons": {
"choice_qa.json": {
"image_root": "RefSpatial/2D/image",
"depth_root": "RefSpatial/2D/depth"
},
"reasoning_template_qa.json": {
"image_root": "RefSpatial/2D/image",
"depth_root": "RefSpatial/2D/depth"
}
}
},
"3D": {
"folder": "RefSpatial/3D",
"jsons": {
"choice_qa.json": {
"depth_root": "RefSpatial/3D/depth",
"image_root": "RefSpatial/3D/image"
},
"multi_view_qa.json": {
"depth_root": "RefSpatial/3D/depth_multi_view",
"image_root": "RefSpatial/3D/image_multi_view"
},
"reasoning_template_qa.json": {
"depth_root": "RefSpatial/3D/depth",
"image_root": "RefSpatial/3D/image"
},
"vacant_qa.json": {
"depth_root": "RefSpatial/3D/depth",
"image_root": "RefSpatial/3D/image"
},
"visual_choice_qa.json": {
"depth_root": "RefSpatial/3D/depth",
"image_root": "RefSpatial/3D/image_visual_choice"
},
}
},
"Simulator": {
"folder": "RefSpatial/Simulator",
"jsons": {
"metadata.json": {
"image_root": "RefSpatial/Simulator/image",
"depth_root": "RefSpatial/Simulator/depth"
}
}
}
}
If you have any questions about the dataset, feel free to email Jingkun An (anjingkun02@gmail.com) Yi Han (hany01@buaa.edu.cn), and Enshen Zhou(zhouenshen@buaa.edu.cn).
Please consider citing our work if this dataset is useful for your research.
@article{zhou2025roborefer,
title={RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Robotics},
author={Zhou, Enshen and An, Jingkun and Chi, Cheng and Han, Yi and Rong, Shanyu and Zhang, Chi and Wang, Pengwei and Wang, Zhongyuan and Huang, Tiejun and Sheng, Lu and others},
journal={arXiv preprint arXiv:2506.04308},
year={2025}
}