data/ folder contains HF-compatible splits:

• location
• placement
• unseen

Each sample includes:

For full reproducibility and visualization, we also include the original files under:

• Location/
• Placement/
• Unseen/

Each folder contains:

Location/
├── image/        # RGB images (e.g., 0.png, 1.png, ...)
├── mask/         # Ground truth binary masks
└── question.json # List of referring prompts and metadata

Each entry in question.json has the following format:

{
  "id": 40,
  "object": "the second object from the left to the right on the nearest platform",
  "prompt": "Please point out the second object from the left to the right on the nearest platform.",
  "suffix": "Your answer should be formatted as a list of tuples, i.e. [(x1, y1)], ...",
  "rgb_path": "image/40.png",
  "mask_path": "mask/40.png",
  "category": "location",
  "step": 2
}

You can load the dataset easily using the datasets library:

from datasets import load_dataset

# Load the entire dataset (all splits: location, placement, unseen)
# This returns a DatasetDict
dataset_dict = load_dataset("BAAI/RefSpatial-Bench")

# Access a specific split, for example 'location'
location_split_hf = dataset_dict["location"]

# Or load only a specific split directly (returns a Dataset object)
# location_split_direct = load_dataset("BAAI/RefSpatial-Bench", name="location")

# Access a sample from the location split
sample = location_split_hf[0] 

# sample is a dictionary where 'rgb' and 'mask' are PIL Image objects
# To display (if in a suitable environment like a Jupyter notebook):
# sample["image"].show()
# sample["mask"].show()

print(f"Prompt (from HF Dataset): {sample['prompt']}")
print(f"Suffix (from HF Dataset): {sample['suffix']}")
print(f"Reasoning Steps (from HF Dataset): {sample['step']}")

If you are working with the raw data format (e.g., after cloning the repository or downloading the raw files), you can load the questions from the question.json file for each split and then load the images and masks using a library like Pillow (PIL).

This example assumes you have the location, placement, and unseen folders (each containing image/, mask/, and question.json) in a known base_data_path.

import json
import os
from PIL import Image

# Set the dataset split name and base directory path
split_name = "Location"
base_data_path = "."  # Or set to your actual dataset path

# Load question.json file
question_file = os.path.join(base_data_path, split_name, "question.json")
try:
    with open(question_file, 'r', encoding='utf-8') as f:
        samples = json.load(f)
except FileNotFoundError:
    print(f"File not found: {question_file}")
    samples = []

# Process the first sample if available
if samples:
    sample = samples[0]
    print(f"\n--- Sample Info ---")
    print(f"ID: {sample['id']}")
    print(f"Prompt: {sample['prompt']}")

    # Construct absolute paths to RGB image and mask
    rgb_path = os.path.join(base_data_path, split_name, sample["rgb_path"])
    mask_path = os.path.join(base_data_path, split_name, sample["mask_path"])

    # Load images using Pillow
    try:
        rgb_image = Image.open(rgb_path)
        mask_image = Image.open(mask_path)
        sample["image"] = rgb_image
        sample["mask"] = mask_image
        print(f"RGB image size: {rgb_image.size}")
        print(f"Mask image size: {mask_image.size}, mode: {mask_image.mode}")
    except FileNotFoundError:
        print(f"Image file not found:\n{rgb_path}\n{mask_path}")
    except Exception as e:
        print(f"Error loading images: {e}")
else:
    print("No samples loaded.")

To evaluate RoboRefer on RefSpatial-Bench:

1. Prepare Input Prompt:

   Concatenate sample["prompt"] and sample["suffix"] to form the complete instruction.

   # Example for constructing the full input for a sample
   full_input_instruction = sample["prompt"] + " " + sample["suffix"]
   

2. Model Prediction & JSON Parsing & Coordinate Scaling:

   • Model Prediction: After providingthe image (sample["image"]) and full_input_instruction to the RoboRefer, it outputs normalized coordinate in a JSON format like[(x, y),...], where each x and y` value is normalized to a range of 0-1.

   • JSON Parsing: Parse this JSON string to extract the coordinate attributes (e.g., x, y).

   • Coordinate Scaling:

     1. Use sample["image"].size to get (width, height) and scale to the original image dimensions (height for y, width for x).

     # Example: model_output_robo is [(0.234, 0.567)] from Roborefer/RoboPoint
     # sample["image"] is a PIL Image object loaded by the datasets library or loaded from the raw data
     
     def text2pts(text, width, height):
         pattern = r"\(([-+]?\d+\.?\d*(?:,\s*[-+]?\d+\.?\d*)*?)\)"
         matches = re.findall(pattern, text)
         points = []
         for match in matches:
             vector = [
                 float(num) if '.' in num else int(num) for num in match.split(',')
             ]
             if len(vector) == 2:    
                 x, y = vector
                 if isinstance(x, float) or isinstance(y, float):
                     x = int(x * width)
                     y = int(y * height)
                 points.append((x, y))
     
     width, height = sample["image"].size
     scaled_roborefer_points = text2pts(model_output_robo, width, height)
     
     # These scaled_roborefer_points are then used for evaluation against the mask.
     

3. Evaluation: Compare scaled_roborefer_points against sample["mask"]. The main metric is average success rate — the percentage of predictions falling within the mask.

To evaluate Gemini Series on RefSpatial-Bench:

1. Prepare Input Prompt:

   Concatenate the string "Locate the points of" and sample["object"] to form the complete instruction.

   # Example for constructing the full input for a sample
   full_input_instruction = "Locate the points of " + sample["object"] + "."
   

2. Model Prediction & JSON Parsing & Coordinate Scaling:

   • Model Prediction: After providing the image (sample["image"]) and full_input_instruction to the Gemini model series, it outputs normalized coordinates in an JSON format like "```json\n[\n {\"point\": [y, x], \"label\": \"free space\"}, ...\n]\n```", where each y and x value is normalized to a range of 0-1000.

   • JSON Parsing: Parse this JSON string to extract the coordinate attributes (e.g., x1, y1, x2, y2, etc.).

   • Coordinate Conversion: To use these coordinates for evaluation against the mask, they must be:

     1. Divided by 1000.0 to normalize them to the 0.0-1.0 range.
     2. Scaled to the original image dimensions (height for y, width for x).

     # Example: model_output_gemini is "```json\n[\n  {\"point\": [438, 330], \"label\": \"free space\"}\n]\n```" from Gemini
     # and sample["image"] is a PIL Image object loaded by the datasets library or loaded from the raw data
     
     def json2pts(text, width, height):
        match = re.search(r"```(?:\w+)?\n(.*?)```", text, re.DOTALL)
        if not match:
            print("No valid code block found.")
            return np.empty((0, 2), dtype=int)
     
        json_cleaned = match.group(1).strip()
     
        try:
            data = json.loads(json_cleaned)
        except json.JSONDecodeError as e:
            print(f"JSON decode error: {e}")
            return np.empty((0, 2), dtype=int)
     
        points = []
        for item in data:
            if "point" in item and isinstance(item["point"], list) and len(item["point"]) == 2:
                y_norm, x_norm = item["point"]
                x = int(x_norm / 1000 * width)
                y = int(y_norm / 1000 * height)
                points.append((x, y))
     
        return np.array(points)
     
     width, height = sample["image"].size 
     scaled_gemini_points = json2pts(model_output_gemini, width, height)
     # These scaled_gemini_points are then used for evaluation against the mask.
     

3. Evaluation: Compare scaled_gemini_points against sample["mask"]. The main metric is average success rate — the percentage of predictions falling within the mask.

To evaluate a Molmo model on this benchmark:

1. Prepare Input Prompt:

   Concatenate "Locate several points of" and sample["object"] to form the complete instruction.

   # Example for constructing the full input for a sample
   full_input_instruction = "Locate several points of " + sample["object"] + "."
   

2. Model Prediction, XML Parsing, & Coordinate Scaling:

   • Model Prediction: After providing the image (sample["image"]) and full_input_instruction to the Molmo, it outputs normalized coordinates in an XML format like <points x1="61.5" y1="40.4" x2="76.8" y2="21.8" ... />, where each x and y value is normalized to a range of 0-100.

   • XML Parsing: Parse this XML string to extract the coordinate attributes (e.g., x1, y1, x2, y2, etc.).

   • Coordinate Conversion:

     1. Divide each coordinate by 100.0 to normalize it to the 0.0-1.0 range.
     2. Scaled to the original image dimensions (height for y, width for x).

     # Example: model_output_molmo is '<points x1="61.5" y1="40.4" x2="76.8" y2="21.8"/>' from Molmo
     # and sample["image"] is a PIL Image object loaded by the datasets library or loaded from the raw data
     
     def xml2pts(xml_text, width, height):
     	import re
         pattern = re.compile(r'(x\d+)="(-?\d+\.?\d*)"\s+(y\d+)="(-?\d+\.?\d*)"')
         matches = pattern.findall(xml_text)
         points = [(int(float(x_val) / 100.0 * width), int(float(y_val) / 100.0 * height) ) for _, x_val, _, y_val in matches]
         return np.array(points)
     
     width, height = sample["image"].size 
     scaled_molmo_points = xml2pts(model_output_molmo, width, height)
     # These scaled_molmo_points are then used for evaluation.
     

3. Evaluation: Compare scaled_molmo_points against sample["mask"]. The main metric is average success rate — the percentage of predictions falling within the mask.