GLUE3D: General Language Understanding Evaluation for 3D Point Clouds

Giorgio Mariani, Alessandro Raganato, Simone Melzi, Gabriella Pasi

Official implementation of GLUE3D: General Language Understanding Evaluation for 3D Point Clouds.

GLUE3D is a Q&A benchmark for evaluation of 3D-LLMs object understanding capabilities. It is built around 128 richly textured surfaces spanning creatures, objects, architecture and transport. Each surface is provided as a 50 k-point RGB point cloud, a 8K-point RGB point cloud, a 512 × 512 RGB rendering, and five RGB-D multiviews. These multiple representations enable point-for-point evaluation across several modalities.

GLUE3D consists of four Q&A task types: binary question answering, multiple-choice question answering, open-ended question answering, and captioning. This diverse set of tasks enables a more robust and comprehensive assessment of multimodal understanding in 3D-LLMs.

---

Table of Contents

• 1. Installation
  • Installation from source
• 2. Answer generation
  • 2.1 Using the GLUE3D data loader
  • 2.2 Using the *HFAnswerGenerator classes
• 3. Answer evaluation

1. Installation

To evaluate your question-answering model on GLUE3D, we offer a PyPI package that can be easily installed with the command:

pip install glue3d

This is the raccomended way to install the package if you are interested in running your multimodal LLM on GLUE3D.

Installation from source

You can install glue3d from source if you want the latest changes in the library or are interested in contributing. However, the latest version may not be stable. Feel free to open an issue if you encounter an error.

git clone https://github.com/giorgio-mariani/GLUE3D.git
cd GLUE3D

pip install -e .

An advantage of installing directly from source is the ability to utilize the docker-images provided with the library. For more information, take a look at the README page in the docker_images folder.

2. Answer generation

Before evaluating your model capabilities, you first need to generate your 3D-LLM's answers for the desired GLUE3D task. You can do so in two main ways:

1. Using the dataset loader (load_GLUE3D_benchmark) with your own model and code; Read section 2.1 for more information.
2. Using the built-in HFAnswerGenerator abstract classes. This option is to be preferred if your model implements the generate() huggingface autoregressive generation method. For example, models like LlavaLlamaForCausalLM, or PointLLMLlamaForCausalLM are easier to evaluate using this approach. Read section 2.2 for more information.

2.1 Using the GLUE3D data loader

The GLUE3D benchmark data can be (down)loaded using:

import pandas as pd
from glue3d.data import load_GLUE3D_benchmark

dataset = load_GLUE3D_benchmark(
    dataset_name="GLUE3D-points-8k", # or "GLUE3D-images", "GLUE3D-multiview", "GLUE3D-points"
    qa_task="binary_task",           # or "multiplechoice_task", "captioning_task"
    cache_dir=None,                  # Optional; defaults to './cache' or $GLUE3D_CACHE_DIR
)

This procedure loads to memory and prepare the necessary GLUE3D data for the specified Q&A task and data-type. It also automatically downloads all necessary data to disk if this is not yet stored. The available tasks are binary_task, multiplechoice_task, captioning_task. Note that the loader uses a local cache directory. You can customize it via the GLUE3D_CACHE_DIR environment variable.

Once the GLUE3D data is loaded, you can iterate through the dataset to generate answers for each question in the Q&A task:

your_model = ...  # Load your 3D-LLM

model_answers = []
for x in dataset:
    oid = x["object_id"]
    qid = x["question_id"]
    q = x["question"]
    pc = x["data"]  # e.g., (8192 x 6) np.ndarray for "GLUE3D-points-8K"

    answer = your_model.answer_question(pc, q)
    model_answers.append({
        "OBJECT_ID": oid,
        "QUESTION_ID": qid,
        "MODEL_ANSWER": answer,
    })

# Save results
pd.DataFrame.from_records(model_answers).to_csv("qa.csv", index=False)

[!IMPORTANT] Ensure your answers follow the expected format for each task.

• For the binary_task, the model answer must be a boolean object (either True or False).
• For the multiplechoice_task, the model answer must be one of A, B, C, D.
• For the captioning_task the model answer must be a string.

2.2 Using the *HFAnswerGenerator classes

If your 3D-LLM inherits the GeneratorMixin class (e.g., LlavaLlamaForCausalLM), then it is possible to use our *HFAnswerGenerator abstract classes to simplify the generation process. The only requirement is to implement the prepare_inputs function, which takes in input the point cloud (or image) and the question and returns the keyword inputs for the GeneratorMixin.generate() method:

import numpy as np
from typing import override
from glue3d import generate_GLUE3D_answers
from glue3d.models.hf import (
    BinaryHFAnswerGenerator,
    MultichoiceHFAnswerGenerator,
    CaptioningHFGenerator
)

# Example custom AnswerGenerator for the binary task
class YourAnswerGenerator(BinaryHFAnswerGenerator): # <- Swap with MultichoiceHFAnswerGenerator
    def __init__(self, your_model, tokenizer):      #   or CaptioningHFGenerator for other tasks.
        super().__init__(your_model, tokenizer)

    @override
    def prepare_inputs(self, data: np.ndarray, text: str) -> dict:
        ... # Preprocess data (e.g., tokenize text, move tensors to device, apply chat templates)
        return {
            "input_ids": ...,
            "points": ...,
            "do_sample": ...,
        }

Once you have your custom implementation, generation can be simply done by calling generate_GLUE3D_answers on your target dataset-type and Q&A task:

your_model = ...
answer_gen = YourAnswerGenerator(your_model)

qa_answers = generate_GLUE3D_answers(
    qa_task="binary_task",
    dataset_type="GLUE3D-points-8K",
    answer_generator=answer_gen,
)

# `qa_answers` is returned as a pandas DataFrame
qa_answers.to_csv("qa.csv", index=False)

---

3. Answer evaluation

As result of the answer generation step, you should have a .CSV file containing the question-answer pairs for a given task. The file (let us call it binary-qa.csv) should have a structure similar to

It is then possible to evaluate the answers produced by your model using the glue3d evaluate CLI command:

glue3d evaluate --input-file binary-qa.csv --output-file out.csv --task binary_task

Or equivalently, using Python

from glu3d.evaluate_answers import evaluate_GLUE3D_answers

out = evaluate_GLUE3D_answers("binary_task", "binary-qa.csv")
out.to_csv("out.csv")

For the binary and multiple choice tasks, the output is a dataframe which indicates extact match between the question answer and the model provided one. For the captioning task, results scores for BLEU, METEOR, ROUGE-L, S-BERT, and SimCSE are provided. All scores are scaled to range between 0-100.

[!NOTE] For the captioning task, it is also possible to change the evaluator to use qwen3-30B-A3B as a judge. To do so, use the command:

glue3d evaluate captions.csv output.csv --task captioning_task --evaluator qwen_3_30B_A3B

---