Molmo2-SGCoT: Spatiotemporal Grounded Chain-of-Thought Training Data

Paper | Project Page | GitHub

Training data for aligning Molmo2 to perform Spatiotemporal Grounded Chain-of-Thought (SGCoT) on VET-Bench — generating explicit object tracking trajectories before answering questions.

Overview

This dataset contains 300 synthetic samples where the model generates a structured trajectory <tracks> producing a final answer. The trajectories encode spatial coordinates (x, y in a 1000×1000 normalized space) at 0.5-second intervals over 12 seconds (25 frames).

Note that you can train with any continuous trajectories, but using in-distribution trajectories causes minimal degradation of the original tracking ability.

Data Format

Each sample is a chat-style messages entry in JSONL format:

{
  "messages": [
    {
      "role": "user",
      "content": [{"type": "text", "text": "Track the cup that contains the ball and answer which cup contains the ball at the end of the video."}]
    },
    {
      "role": "assistant",
      "content": [{"type": "text", "text": "<tracks coords=\"0.0 1 772 524;0.5 1 805 310;...;12.0 1 216 517\">the cup that contains the ball</tracks> Answer: left."}]
    }
  ]
}

Tracks Format

<tracks coords="t obj x y;t obj x y;...">object description</tracks>

• t: timestamp in seconds (0.0 to 12.0, step 0.5)
• obj: object index (always 1)
• x, y: spatial coordinates in [0, 1000] normalized space
• Final answers are one of: left, middle, right

How It Was Generated

The trajectories are synthetically generated from movement extracted from real tracking data:

Usage

from datasets import load_dataset

dataset = load_dataset("tiedong/Molmo2-SGCoT", split="train")