SG-Ego Annotations

Project Page: https://francescapistilli.github.io/GLEN.

Data annotation pipeline: https://github.com/francescapistilli/sg-ego

SG-Ego is a large scale annotation set extending Ego4D with spatio-temporal scene graphs, where relations triplets are consolidated over time into explicit time-evolving descriptions of the scene state. SG-Ego is released as part of our paper "Learning to Evolve Scenes: Reasoning about Human Activities with Scene Graphs", which presents a novel graph edit formulation to model dynamic scenes of human activities.

SG-Ego covers a subset of Ego4D that corresponds to the EgoClip and EgoMCQ, consisting of approx. 18.7M frame-level scene graphs and 3.8M clip-level scene graphs.

Usage

from datasets import load_dataset

# Load frame-level scene graphs
frame_graphs = load_dataset("francescapistilli/sg-ego", "frame_graphs")['train']
print(frame_graphs[0])

# Load clip-level consolidated scene graphs
video_graphs = load_dataset("francescapistilli/sg-ego", "video_graphs")['train']
print(video_graphs[0])

Data schema

Frame-level graphs

{
   "video_id": "770b0de4-9f4a-4c51-8b37-3c22fca3e6a6",
   "frame_id": 3678,
   "obj": [
      "person",
      "sofa",
      "main actor"
   ],
   "rel": [
      "in front of",
      "sit on"
   ],
   "confidence": [
      0.48,
      0.22,
      1.00
   ],
   "bbox": [
      [0.31, 0.08, 0.62, 0.84],
      [0.02, 0.60, 0.34, 0.92],
      [0.00, 0.00, 1.00, 1.00]
   ],
   "pair": [
      [0, 1],
      [2, 1]
   ],
   // ...
}

• video_id: unique identifier of the Ego4d video.
• frame_id: frame index at 5 FPS.
• obj: labels of the graph nodes.
• bbox: bounding boxes of the graph nodes, in xyxy format.
• confidence: confidence associated to the bounding boxes of the graph nodes.
• pair: graph edges encoded in the (source, target) format.
• rel: labels of the graph edges.

Video-level graphs

{
   "video_id":"770b0de4-9f4a-4c51-8b37-3c22fca3e6a6",
   "start_frame":3678,
   "end_frame":3682,
   "obj":[
      "person",
      "sofa",
      "main actor",
      // ...
   ],
   "confidence":[
      0.81,
      0.63,
      1.0,
      // ...
   ],
   "bbox":[
      [0.31, 0.08, 0.62, 0.84],
      [0.00, 0.15, 0.17, 0.90],
      [0.00, 0.00, 1.00, 1.00],
      // ...
   ],
   "frame_idx":[
      0,
      4,
      0,
      //...
   ],
   "pair":[
      [2, 4],
      [0, 1],
      [2, 3],
      //...
   ],
   "rel":[
      "hold",
      "in front of",
      "sit on",
      //...
   ],
   "history":[
      [
         {"frame_idx": 0, "label": "person", "obj_idx": 0},
         {"frame_idx": 0, "label": "person", "obj_idx": 0},
         {"frame_idx": 4, "label": "person", "obj_idx": 0}
      ],
      // ...
   ]
}

• video_id: unique identifier of the Ego4d video.
• start_frame and end_frame: frame boundaries of the consolidation window.
• obj: labels of the graph nodes.
• bbox: bounding boxes of the graph nodes, in xyxy format at the frame stored in frame_idx.
• confidence: confidence associated to the bounding boxes of the graph nodes at the frame stored in frame_idx..
• pair: graph edges encoded in the (source, target) format.
• rel: labels of the graph edges.
• history: this tracks the consolidation history of the graph objects, mapping each consolidated node back to its original source nodes in the frame-level graphs.