OV-SGG-Bench: evaluation packs for open-vocabulary relation prediction
0
6 commits
2 linked in READMEs
updated Oct 11, 2026
The six evaluation sources of OV-SGG-Bench, the protocol of
RelateAnything (code · paper:
RelateAnything: Real-Time Open-Vocabulary Relation Prediction From Any
Inputs), packed in the memmap format the evaluators read, plus the
negative-set and cell files the scorers need and the training-side artifacts
the shipped recipe names. No images are redistributed: every pack stores
boxes, labels and file names, and the evaluator reads pixels from your copy of
the source dataset (set RA_DATASETS).
The protocol has six axes, chosen so that no single one can be won by matching
a benchmark's annotation prior. The definition, the sources, and the reasons
are in
benchmark/SPEC.md.
| pack | images | role | axis |
|---|---|---|---|
vg150 (test) | 26,404 | in-domain control (100 % vocabulary overlap with VG150-trained baselines) | A1, A3 |
psg (test) | 2,179 | primary transfer, COCO images, 56 predicates | A1, A3, A4, A5 |
indoorvg (test) | 4,403 | domain shift | A1, A3 |
hicodet (test) | 9,490 | 116 external verbs; federated negatives (its train split, 33,415 images, is a training source) | A1, A2 |
haystack | 11,368 | explicit negatives on SA-1B images (no image overlap with any training set) | A2 |
spatialsense, spatialsense_test, spatialsense_valid | 1,920 test images, 2,758 balanced true/false triples | adversarial spatial understanding | A6 |
| path | contents |
|---|---|
packs/<name>/<split>/ | meta.json, file_names.json, img_meta.npy, boxes.npy, box_cats.npy, rels.npy |
datamix/haystack_negatives.json, datamix/hicodet_negatives*.json | explicit negative cells for the A2 precision axis |
datamix/spatialsense_{test,valid}_cells.json | the balanced true/false cells for A6 |
datamix/indoorvg_holdout.json | image ids excluded from training (leakage audit) |
datamix/registry.json, vg2coco.json, psg2coco.json | id registry used by the converters |
detect/yoloworld_ov_psg_test.npz | the shared YOLO-World detections on PSG test that A4 and A5 score every system on |
text_student_v2_512/student.pt | the distilled predicate text encoder every released checkpoint was trained with |
datamix_v22/pair_opportunity.npz, datamix_v22/text_space/* | the soft-supervision and synonym artifacts the training recipe names (train.sh) |
Two commands from a clone of RelateAnything:
pip install -e ".[train,hub]"
python benchmark/setup_data.py # this dataset, the test images (~15 GB), every pack relocated and checked
python benchmark/run_benchmark.py --checkpoint maelic/relsgg-vits16plus
setup_data.py --axes a1 a4 a6 downloads only the images those axes read. The images come
from their sources (VG150, PSG and IndoorVG from maelic/*-coco-format, HICO-DET from
zhimeng/hico_det, Haystack from its authors, SpatialSense from Zenodo); none are
redistributed here. Each pack's meta.json records img_dir as datasets/<source>/<split>,
which setup_data.py re-roots at $RA_DATASETS.
run_benchmark.py takes a Hugging Face model id or any checkpoint of the recipe and prints
OVS-F1 over A1, A2, A4 and A6. A5, the graph-quality axis, needs a VLM judge (vLLM,
Qwen/Qwen3.6-27B) and runs only with --judge; it adds the fifth axis of the paper's
composite. The protocol and the composite are defined in
benchmark/SPEC.md
and docs/evaluation.md.
Each pack retains the license of its source: Visual Genome / VG150 and
IndoorVG (CC BY 4.0), PSG (COCO images, CC BY 4.0), HICO-DET (research use),
Haystack (SA-1B research license), SpatialSense (CC BY 4.0). The text student
and the datamix_v22 artifacts are derivatives of Meta DINOv3 (dino.txt) and
follow the DINOv3 license.
@article{neau2026relateanything,
title = {RelateAnything: Real-Time Open-Vocabulary Relation Prediction From Any Inputs},
author = {Neau, Ma\"elic},
journal = {arXiv preprint arXiv:2609.12552},
eprint = {2609.12552},
archivePrefix = {arXiv},
primaryClass = {cs.CV},
url = {https://arxiv.org/abs/2609.12552},
year = {2026}
}
OV-SGG-Bench: evaluation packs for open-vocabulary relation prediction
0
6 commits
2 linked in READMEs
updated Oct 11, 2026
The six evaluation sources of OV-SGG-Bench, the protocol of
RelateAnything (code · paper:
RelateAnything: Real-Time Open-Vocabulary Relation Prediction From Any
Inputs), packed in the memmap format the evaluators read, plus the
negative-set and cell files the scorers need and the training-side artifacts
the shipped recipe names. No images are redistributed: every pack stores
boxes, labels and file names, and the evaluator reads pixels from your copy of
the source dataset (set RA_DATASETS).
The protocol has six axes, chosen so that no single one can be won by matching
a benchmark's annotation prior. The definition, the sources, and the reasons
are in
benchmark/SPEC.md.
| pack | images | role | axis |
|---|---|---|---|
vg150 (test) | 26,404 | in-domain control (100 % vocabulary overlap with VG150-trained baselines) | A1, A3 |
psg (test) | 2,179 | primary transfer, COCO images, 56 predicates | A1, A3, A4, A5 |
indoorvg (test) | 4,403 | domain shift | A1, A3 |
hicodet (test) | 9,490 | 116 external verbs; federated negatives (its train split, 33,415 images, is a training source) | A1, A2 |
haystack | 11,368 | explicit negatives on SA-1B images (no image overlap with any training set) | A2 |
spatialsense, spatialsense_test, spatialsense_valid | 1,920 test images, 2,758 balanced true/false triples | adversarial spatial understanding | A6 |
| path | contents |
|---|---|
packs/<name>/<split>/ | meta.json, file_names.json, img_meta.npy, boxes.npy, box_cats.npy, rels.npy |
datamix/haystack_negatives.json, datamix/hicodet_negatives*.json | explicit negative cells for the A2 precision axis |
datamix/spatialsense_{test,valid}_cells.json | the balanced true/false cells for A6 |
datamix/indoorvg_holdout.json | image ids excluded from training (leakage audit) |
datamix/registry.json, vg2coco.json, psg2coco.json | id registry used by the converters |
detect/yoloworld_ov_psg_test.npz | the shared YOLO-World detections on PSG test that A4 and A5 score every system on |
text_student_v2_512/student.pt | the distilled predicate text encoder every released checkpoint was trained with |
datamix_v22/pair_opportunity.npz, datamix_v22/text_space/* | the soft-supervision and synonym artifacts the training recipe names (train.sh) |
Two commands from a clone of RelateAnything:
pip install -e ".[train,hub]"
python benchmark/setup_data.py # this dataset, the test images (~15 GB), every pack relocated and checked
python benchmark/run_benchmark.py --checkpoint maelic/relsgg-vits16plus
setup_data.py --axes a1 a4 a6 downloads only the images those axes read. The images come
from their sources (VG150, PSG and IndoorVG from maelic/*-coco-format, HICO-DET from
zhimeng/hico_det, Haystack from its authors, SpatialSense from Zenodo); none are
redistributed here. Each pack's meta.json records img_dir as datasets/<source>/<split>,
which setup_data.py re-roots at $RA_DATASETS.
run_benchmark.py takes a Hugging Face model id or any checkpoint of the recipe and prints
OVS-F1 over A1, A2, A4 and A6. A5, the graph-quality axis, needs a VLM judge (vLLM,
Qwen/Qwen3.6-27B) and runs only with --judge; it adds the fifth axis of the paper's
composite. The protocol and the composite are defined in
benchmark/SPEC.md
and docs/evaluation.md.
Each pack retains the license of its source: Visual Genome / VG150 and
IndoorVG (CC BY 4.0), PSG (COCO images, CC BY 4.0), HICO-DET (research use),
Haystack (SA-1B research license), SpatialSense (CC BY 4.0). The text student
and the datamix_v22 artifacts are derivatives of Meta DINOv3 (dino.txt) and
follow the DINOv3 license.
@article{neau2026relateanything,
title = {RelateAnything: Real-Time Open-Vocabulary Relation Prediction From Any Inputs},
author = {Neau, Ma\"elic},
journal = {arXiv preprint arXiv:2609.12552},
eprint = {2609.12552},
archivePrefix = {arXiv},
primaryClass = {cs.CV},
url = {https://arxiv.org/abs/2609.12552},
year = {2026}
}