GroOT-MOT17 tracklets

A text-to-tracklet retrieval benchmark for MOT17 pedestrians. Each captioned MOT17 track is cropped into its own short mp4. A text query must retrieve the matching track from a gallery of 454 tracks. The layout is the same as bassatbassat/TVPReid, so the same evaluation code runs on both.

This is an unofficial derivative. The videos come from MOT17 and the captions come from GroOT. The credit for both belongs to their authors (see below). What this repository adds is listed under Our contribution.

Configs

All configs share the same 454-track gallery. Only the queries differ.

Tracks without a caption of the requested type stay in the gallery as distractors.

Read the action numbers with care. GroOT action captions are mostly generic. The most common one, "a person walking on the street", is shared by 78 tracks. Only 112 of 454 action captions are unique. An action query often matches several tracks equally well, yet exactly one counts as correct. The combined config was made to reduce this ambiguity.

Original work (credit)

• Captions: GroOT, from Type-to-Track: Retrieve Any Object via Prompt-based Tracking by Pha Nguyen, Kha Gia Quach, Kris Kitani, and Khoa Luu (NeurIPS 2023). GroOT gives every MOT17 box captions = [appearance, action]. We use mot17_train_coco.json from uark-cviu/Type-to-Track. The authors mark the MOT17 test captions as sub-optimal, so we do not use them.
• Videos and boxes: MOT17 / MOT16 by Anton Milan, Laura Leal-Taixé, Ian Reid, Stefan Roth, and Konrad Schindler (motchallenge.net).

The appearance and action captions in this repository are the GroOT captions unchanged, apart from whitespace trimming and dropping empty or None values.

Our contribution

1. Tracklet videos. Each GroOT track is cropped from the MOT17 FRCNN frames into one mp4, so the track itself can be retrieved instead of individual boxes.

2. Retrieval splits. The appearance, action, all, and combined query sets share one gallery, in the TVPReid format.

3. Refined combined captions (combined). For the 344 tracks that have both captions, the two GroOT captions were merged into one fluent sentence about a single subject. It works in two passes:

   • A rule-based merge (combine_captions in the build code) joins the two captions and keeps the more specific subject ("woman" over "person").
   • An LLM assistant (Cursor agent, Auto model routing) then reviewed all 344 merged captions one by one. It fixed grammar and articles, removed repeated phrases such as "carrying a bag, walking down the street carrying two bags", and resolved two gender conflicts between a track's appearance and action captions by looking at the frames (MOT17-05 tracks 54 and 59). No new visual attributes were added. Every refined caption uses only facts already present in the two GroOT captions.

   The refined captions are versioned in groot_mot17_combined.json.

4. Build code, so the dataset can be regenerated from the original sources: scripts/build_groot_mot17.py.

Construction

• Only the FRCNN copy of each sequence is used, because DPM and SDP share the same frames.
• Boxes with ignore=1, height < 50px, or visibility < 0.25 are dropped.
• Tracks need at least 8 boxes and at least one caption. Long tracks are subsampled uniformly to at most 300 boxes.
• Each box is grown to the track's median aspect ratio, so the extra width is real scene context rather than black padding. It is then resized to height 256 and written at the sequence frame rate.

Layout

videos/<MOT17-XX>_t<track>.mp4
metadata/all-test.jsonl
metadata/appearance-test.jsonl
metadata/action-test.jsonl
metadata/combined-test.jsonl

Each row has these fields: video_id, video, captions, caption_types, subset, split, sequence, track_id, num_frames.

Usage

# pip install "shawaf-vlm @ git+https://github.com/BASSAT-BASSAT/Benchmarking-Video-Image-language-models-for-Tracklet-retrieval-.git"
from shawaf_vlm.data import load_groot

splits = load_groot("combined")  # downloads the metadata and mp4s on first use
print(len(splits.query), len(splits.gallery))

Standard protocol: each caption is a query, and its own track is the only positive. Report Rank-1/5/10, mAP, and median rank over the whole gallery.

Intended use and limitations

• Zero-shot evaluation only. There is no train split, and GroOT's MOT17 test captions are not included.
• Captions describe a single pedestrian in a crowded street, mall, or hallway. Many people wear similar clothes, so some queries are still ambiguous even in combined (for example "a person in a black jacket walking on the sidewalk").
• The combined captions are machine-refined. Wording can differ from what a human annotator would write, but the content is limited to the source captions.

License

GroOT annotations and MOT17 videos are released under CC BY-NC-SA 3.0. This dataset, including the refined captions, keeps that license: non-commercial use, with attribution, share-alike.

Citation

If you use this dataset, cite GroOT and MOT17. If you use the tracklet splits or the combined captions, please also cite this repository.

@article{nguyen2023type,
  title   = {Type-to-Track: Retrieve Any Object via Prompt-based Tracking},
  author  = {Nguyen, Pha and Quach, Kha Gia and Kitani, Kris and Luu, Khoa},
  journal = {Advances in Neural Information Processing Systems},
  year    = {2023}
}
@article{milan2016mot16,
  title   = {MOT16: A Benchmark for Multi-Object Tracking},
  author  = {Milan, Anton and Leal-Taix{\'e}, Laura and Reid, Ian and Roth, Stefan and Schindler, Konrad},
  journal = {arXiv:1603.00831},
  year    = {2016}
}
@misc{shawaf_groot_mot17,
  title        = {GroOT-MOT17 tracklets: text-to-tracklet retrieval splits with refined combined captions},
  howpublished = {\url{https://huggingface.co/datasets/bassatbassat/GroOT-MOT17}},
  note         = {Derived from GroOT (Nguyen et al., 2023) and MOT17. Code: https://github.com/BASSAT-BASSAT/Benchmarking-Video-Image-language-models-for-Tracklet-retrieval-},
  year         = {2026}
}