SHAWAF: text-to-tracklet retrieval

SHAWAF is a Stage 1 benchmark for text-to-tracklet retrieval. A frozen vision-language model embeds a text description and each video tracklet; cosine similarity ranks the gallery.

text description ──► text encoder ──► text embedding
                                      │ cosine similarity
tracklet ──────────► video/image encoder ──► tracklet embedding ──► ranked gallery

This repository does not fuse VLM and Re-ID features or train a projection into the VLM space (later SHAWAF stages). Optional Qdrant + Gradio demo support interactive text→tracklet search with multiple pooling modes; the core benchmark remains embedding-only. The identity encoder remains in the sibling Person-ReID-BenchMark project.

Executive summary

All results below are frozen, zero-shot evaluations: no model was fine-tuned on the target benchmark. Values are Rank-1 / mAP unless stated otherwise.

Practical model recommendations

• Use IRRA for person-search descriptions. It is the clear TVPReid and RSTPReid winner and is also among the most efficient accurate choices.
• Use Perception Encoder for a balanced general-purpose baseline. It is the best ActivityNet sentence model, wins ActivityNet on mean Rank-1, and is the strongest general model on GroOT combined captions.
• Use InternVideo2 CLIP-S for long videos and paragraph-level events. It is the ActivityNet paragraph winner; its video pretraining is useful when the query describes several events over time.
• Use SigLIP 2 when image/appearance evidence dominates. It is the strongest RSTPReid general model and the best GroOT appearance model.
• Use Qwen3-VL-Embedding-2B as the most promising extended model. It is substantially better than the other extended models on TVPReid, especially PRID, but it is slower and heavier than IRRA.
• Avoid choosing a model from speed alone. X-CLIP is fast but consistently weak on retrieval quality; GME-Qwen2-VL-2B is expensive and does not close the gap to the task-specific baselines.

Benchmark results

TVPReid: person descriptions and video tracklets

Each cell is the best Rank-1 / mAP for that model on the indicated test subset. The gallery sizes are PRID 185, iLIDS 75, and Duke 603 tracklets.

Why the gap? IRRA was trained for text-to-person retrieval, so its embedding space is aligned with clothing, body, and identity descriptions. General video-language models learn events and actions rather than fine-grained person identity. Qwen3 is the strongest of the newer general encoders, but specialization still matters more than parameter count.

RSTPReid: person descriptions and still images

This is an image-only check: 2,000 descriptions query 1,000 images from 200 identities.

The ordering supports the TVPReid conclusion: image-language pretraining helps more than video pretraining when there is no temporal signal, but task-specific person retrieval remains the strongest prior.

GroOT-MOT17: appearance versus action

The same 454 street tracks are queried using appearance descriptions, action descriptions, and a cleaned sentence containing both.

Action Rank-1 is only 0.44–3.52 because the action captions are highly repetitive: only 112 of 454 are unique, and “a person walking on the street” alone matches 78 tracks. This is primarily a dataset/caption ceiling, not a model failure. Appearance is discriminative; combining appearance and action improves both leading models. The reviewed combined captions are stored in shawaf_vlm/data/groot_mot17_combined.json.

ActivityNet Captions: general long-video events

This out-of-domain check uses a seeded sample of 1,000 val1 videos (ANET_SEED=0, about 180 seconds per video). paragraph joins a video's event sentences; sentence queries individual events. Only the seven base models were evaluated.

The ranking reverses relative to person retrieval: video-pretrained encoders transfer better to general events, while IRRA's person-specific training does not. Dense temporal sampling helps sentence queries because one window can match the queried event; mean pooling is safer for paragraphs.

These are not published ActivityNet numbers. They use a 1,000-video sample and frozen encoders rather than an ActivityNet-trained retrieval model.

Scope and reproducibility

Evaluation protocol

• Frozen encoders, cosine similarity, L2-normalized embeddings.
• Default clip: 8 frames at 224x224.
• Reported headline values select the best tested temporal protocol and pool.
• Available protocols: uniform8, vt_1fps_n12, vt_2fps_n32, and reid_8fps_n64.
• Available pools: mean, mean_s8, max, and query_max.
• query_max is this repository's late-interaction diagnostic; it is not the official X-Pool implementation.

Cost versus accuracy

This figure belongs to the TVPReid + RSTPReid + GroOT-MOT17 experiments, not ActivityNet. Each point is one frozen model. The x-axis is the measured time to encode one 8-frame clip on a T4; the y-axis is mean Rank-1 across the six available non-action test sets: TVPReid PRID, iLIDS, and Duke; RSTPReid; and GroOT-MOT17 appearance and combined queries. GroOT action is excluded because its highly repetitive captions create a dataset ceiling rather than a meaningful model comparison. Timings were collected in the benchmark runs and are hardware/runtime-specific, so the chart is a practical trade-off view, not a universal speed ranking.

The extended checkpoint contains all 156 completed configurations for OpenAI CLIP ViT-L/14, Jina CLIP v2, GME-Qwen2-VL-2B, and Qwen3-VL-Embedding-2B across PRID, iLIDS, and Duke: results/checkpoints/extended_zero_shot_4models_3datasets_v1.csv.

Repository layout

Large datasets, model caches, generated frames, checkpoints, build/, results/ outputs, and notebook checkpoint folders are intentionally ignored by Git. The tracked extended CSV is the exception: it is the compact, reviewable experiment checkpoint.

Install

Python 3.10+ is required. Install the common environment or a model-specific extra:

pip install -e ".[all]"
# or: pip install -e ".[xclip]"
# or: pip install -e ".[extended_modern]"
# or: pip install -e ".[extended_gme]"

extended_modern uses Transformers 4.57.3 for OpenAI CLIP, Jina CLIP v2, and Qwen3-VL-Embedding. extended_gme uses Transformers 4.51.3 and is kept separate because the two profiles are not mutually compatible.

Run an evaluation

python scripts/eval_encoder.py --model xclip --data-root "../Person-ReID-BenchMark/datasets/MARS" --ann-root "/path/to/tv-mars-captions" --device cuda

For sliding windows and pooling (including Method C / global_local):

python scripts/eval_encoder.py --model languagebind --windows --stride 4 --sample-fps 2 --max-frames 32 --pools mean,query_max,global_local --hybrid-alpha 0.5

Re-run the same command on TVPReid, RSTPReid, and GroOT (via dataset loaders in the Kaggle notebook or shawaf-vlm-eval) to compare pooling strategies on Tier A benchmarks.

Interactive demo (Gradio + optional Qdrant)

pip install -e ".[demo]"
# If transformers breaks after demo install: scripts/env/repair_crowd_gpu_deps.ps1

python scripts/download_tier_a.py   # TVPReid + GroOT (HF) + RSTPReid (Drive)
docker compose up -d qdrant         # optional; demo falls back to on-disk numpy index
python scripts/index_qdrant.py --model siglip2 --dataset tvpreid:prid
python demo/app.py
python scripts/smoke_demo_stack.py   # smoke checks (no server)

Set SHAWAF_MODEL_GROUP to base, extended_modern, or extended_gme when switching encoder families (restart the app). Downloads use HF_TOKEN / HUGGING_FACE_HUB_TOKEN from the environment (CLI only, not Gradio).

See docs/datasets_catalog.md for integrated and planned text-to-person datasets.

The loaders also support the public dataset mirrors used by the notebooks: TVPReid, GroOT-MOT17, and ActivityNet Captions. MSMT17/RSTPReid data is not redistributed here.

Notebooks

• notebooks/kaggle-zero-shot.ipynb: multi-dataset zero-shot benchmark.
• notebooks/colab_extended_zero_shot.ipynb: extended-model TVPReid checkpoint.
• notebooks/activity-net-siglip-irra-xclip-perception.ipynb: ActivityNet base models, first group.
• notebooks/activity-net-languagebind-internvideo.ipynb: ActivityNet base models, second group.
• notebooks/kaggle_finetuning.ipynb: exploratory InternVideo2 fine-tuning; it is not part of the frozen tables.

Dataset-specific paths can be overridden with SHAWAF_TVPREID_ROOT, SHAWAF_RSTPREID_ROOT, SHAWAF_GROOT_ROOT, SHAWAF_MARS_ROOT, and SHAWAF_TV_MARS_ANN.

Limitations

• Results are zero-shot and depend on the selected checkpoint, GPU runtime, frame sampling, and pool. They are not claims about fine-tuned performance.
• ActivityNet uses a 1,000-video sample, so it should be treated as a transfer diagnostic rather than a direct comparison with published benchmarks.
• GroOT action captions are not sufficiently diverse for a strong action retrieval conclusion.
• Extended models were not evaluated on every dataset; the extended CSV covers TVPReid only.
• The TV-MARS captions are gated. Follow the dataset agreement rather than committing the data to this repository.

License

See LICENSE. Dataset and checkpoint licenses remain those of their original providers.