OmniCVR: A Benchmark for Omni-Composed Video Retrieval with Vision, Audio, and Text

OmniCVR is a benchmark for omni-composed video retrieval: given a source video and a natural-language modification instruction, the goal is to retrieve the target video from a candidate gallery. The modifications span vision, audio, and text jointly.

Dataset summary

• 5,000 evaluation queries (source / target / instruction triples).
• Each query is paired with a 2,000-candidate retrieval gallery that always contains the ground-truth target.
• 16,316 unique videos in total.
• All video ids are anonymized to omnicvr_video{N}.mp4.

Splits

The 5,000 queries (in their line order in omnicvr.jsonl) are organized into three categories by the dominant modality of the modification:

Files

omnicvr.jsonl schema

{
  "source_id": "omnicvr_video1330.mp4",
  "target_id": "omnicvr_video1331.mp4",
  "instruction": "Maintain the ... Replace the action of ...",
  "candidates": ["omnicvr_video2298.mp4", "omnicvr_video2895.mp4", "...2000 ids..."]
}

• source_id — the query (reference) video.
• target_id — the ground-truth video to retrieve (always inside candidates).
• instruction — the textual modification describing source → target.
• candidates — the 2,000-video retrieval gallery for this query.

Gallery construction

Each 2,000-candidate gallery contains the target, the source, up to 2 hard distractors (other temporal segments of the same underlying video, where applicable), and the remainder sampled from the corresponding video pool. The audio-centric split uses a single shared 2,000-video pool.

Usage

import json

# Load annotations
with open("omnicvr.jsonl") as f:
    data = [json.loads(line) for line in f]

ex = data[0]
print(ex["source_id"], ex["target_id"])
print(ex["instruction"])
print(len(ex["candidates"]))   # 2000

# Videos: download and extract the tar shards into ./videos/
#   cat videos/omnivideos-*.tar | tar -xf - -C videos/   (or extract each shard)
# Then each id maps to videos/<id>  (ids already include the .mp4 extension)

Citation

@inproceedings{
ji2026omnicvr,
title={Omni{CVR}: A Benchmark for Omni-Composed Video Retrieval with Vision, Audio, and Text},
author={Junyang Ji and Shengjun Zhang and Da Li and Yuxiao Luo and Yan Wang and Di Xu and Biao Yang and Wei Yuan and Fan Yang and Zhihai He and Wenming Yang},
booktitle={The Fourteenth International Conference on Learning Representations},
year={2026},
url={https://openreview.net/forum?id=KxxR7emO5K}
}