Dataset used in D2VLM

Paper | Code

Here we provided the instruction for dataset preparation. We provided the transformed annotation (or steps for transformation) tailor for D2VLM, due to the new training objective design.

E.T. Bench

1. Download and process the data following official document.

2. Use the annotations provided in this Hugging Face repo (i.e., D2VLM-Dataset/ETBench/evi).

Charades-STA

1. Follow the provided link in the table above to download the raw videos and the annotations. The folder is organized as follows.

xx/charades
├─ Charades_v1 (Downloaded and extracted raw videos from Charades_v1.zip)
│  ├─ 001TG.mp4
│  ├─ 003WS.mp4
│  └─ ...
└─ test.caption_coco_format.json

2. Run the post-processing scripts (in project repo). Remember to update paths in the .py files referenced by the script below.

bash other_benchmark_organize/charades/run.sh

Youcook2

1. Follow the provided link in the table above to download the raw videos and the annotations. The folder is organized as follows.

xx/youcook2
├─ raw_videos (Downloaded and extracted raw videos from raw_videos.tar.gz)
│  ├─ testing
│  ├─ training
│  └─ validation
└─ val.caption_coco_format.json

2. Run the post-processing scripts (in project repo). Remember to update paths in the .py files referenced by the script below.

 bash other_benchmark_organize/youcook2/run.sh

E.T. Instruct

1. Download and process the data following official document.

2. Use the annotations provided in this Hugging Face repo.

• D2VLM-Dataset/ET-Instruct/evi.json --> For supervised finetuning (SFT).

• /D2VLM-Dataset/ET-Instruct/FPO/tokenized_fpo_annotation.pt --> For FPO.

For FPO, we applied an optimization: the dispreferred sample is appended after the preferred (positive) sample within the same sequence, and we adjust the positional encodings and attention mask accordingly. This makes training more efficient than naively training paired samples as separate batch items. One may check the fpo_anno_gen/run.sh in the project repo to learn more.

Citation

If you find our work useful in your research, please consider citing our paper:

@inproceedings{d2vlm,
  title={Factorized Learning for Temporally Grounded Video-Language Models},
  author={Zeng, Wenzheng and Gao, Difei and Shou, Mike Zheng and Ng, Hwee Tou},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
  year={2025},
  pages={20683-20693}
}