All dataset and models can be found at ShareGPTVideo.
Contents: Test video frames: contains video frames used for testing
Testing pipeline refer to LLaVA-Hound-DPO test
Inference and results: result, replying to issue
DPO ranking results: dpo ranking
git clone git@github.com:RifleZhang/LLaVA-Hound-DPO.git
source setup/setup_env.sh
source setup/setup_test_data.sh
Video Frames:
video_data
└── test
├── actnet
├── msrvtt
├── msvd
├── ssv2
├── tgif
├── vidal
└── webvid
For raw videos, we upload in ShareGPTVideo/test_video_data
Test Video QA Data:
video_instruction
├── test
│ ├── actnet.qa.jsonl
│ ├── msrvtt.qa.jsonl
│ ├── msrvtt.qa.official.jsonl
│ ├── msvd.qa.jsonl
│ ├── msvd.qa.official.jsonl
│ ├── ssv2.qa.jsonl
│ ├── tgif.qa.jsonl
│ ├── tgif.qa.official.jsonl
│ ├── vidal.qa.jsonl
│ └── webvid.qa.jsonl
└── test_result
├── eval_results_official.jsonl
{
"id":"v_video7012_0",
"video":"test/msrvtt/video7012",
"conversations":[
{
"from":"human",
"value":"<video>\nwhat is a suit man doing?"
},
{
"from":"gpt",
"value":"talk"
}
]
}
{
"id":"video1_0",
"video":"test/msrvtt/video1",
"conversations":[
{
"from":"human",
"value":"<video>\nWhat is being added to the red liquid in the cooking pot at the beginning of the video?"
},
{
"from":"gpt",
"value":"All-purpose seasoning and curry powder are being added to the red liquid in the cooking pot at the beginning of the video."
}
],
"caption":"The video opens with a close-up of a cooking pot containing a red liquid to which all-purpose seasoning and curry powder are being added. The text overlay mentions these ingredients. The next frame shows a hand holding a container with dried/smoked prawns over a cooking pot; a text overlay confirms the ingredient. Subsequently, the pot is now shown with prawns added, stirring is underway, and the same descriptive text is visible. The following frames reveal someone adding dried crayfish to the pot, with a textual indication of the ingredient and giving an alternative preference option. Then, one or two scoops of tomato puree are added to the mixture in the pot, and a textual description quantifies the amount. The scene transitions to show someone adding salt to the pot and stirring its contents, as narrated by the respective text overlay. Finally, a lid is being placed on the pot, and the instruction to cover and cook for 30-45 minutes appears as text on the screen."
}
send to project page
@misc{zhang2024direct,
title={Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward},
author={Ruohong Zhang and Liangke Gui and Zhiqing Sun and Yihao Feng and Keyang Xu and Yuanhan Zhang and Di Fu and Chunyuan Li and Alexander Hauptmann and Yonatan Bisk and Yiming Yang},
year={2024},
eprint={2404.01258},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
30 commits
All dataset and models can be found at ShareGPTVideo.
Contents: Test video frames: contains video frames used for testing
Testing pipeline refer to LLaVA-Hound-DPO test
Inference and results: result, replying to issue
DPO ranking results: dpo ranking
git clone git@github.com:RifleZhang/LLaVA-Hound-DPO.git
source setup/setup_env.sh
source setup/setup_test_data.sh
Video Frames:
video_data
└── test
├── actnet
├── msrvtt
├── msvd
├── ssv2
├── tgif
├── vidal
└── webvid
For raw videos, we upload in ShareGPTVideo/test_video_data
Test Video QA Data:
video_instruction
├── test
│ ├── actnet.qa.jsonl
│ ├── msrvtt.qa.jsonl
│ ├── msrvtt.qa.official.jsonl
│ ├── msvd.qa.jsonl
│ ├── msvd.qa.official.jsonl
│ ├── ssv2.qa.jsonl
│ ├── tgif.qa.jsonl
│ ├── tgif.qa.official.jsonl
│ ├── vidal.qa.jsonl
│ └── webvid.qa.jsonl
└── test_result
├── eval_results_official.jsonl
{
"id":"v_video7012_0",
"video":"test/msrvtt/video7012",
"conversations":[
{
"from":"human",
"value":"<video>\nwhat is a suit man doing?"
},
{
"from":"gpt",
"value":"talk"
}
]
}
{
"id":"video1_0",
"video":"test/msrvtt/video1",
"conversations":[
{
"from":"human",
"value":"<video>\nWhat is being added to the red liquid in the cooking pot at the beginning of the video?"
},
{
"from":"gpt",
"value":"All-purpose seasoning and curry powder are being added to the red liquid in the cooking pot at the beginning of the video."
}
],
"caption":"The video opens with a close-up of a cooking pot containing a red liquid to which all-purpose seasoning and curry powder are being added. The text overlay mentions these ingredients. The next frame shows a hand holding a container with dried/smoked prawns over a cooking pot; a text overlay confirms the ingredient. Subsequently, the pot is now shown with prawns added, stirring is underway, and the same descriptive text is visible. The following frames reveal someone adding dried crayfish to the pot, with a textual indication of the ingredient and giving an alternative preference option. Then, one or two scoops of tomato puree are added to the mixture in the pot, and a textual description quantifies the amount. The scene transitions to show someone adding salt to the pot and stirring its contents, as narrated by the respective text overlay. Finally, a lid is being placed on the pot, and the instruction to cover and cook for 30-45 minutes appears as text on the screen."
}
send to project page
@misc{zhang2024direct,
title={Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward},
author={Ruohong Zhang and Liangke Gui and Zhiqing Sun and Yihao Feng and Keyang Xu and Yuanhan Zhang and Di Fu and Chunyuan Li and Alexander Hauptmann and Yonatan Bisk and Yiming Yang},
year={2024},
eprint={2404.01258},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
30 commits