LMM-EVQA

---

📖 Introduction

The rapid proliferation of user-generated content (UGC) on short-form video platforms has made video engagement prediction increasingly important for optimizing recommendation systems and guiding content creation. However, this task remains challenging due to the complex interplay of factors such as semantic content, visual quality, audio characteristics, and user background.

In this work, we empirically investigate the potential of large multimodal models (LMMs) for video engagement prediction. We adopt two representative LMMs:

• VideoLLaMA2: Integrates audio, visual, and language modalities
• Qwen2.5-VL: Models only visual and language modalities

Both models demonstrate competitive performance against state-of-the-art baselines, showcasing the effectiveness of LMMs in engagement prediction. Notably, VideoLLaMA2 consistently outperforms Qwen2.5-VL, highlighting the importance of audio features in engagement prediction.

🏆 Performance

ICCV VQualA 2025 EVQA Challenge Results

Note: Final score = 0.6 × SROCC + 0.4 × PLCC

For more detailed results, please refer to the challenge report paper.

🏗️ Model Architecture

📊 Dataset

SnapUGC Dataset

📥 Download Links

📄 CSV Files

• Training CSV: Google Drive
• Validation CSV: Google Drive

🎥 Video Files

• Training Videos: Google Drive | Baidu Yun (extraction code: edts)
• Validation Videos: Google Drive | Baidu Yun (extraction code: 28tq)

📁 Dataset Structure

SnapUGC/
├── train_data.csv          # Training annotations
├── val_data.csv            # Validation annotations  
├── train_videos/           # Training video files
└── val_videos/            # Validation video files

🚀 Quick Start

VideoLLaMA2

Environment Setup

cd VideoLLaMA2-audio_visual

# Create conda environment
conda create -n videollama2 python=3.9
conda activate videollama2

# Install dependencies
pip install -r requirements.txt

Download Pre-trained Models

# Download model weights
python download_model_weight.py
# --local_dir specifies the target directory for model weights

Dataset Preparation

# Prepare training dataset
python prepare_dataset.py
# VIDEO_DIR = " " # Path to video directory
# AUDIO_DIR = " " # Audio directory parameter (deprecated) - VideoLLaMA2 automatically extracts audio features from video files
# CSV_PATH = "train_data.csv" # Input CSV file containing video metadata
# OUTPUT_TRAIN = "train.json" # Output JSON file for training data

Training

# Start training
sh train.sh
# --model_path # Path to pre-trained model weights
# --data_path # Path to training data JSON file
# --num_frames # Number of frames to extract from each video (first N frames)
# --output_dir # Directory for saving trained model weights
# --num_train_epochs # Total number of training epochs
# --per_device_train_batch_size # Batch size per GPU device

Testing

Batch Testing:

# Generate validation dataset
python prepare_dataset.py

# Run validation
sh run_validation.sh
# --model-path # Path to the trained model weights
# --modal-type # Specify "av" mode for audio-visual evaluation
# --json_file # Path to the validation JSON file

Single Video Testing:

python videollama2/test_single_video.py \
    --model-path /path/to/trained/model \
    --modal-type av \
    --video-path /path/to/video.mp4 \
    --title "Your Video Title" \
    --description "Your Video Description"

Parameters:

• --model-path: Path to trained model weights (required)
• --modal-type: Modal type - "a" (audio), "v" (video), "av" (audio-visual)
• --video-path: Path to video file (required)
• --title: Video title (optional, default: None)
• --description: Video description (optional, default: None)

Qwen2.5-VL

Environment Setup

cd Qwen2.5-VL

# Create conda environment
conda create -n qwenvl python=3.9
conda activate qwenvl

# Follow official Qwen2.5-VL installation
# https://github.com/QwenLM/Qwen2.5-VL

Dataset Preparation

# Prepare dataset for testing
python prepare_dataset_qwenvl.py
# VIDEO_DIR = " " # Path to the folder containing video files
# CSV_PATH = " " # Path to the provided CSV file
# OUTPUT_TRAIN = " " # Path to the output JSON file used for testing

Testing

Batch Testing:

python infer_evqa.py \
    --model-path /path/to/trained/model \
    --video-folder /path/to/video/folder \
    --question-file /path/to/question.json \
    --save-csv /path/to/result.csv

Single Video Testing:

python test_single_video_qwenvl.py \
    --model-path /path/to/trained/model \
    --video-path /path/to/video.mp4 \
    --title "Your Video Title" \
    --description "Your Video Description"

Parameters:

• --model-path: Path to trained Qwen2.5-VL model weights (required)
• --video-path: Path to video file (required)
• --title: Video title (optional, default: None)
• --description: Video description (optional, default: None)

📦 Pre-trained Models

VideoLLaMA2

• Download: Baidu Yun (extraction code: 3aqc)

Qwen2.5-VL

• Download: Baidu Yun (extraction code: 98zr)

📝 Citation

If you find this code useful for your research, please cite our paper:

@inproceedings{sun2025engagement,
  title={Engagement Prediction of Short Videos with Large Multimodal Models},
  author={Sun, Wei and Cao, Linhan and Cao, Yuqin and Zhang, Weixia and Wen, Wen and Zhang, Kaiwei and Chen, Zijian and Lu, Fangfang and Min, Xiongkuo and Zhai, Guangtao},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision (ICCV) Workshops},
  pages={1--10},
  year={2025}
}

📄 License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.

🙏 Acknowledgments

• Thanks to the organizers of VQualA 2025 and Snap Chat for hosting the challenge
• Thanks to the SnapUGC dataset providers
• Thanks to VideoLLaMA2 for providing the excellent multimodal foundation model
• Thanks to Qwen2.5-VL for providing the powerful vision-language model
• Thanks to the open-source community for providing excellent tools and frameworks

---

⭐ Star this repository if you find it helpful!