PerceptionLM Video Anticipation Evaluation

Video anticipation evaluation using Meta's Perception-LM-3B model.

Data Download

Download video clips from Google Drive:

# Install gdown
pip install gdown

# Download data (50 clips: clip_1 to clip_50)
gdown --folder https://drive.google.com/drive/folders/1SRnl8DgXYZ5k-sw0Q1a-DSn47VNSV0b0 -O test_clips/

Alternatively, download manually from: Google Drive Link

Setup

Create Conda Environment

conda create -n meta_eval python=3.10
conda activate meta_eval

Install Dependencies

# PyTorch with CUDA 12.4 (adjust for your CUDA version)
pip install torch==2.6.0 torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124

# Core packages
pip install transformers huggingface-hub decord accelerate timm av

# Install ffmpeg for video processing
conda install -c conda-forge ffmpeg -y

HuggingFace Login

huggingface-cli login
# Enter your HF token when prompted

Running Inference

Basic Inference (inference.py)

python inference.py

Modify video_path and query in the script as needed.

Anticipation Evaluation (eval_anticipation.py)

Evaluates short-term anticipation using two video segments:

• Context segment: t_ctx_start to t_ant_start (custom query)
• Anticipation segment: t_ant_start to t_ant_end (custom query)

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True python eval_anticipation.py \
  --clip_id clip_1 \
  --t_ctx_start 00:00 \
  --t_ant_start 00:05 \
  --t_ant_end 00:10 \
  --query "What objects are visible in the scene?"

Parameters:

• --clip_id: Video clip ID (clip_1 to clip_50)
• --t_ctx_start: Context start time (MM:SS)
• --t_ant_start: Anticipation start time (MM:SS)
• --t_ant_end: Anticipation end time (MM:SS)
• --query: Question for context segment

Example:

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True python eval_anticipation.py \
  --clip_id clip_2 \
  --t_ctx_start 00:10 \
  --t_ant_start 00:20 \
  --t_ant_end 00:30 \
  --query "What is the person doing?"

Frame Settings

Recommended num_frames based on GPU VRAM:

• 12GB GPU: 16 frames (default in inference.py)
• 8-10GB GPU: 12 frames
• 6-8GB GPU: 8 frames

Adjust in scripts:

num_frames=16  # Change based on your GPU

Memory Optimization

For OOM errors:

1. Reduce num_frames
2. Reduce max_new_tokens
3. Use memory flag: PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
4. Clear cache between runs: torch.cuda.empty_cache()

Files

• inference.py: Basic video inference script
• eval_anticipation.py: Anticipation evaluation script
• test_clips/: Video data directory (50 clips)
• hf_token: HuggingFace token storage