Soccer Video Understanding — Thesis Codebase

This repository contains the full codebase developed for a Master's thesis on soccer video understanding. The central contribution is TrackSupport: a track-augmented, multi-view foul classification model built on V-JEPA 2, combined with evaluation environments for three SoccerNet challenges.

---

Repository Structure

.
├── tracksupport/          # ★ Main contribution: Hydra-driven training system
│   ├── cfg/               #   Hydra config tree (model / training / modules)
│   └── src/               #   Source code (models, datasets, training loops)
│
├── vjepa2/                # Official V-JEPA 2 implementation (Meta FAIR)
│
├── replicate_MVF/         # TAda-based MVFoul baseline replication
│
└── soccernet_suite/       # SoccerNet challenge environments
    ├── sn-mvfoul/         #   VARS — multi-view foul classification baseline
    ├── sn-teamspotting/   #   Team Ball Action Spotting (T-DEED baseline)
    └── FAANTRA/           #   Ball Action Anticipation (FUTR-based)

---

Main System — TrackSupport

TrackSupport classifies foul type (8 classes) and severity (4 classes) from multi-view soccer clips. It pairs a V-JEPA 2 encoder (LoRA fine-tuned) with an RT-DETR object tracker; player tracks are injected into the encoder via TrackTention cross-attention before an attentive pooler aggregates the multi-view embeddings into a final prediction.

Full usage details are in tracksupport/README.md.

Installation

# Requires Python ≥ 3.11 and CUDA-capable GPUs.
# Install via uv (recommended):
pip install uv
uv sync                        # installs all dependencies from uv.lock

# Or with pip:
pip install -r tracksupport/requirements.txt   # if present, else:
pip install -e .

Dataset

Download the SoccerNet MVFoul dataset (NDA required):

from SoccerNet.Downloader import SoccerNetDownloader as SNdl
dl = SNdl(LocalDirectory="path/to/data")
dl.downloadDataTask(task="mvfouls", split=["train","valid","test","challenge"],
                    password="<NDA password>")

Then set data_path in tracksupport/cfg/training/default.yaml to the folder containing the unpacked train/, valid/, test/, and Chall/ directories.

Training

All hyperparameters live in the Hydra config tree under tracksupport/cfg/.
Select a backbone via model=<name> on the command line:

cd tracksupport
torchrun --nproc_per_node=<N_GPUS> src/runs/train.py model=vjepa

Available model configs:

Common config overrides (append to the command above):

# Change number of GPUs / batch size / learning rate:
torchrun --nproc_per_node=2 src/runs/train.py model=vjepa \
    training.num_gpus=2 \
    training.train_batch_size=4 \
    training.lr=5e-4

# Resume from a checkpoint:
torchrun --nproc_per_node=4 src/runs/train.py model=vjepa \
    training.checkpoint.resume=true \
    training.checkpoint.resume_from=checkpoints/best_checkpoint.pt

# Disable WandB logging:
torchrun --nproc_per_node=4 src/runs/train.py model=vjepa \
    training.wandb.enable=false

# Quick debug run on a small data subset:
torchrun --nproc_per_node=1 src/runs/train.py model=vjepa \
    training.debug.enable=true training.debug.size=64

See tracksupport/README.md for the complete config reference.

---

SoccerNet Challenges

Each challenge sub-directory is self-contained with its own README.

MVFoul — Multi-View Foul Classification

soccernet_suite/sn-mvfoul/

The VARS baseline. Uses argparse rather than Hydra.

cd soccernet_suite/sn-mvfoul/"VARS model"
python main.py --path path/to/mvfoul_dataset

Team Ball Action Spotting

soccernet_suite/sn-teamspotting/

T-DEED baseline adapted to predict both the action and which team performed it.

cd soccernet_suite/sn-teamspotting
# 1. Extract frames (first run only):
uv run extract_frames_snb.py --out_dir /data/soccernetball/frames \
    --video_dir /data/soccernetball
# 2. Train:
python train_tdeed_bas.py SoccerNetBall_baseline

Update frame paths and checkpoint locations in config/SoccerNetBall/SoccerNetBall_baseline.json before training. See the sub-directory README for the full step-by-step.

Ball Action Anticipation (FAANTRA)

soccernet_suite/FAANTRA/

FUTR-based action anticipation baseline. Config driven by JSON files under config/.

cd soccernet_suite/FAANTRA
# Download dataset (BAA):
python setup_dataset_BAA.py --download-key <NDA key>
# Train:
python main.py config/your_config.json model_name
# Evaluate on a specific split:
python test.py config/your_config.json path/to/checkpoint.pth model_name -s test

---

Baselines — TAda Replication

replicate_MVF/

Replication of the Temporally-Adaptive Convolutions (TAda) baselines using the PyTorch Video Understanding framework. See the sub-directory README for model and feature zoos.

---

V-JEPA 2 Reference

vjepa2/

Official Meta FAIR implementation included as a reference. TrackSupport loads V-JEPA 2 weights directly from HuggingFace (facebook/vjepa2-vith-fpc64-256) and does not require this directory to be installed separately.

---

Dependencies

All dependencies are managed at the root level via uv:

pip install uv
uv sync

For reproducibility the resolved lock file uv.lock is committed. The key runtime requirements are:

• Python ≥ 3.11
• PyTorch ≥ 2.9 (with CUDA)
• hydra-core ≥ 1.3.2
• transformers ≥ 5.3 (for HuggingFace V-JEPA 2 weights)
• peft ≥ 0.19 (LoRA adapters)
• ultralytics ≥ 8.4 (RT-DETR tracker)
• pytorch-ignite ≥ 0.5.3 (training engine)
• wandb ≥ 0.25 (experiment tracking)
• soccernet ≥ 0.1.62 (dataset download utilities)

---

Citation

If you use this code, please cite the relevant challenge papers:

@InProceedings{Held2023VARS,
    author    = {Held, Jan and Cioppa, Anthony and Giancola, Silvio and Hamdi, Abdullah and Ghanem, Bernard and Van Droogenbroeck, Marc},
    title     = {{VARS}: Video Assistant Referee System for Automated Soccer Decision Making From Multiple Views},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
    year      = {2023},
    pages     = {5085--5096}
}

@InProceedings{Dalal_2025_CVPR,
    author    = {Dalal, Mohamad and Xarles, Artur and Cioppa, Anthony and Giancola, Silvio and Van Droogenbroeck, Marc and Ghanem, Bernard and Clap\'{e}s, Albert and Escalera, Sergio and Moeslund, Thomas B.},
    title     = {Action Anticipation from SoccerNet Football Video Broadcasts},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
    year      = {2025},
    pages     = {6079--6090}
}

@inproceedings{xarles2024t,
    title     = {T-DEED: Temporal-Discriminability Enhancer Encoder-Decoder for Precise Event Spotting in Sports Videos},
    author    = {Xarles, Artur and Escalera, Sergio and Moeslund, Thomas B and Clap{\'e}s, Albert},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
    pages     = {3410--3419},
    year      = {2024}
}