Challenge infrastructure accompanying the paper Benchmarking Cross-Modal Learning from Naturalistic Egocentric Video Data. Brought to you by researchers at Meta, Stanford, ENS and UTokyo.

The challenge

Train a VLM on the BabyView 2025.1 corpus (≈863 h of head-mounted-camera video from children, collected by Stanford University and hosted on NYU Databrary) and nothing else — no extra image, video, text or audio data may be used for any encoder pretraining, fine-tuning or evaluation. The challenge is to beat the baselines on a fixed evaluation suite and close the gap to web-scale pretrained models.

Submissions are scored on three families of tasks, each with subgroup aggregates and an overall:

The paper shows weakly-aligned naturalistic egocentric input drives contrastive and generative reference baselines to near-chance on the cross-modal probes, while curated captions (COCO) approach off-the-shelf CLIP. The challenge is to close that gap to web-scale pretrained models algorithmically, without changing the data.

Repository layout

Each directory under apps/ has a README with component-specific usage.

apps/
├── data_preprocessing/   # video → frames + WhisperX transcripts + train/val/test manifests
├── baselines/
│   ├── dinov2/           # DINOv2 SSL + ViT-B/14 feature extractor
│   ├── lm_training/      # BERT MLM and GPT-2 from-scratch trainers
│   ├── clip/             # CLIP+ contrastive trainer (4 modes: contrastive, +MLM, +DINOv2, triple)
│   └── llava/            # LLaVA-style generative VLM trainer
├── alignment_scoring/    # CLIP / VQA / STS / captioning pipelines for cross-modal alignment scoring
├── benchmark_creation/   # Machine-DevBench corpus-grounded benchmark generator
└── swapbench/            # LongTail-Swap + VP-Swap generators
core/                     # Protocols (FeatureExtractor, …), DDP / config / seed utils
evaluation/               # Hydra+Stopes eval pipeline (vision, text, multimodal task launchers)
docs/                     # Per-component design notes
scripts/eval_data/        # Eval-dataset download helpers
tests/                    # Unit + opt-in integration tests

Submitting

A submission is leaderboard scores plus, ideally, the training code that produced them. The minimum end-to-end path:

1. Cache the eval datasets — see docs/eval_data.md. The download scripts handle the open-access ones automatically; only ImageNet and NYUv2 need a one-time manual prep documented inline.
2. Validate the eval setup on an off-the-shelf model — point the evaluation/eval_launcher at model=dino / model=clip_image / model=bert_base to sanity-check that the three eval families run on your machine and reproduce numbers in the same ballpark as the leaderboard's off-the-shelf rows.
3. Train your own model on BabyView — any architecture, any objective, as long as the training data comes only from BabyView 2025.1 (no extra image / video / text / audio data for pretraining, fine-tuning or evaluation).
4. Score your model with evaluation/eval_launcher over the three families.
5. Open a PR against the leaderboard repo with a JSON file containing your scores and a link to the training code so the submission can be reproduced. See the leaderboard submission page for the JSON schema and PR template.

[!IMPORTANT] Submissions linking to runnable training code are marked reproducible. Organizers don't rerun the code — the mark just signals anyone could. Leaderboard-only submissions (paper or report without runnable code) are welcome but unmarked.

[!TIP] For validation and method exploration we recommend training on Ego4D rather than BabyView. Ego4D is naturalistic egocentric video too — just not developmentally plausible — and is faster to iterate on (more shards, fewer access hoops). Then re-train your final submission on BabyView 2025.1.

Reference baselines

The apps/baselines/ trees ship the four trainers we used in the paper, along with apps/data_preprocessing/ for the BabyView → manifest pipeline they consume. Useful as starting points, sanity checks, or warm-starts for your own approach — but bring your own trainer if you prefer; the challenge encourages alternative architectures, objectives and curricula.

[!NOTE] apps/alignment_scoring/, apps/benchmark_creation/ and apps/swapbench/ are toolkits used to produce the paper's analyses and benchmarks. Not on the critical path for a submission, but useful for re-running those experiments or generating benchmarks for a new training corpus.

Install

The full environment is pinned in pixi.toml (Python 3.12, PyTorch 2.8 + CUDA 12.6).

# install pixi: https://pixi.sh/latest/installation/
pixi install -e dev

This produces installable CLI entry points. The most commonly used:

egobabyvlm-train-contrastive              # apps/baselines/clip/training
egobabyvlm-extract-frames                 # apps/data_preprocessing/frames
egobabyvlm-transcribe-whisperx            # apps/data_preprocessing/transcription
egobabyvlm-build-clip-manifest            # apps/data_preprocessing/manifests

egobabyvlm-extract-frames                 # apps/data_preprocessing/frames
egobabyvlm-transcribe-whisperx            # apps/data_preprocessing/transcription
egobabyvlm-filter-vtc                     # apps/data_preprocessing/transcription (BabyView KCHI)
egobabyvlm-build-clip-manifest            # apps/data_preprocessing/manifests
egobabyvlm-train-contrastive              # apps/baselines/clip/training
egobabyvlm-export-text-encoder-to-hf      # apps/baselines/clip/scripts (BERT → HF for lm_eval)
egobabyvlm-swapbench-build-word-lists     # apps/swapbench (LongTail-Swap + VP-Swap)
egobabyvlm-swapbench-lt-swap              # apps/swapbench
egobabyvlm-swapbench-vp-swap              # apps/swapbench
alignment-{clip,sts,vqa}-scoring          # apps/alignment_scoring
alignment-captioning                      # apps/alignment_scoring
alignment-finetune-lora                   # apps/alignment_scoring (LoRA finetune)
alignment-create-shuffled-manifest        # apps/alignment_scoring

Quickstart

Run an evaluation

Download the eval datasets once (see docs/eval_data.md), then point the launcher at your model:

# All vision tasks
pixi run -e dev python -m evaluation.eval_launcher \
    eval=vision/vision_pipeline \
    model=dino \
    name=my_run

# All Machine-DevBench tasks (realistic + cartoon styles)
pixi run -e dev python -m evaluation.eval_launcher \
    eval=multimodal/machine_devbench_pipeline \
    model=clip_image \
    name=my_run

# All text tasks
pixi run -e dev python -m evaluation.eval_launcher \
    eval=text/text_pipeline \
    model=bert_base \
    name=my_run

Override model=… to swap encoders or any individual task YAML in evaluation/configs/eval/. Off-the-shelf model configs (dino, clip_image, bert_base, …) are useful for sanity-checking the eval setup before plugging in your own.

Train a reference baseline (optional)

The shipped baselines reproduce paper numbers and can serve as starting points.

pixi run -e dev torchrun --standalone --nproc-per-node=4 \
    -m apps.baselines.clip.training.train \
    name=babyview_clip mode=triple data=ego4d \
    data.train_dataset.manifest_path=/path/to/manifests/train.json \
    data.train_dataset.image_root=/path/to/frames \
    data.val_dataset.manifest_path=/path/to/manifests/val.json \
    +text_only_data=default text_only_data.train_file=/path/to/utterances.txt \
    +dinov2=vitb14_coco

sbatch apps/baselines/llava/scripts/phase1_pretrain.sh
sbatch apps/baselines/llava/scripts/phase2_finetune.sh

Tests

pixi run -e dev ci   # ruff check + format + typos + pytest

GPU- and integration-marked tests are excluded by default; opt in with pytest -m gpu or pytest -m integration.

Citing our work

If you use our benchmark or find our work useful in your research, please consider citing:

@article{lin2026egobabyvlm,
  title   = {EgoBabyVLM: Benchmarking Cross-Modal Learning from Naturalistic Egocentric Video Data},
  author  = {Lin, Dongyan and Rust, Phillip and Villar-Corrales, Angel and Tan, Alvin W. M.
             and Luthra, Mahi and Saint-James, Charles-{\'E}ric and Moritz, Rashel
             and Krogh-Jespersen, Sheila and Stark, Vanessa and Parimi, Surya
             and Shen, Jiayi and Benchekroun, Youssef and Higuchi, Yosuke
             and Gleize, Martin and Fizycki, Tom and Hamilakis, Nicolas
             and Khentout, Manel and Tsuji, Sho and K{\'e}gl, Bal{\'a}zs
             and Pino, Juan and Frank, Michael C. and Dupoux, Emmanuel},
  journal = {arXiv preprint},
  year    = {2026},
  url     = {https://arxiv.org/abs/2605.19130},
}

Licenses

The majority of this code is licensed under CC-BY-NC 4.0, with the following exceptions:

Please retain all upstream copyright notices and license headers when reusing files from these directories.

The released data is CC-BY-NC and intended for benchmarking only. Some annotations are outputs of Llama 3.1 and subject to its license (Llama 3.1 license; Llama 4 model card). Third-party content pulled from other locations is subject to its own licenses, and you may have other legal obligations governing your use of that content.