EgoMemReason

A Memory-driven Reasoning Benchmark for Long-Horizon Egocentric Video Understanding

Accepted at COLM 2026.

Project page · Paper · Benchmark (HF) · Leaderboard (HF Space)

⚠️ Benchmark revision — v1.1 (2026-07-13). The per-question option-letter mapping in annotations_public.jsonl has been reshuffled. If you downloaded the benchmark before 2026-07-13, please re-download it before your next evaluation run — letters A–J now map to different option strings. Question text and the set of option strings per question are unchanged, so previously published aggregate scores remain valid; but any locally-stored letter-only predictions must be remapped (or the model re-run) against the new mapping before submitting to the leaderboard.

Ziyang Wang*, Yue Zhang*, Shoubin Yu, Ce Zhang, Zengqi Zhao, Jaehong Yoon, Hyunji Lee, Gedas Bertasius, Mohit Bansal UNC Chapel Hill · NTU Singapore — Equal contribution

---

Overview

Next-generation visual assistants — smart glasses, embodied agents, always-on life-logging systems — must reason over an entire day or more of continuous visual experience. In ultra-long video, relevant information is sparsely distributed across hours or days, making memory the fundamental challenge: models must accumulate information over time, recall previously observed states, track temporal order, and abstract recurring patterns.

EgoMemReason is a comprehensive benchmark that systematically evaluates week-long egocentric video understanding through the lens of memory-driven reasoning. It targets three complementary memory types:

• Entity memory — track how object states evolve across days
• Event memory — recall and order activities separated by hours or days
• Behavior memory — abstract recurring patterns from sparse, repeated observations

500 multiple-choice questions across 6 core challenges, with 5.1 evidence segments / question and 25.9 h memory backtracking on average — 2× both metrics over the strongest prior week-long benchmark.

Main Results

We evaluate 17 systems spanning general-purpose MLLMs, video-specific MLLMs, and agentic video frameworks. The strongest model reaches 39.6% overall — long-horizon memory is far from solved. Numbers are from the paper's Table 1; best in each column bold, second-best underlined.

Repository Layout

EgoMemReason/
├── data/                  # Dataset access instructions (see data/README.md)
├── evaluation/            # Per-model evaluation scripts (one folder per system)
│   ├── Gemini/            # Gemini-3 Flash / 3.1 Pro
│   ├── GPT5/              # GPT-5 (via Azure OpenAI)
│   ├── InternVL/          # InternVL3.5-8B / 38B
│   ├── InternVideo/       # InternVideo2.5-8B
│   ├── LongVA/            # LongVA-7B
│   ├── Molmo2/            # Molmo2-8B
│   ├── Qwen3VL/           # Qwen-3-VL 8B / 32B / 30B-A3B (+ ablations)
│   └── VideoLLaMA3/       # VideoLLaMA3-8B
└── agentic/               # Agentic video frameworks
    ├── AVP/               # Ours — Agentic Video Pipeline (Gemini backbone)
    ├── EgoR1/             # Ego-R1 reasoning agent
    ├── SILVR/             # SiLVR
    └── WorldMM/           # WorldMM retrieval + reasoning framework

Each method directory ships only the scripts for evaluation. The underlying model code (LongVA, VideoLLaMA3, Ego-R1-Agent, etc.) is not vendored — install each from its upstream repo, then point the run script at the right environment. See each subfolder's README for the exact upstream link and install snippet.

Quick Start

1. Get the benchmark

The 500 questions are on Hugging Face: https://huggingface.co/datasets/Ted412/EgoMemReason

hf download Ted412/EgoMemReason annotations_public.jsonl --repo-type dataset --local-dir ./data
# Video frames come from EgoLife (separate license): https://egolife-ai.github.io/
# Then set:
export EGOMEM_DATA=/path/to/benchmark.json            # see data/README.md
export EGOLIFE_FRAMES_INDEX=/path/to/egolife_frames_index.json

See data/README.md for the full schema, the frame-index format, and how submissions are scored.

2. Run a method

Closed-source API models (Gemini, GPT-5) are the simplest:

cd evaluation/Gemini
export GOOGLE_API_KEY=...
INPUT_JSON=$EGOMEM_DATA \
FRAMES_INDEX=$EGOLIFE_FRAMES_INDEX \
bash run_final_benchmark_500_apr22_gemini_flash_512.sh 20   # 20 parallel jobs

Open-source MLLMs require the upstream environment first (see per-folder README), then:

cd evaluation/Qwen3VL
INPUT_JSON=$EGOMEM_DATA bash run_final_benchmark_500_apr22.sh

3. Score a run

Each eval_*.py writes a JSON file with one prediction per question. Per-task / overall accuracy is printed at the end of every run. Prediction shards from parallel jobs are merged by merge_temporal_ordering_eval_shards.py (under evaluation/Gemini/ and evaluation/GPT5/).

4. Test your model and (optionally) join the leaderboard

Convert your prediction file to the submission format (one entry per question, all 500):

import json
src = json.load(open("results_my_model.json"))
sub = [{"example_id": r["example_id"], "predicted_answer": r["pred"]} for r in src]
json.dump(sub, open("submission.json", "w"))

Get your score — anyone can score anonymously on the EgoMemReason Leaderboard Space → Score tab. Upload submission.json and the Space returns per-split + Overall accuracy against the held-out answer key. The score is shown to you only; nothing is stored and it does not appear on the public leaderboard.

Ask to be listed — the public leaderboard is curated by hand. To be added, email ziyangw@cs.unc.edu with submission.json attached, plus:

• method name, team name, model size / API tier, video-input modality,
• one-sentence method description,
• project page URL, arXiv or OpenReview URL.

The maintainer will re-score against the private answer key, sanity-check the metadata, and add a row within ~5 business days.

Adding a New Method

1. Create evaluation/<YourMethod>/ (or agentic/<YourMethod>/).
2. Implement an inference script that reads final_benchmark_500_apr22.json and writes [{"id", "predicted_answer", "correct", ...}, ...].
3. Compare against the schema used by any existing method (e.g. evaluation/Gemini/eval_gemini_frames.py is a clean reference).
4. Report Tracking / Counting / Ordering / Linking / Spatial / Activity / Overall — splits are tagged in the dataset.

Citation

@misc{wang2026egomemreasonmemorydrivenreasoningbenchmark,
      title={EgoMemReason: A Memory-Driven Reasoning Benchmark for Long-Horizon Egocentric Video Understanding},
      author={Ziyang Wang and Yue Zhang and Shoubin Yu and Ce Zhang and Zengqi Zhao and Jaehong Yoon and Hyunji Lee and Gedas Bertasius and Mohit Bansal},
      year={2026},
      eprint={2605.09874},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2605.09874},
}

License

Code in this repository is released under the MIT License (see LICENSE). The benchmark is built on top of EgoLife — please follow EgoLife's data license when using the underlying video frames.