EgoMemReason

Accepted at COLM 2026.

⚠️ Benchmark revision — v1.1 (2026-07-13). The per-question option-letter mapping in annotations_public.jsonl has been reshuffled. If you downloaded this dataset before 2026-07-13, please re-download it before your next evaluation run — the letters A–J now map to different option strings. Question text and the set of option strings per question are unchanged, so previously published aggregate scores remain valid; but any locally-stored letter-only predictions must be remapped (or the model re-run) against the new mapping before submitting to the leaderboard.

A Memory-driven Reasoning Benchmark for Long-Horizon Egocentric Video Understanding.

500 multiple-choice questions over week-long egocentric video (built on EgoLife) that evaluate three complementary kinds of memory:

• Entity memory — track how object states evolve across days
• Event memory — recall and order activities separated by hours or days
• Behavior memory — abstract recurring patterns from sparse, repeated observations

Average 5.1 evidence segments per question and 25.9 hours of memory backtracking — 2× both metrics over the strongest prior week-long benchmark.

Links

• 🧠 Leaderboard (HF Space): https://huggingface.co/spaces/Ted412/EgoMemReason
• 💻 Code & reference eval scripts: https://github.com/Ziyang412/EgoMemReason
• 🌐 Project page: https://egomemreason.github.io/
• 🎬 EgoLife video frames (separate license): https://egolife-ai.github.io/
• 📄 Paper: https://arxiv.org/abs/2605.09874

Composition

Schema

This dataset releases the public version — questions and options only, no answer keys (the held-out answer key lives in a private dataset, and submissions are scored against it by the leaderboard Space).

{
  "example_id": 1,
  "p_id": "A1_JAKE_DAY7_19_00_00_q001",
  "identity": "A1_JAKE",
  "query_time": "DAY7, 19:00:00",
  "question": "What do I most often eat for breakfast?",
  "options": {
    "A": "Pancake",
    "B": "Rice",
    "C": "Burger",
    "D": "Dumplings"
  },
  "query_type": "Activity Pattern"
}

Note that questions have 4-10 options (letters A-J). The valid answer set for any given question is the keys of its options dict; Event Ordering questions tend to have the most options.

How to evaluate

1. Get this dataset:

   from datasets import load_dataset
   ds = load_dataset("Ted412/EgoMemReason")["test"]
   

2. Get the underlying EgoLife video frames (separate license, see https://egolife-ai.github.io/) — we don't redistribute video here.
3. For each item, sample frames from (identity, query_time) backwards in time and run your model to pick one letter from options.keys().
4. Format the predictions as a JSON list:

   [
     {"example_id": 1, "predicted_answer": "A"},
     ...
   ]
   

5. Score your model. Anyone can score anonymously on the Leaderboard Space → Score tab: upload submission.json and get per-split + Overall accuracy against the held-out answer key. Nothing is stored and nothing is posted to the public leaderboard.
6. Ask to be listed. To have your model appear on the public leaderboard, email your submission.json to ziyangw@cs.unc.edu along with your method name, team name, model size, video-input modality, a one-sentence method description, and any project / paper links. The maintainer re-scores against the private key, sanity-checks the metadata, and adds a row within ~5 business days. Every leaderboard entry is personally verified — there is no self-serve upload.

The reference inference scripts for 12 MLLMs and 5 agentic frameworks (Gemini, GPT-5, Qwen3-VL, InternVL3.5, Molmo2, VideoLLaMA3, InternVideo2.5, LongVA, AVP, Ego-R1, SiLVR, WorldMM, …) live in the GitHub repo.

License

• EgoMemReason annotations (this dataset): CC BY-NC 4.0 — academic research and benchmarking are permitted; commercial use requires written permission.
• EgoLife video frames (not redistributed here): governed by the EgoLife data license — you must accept their terms separately.

Citation

@misc{wang2026egomemreasonmemorydrivenreasoningbenchmark,
      title={EgoMemReason: A Memory-Driven Reasoning Benchmark for Long-Horizon Egocentric Video Understanding},
      author={Ziyang Wang and Yue Zhang and Shoubin Yu and Ce Zhang and Zengqi Zhao and Jaehong Yoon and Hyunji Lee and Gedas Bertasius and Mohit Bansal},
      year={2026},
      eprint={2605.09874},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2605.09874},
}