EgoMemReason — Leaderboard Space

Live leaderboard + anonymous scoring endpoint for the EgoMemReason benchmark: 500 multiple-choice questions over week-long egocentric video, evaluating entity / event / behavior memory. Accepted at COLM 2026.

• 🌐 Project page: https://egomemreason.github.io/
• 📄 Paper: https://arxiv.org/abs/2605.09874
• 💻 Reference eval scripts: https://github.com/Ziyang412/EgoMemReason
• 📦 Public questions: https://huggingface.co/datasets/Ted412/EgoMemReason
• 🎬 Source frames: https://egolife-ai.github.io/

What the Space does

Three tabs:

• About — paper description + submission instructions + citation.
• Leaderboard — read-only view of Ted412/EgoMemReason-Leaderboard. Every row is a submission the maintainer has personally verified.
• Score — anonymous scoring endpoint. Users upload a submission JSON; the Space validates it against the held-out answer key (pulled once at boot from Ted412/EgoMemReason-Private) and returns per-split + Overall accuracy. Nothing is persisted — the score is shown to the submitter only, and it is not written to any dataset. To be listed on the public leaderboard, the submitter must email ziyangw@cs.unc.edu.

Operator notes

Required Space secret

The Space token needs no write scope on any repo — since scoring results are never persisted, the Space never uploads anything. Only the maintainer's personal creds (used off-Space by scripts/publish_submission.py) touch the leaderboard dataset.

Create at https://huggingface.co/settings/tokens → fine-grained → grant only Read: Ted412/EgoMemReason-Private. Leave every user-level checkbox off.

Local development

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# Either export HF_TOKEN with read scope on -Private, or copy the file locally:
cp /path/to/annotations_private.json .

python app.py
# → http://127.0.0.1:7860

Publishing a new leaderboard entry

Run on the maintainer's local machine (not on the Space), after hf auth login:

python scripts/publish_submission.py \
    --submission path/to/user_submission.json \
    --private   /path/to/annotations_private.json \
    --team-name "…" --method-name "…" \
    --model-size "…" --uses-external no --uses-frames frames-only \
    --method-description "…" \
    --project-url "…" --publication-url "…"

The script validates the JSON against the held-out answer key, computes per-split + overall metrics, and pushes a submissions/<uuid>.json record to Ted412/EgoMemReason-Leaderboard. The live Space picks it up within ~60 s (cache TTL).

Architecture

EgoMemReason-Space (this Space, public)
├── app.py            Gradio UI (About / Leaderboard / Score)
├── evaluator.py      pure scoring — used by both Score tab and the
│                     local publish script
├── ledger.py         HF I/O: pulls private annotations at boot; reads
│                     public leaderboard dataset for the viewer
└── scripts/
    ├── publish_submission.py  maintainer-only CLI to add a new row after
    │                          verifying an emailed submission
    └── reshuffle_options.py   one-shot script to permute option-letter
                               mappings when the answer key needs rotating
    (seed_paper_baselines.py is retained locally, gitignored — see seeds/.)

Ted412/EgoMemReason-Private (HF dataset, private)
└── annotations_private.json    500 Qs WITH correct_answer, pulled at
                                Space boot with Read scope

Ted412/EgoMemReason-Leaderboard (HF dataset, public)
└── submissions/
    └── <uuid>.json             one immutable record per verified entry