EgoServe: A Benchmark for Proactive Service in Egocentric Videos

Project Page | Paper | Code

EgoServe provides cleaned proactive-service annotations on top of three egocentric video sources — HoloAssist, CaptainCook4D, and EgoLife. Each annotation marks when an intelligent assistant should proactively intervene, which type of service it should offer, and what it should say (a short assistant↔user dialogue).

This repository hosts the annotations only. The underlying videos must be obtained from the original datasets (links below). Only the videos/recordings actually used in our evaluation are included.

Proactive service taxonomy

HoloAssist and CaptainCook4D cover Instant + Short-Term services (derived from the datasets' human error annotations). EgoLife additionally covers Episodic + Long-Term. personal_progressive is excluded from the released benchmark.

Contents

HoloAssist/holoassist_service_annotations.json        # 191 videos, 1,717 events
CaptainCook4D/captaincook4d_service_annotations.json  # 87 recordings, 828 events
EgoLife/
  A1_JAKE/   { instant, short_term, long_term, episodic }.json   # 364 events
  A4_LUCIA/  { instant, short_term, long_term, episodic }.json   # 265 events
  A5_KATRINA/{ instant, short_term, long_term, episodic }.json   # 263 events
dataset_statistics.json                               # per-dataset counts + durations

Statistics (evaluation scope)

Per-sub-type counts (Total / EgoLife / HoloAssist / CaptainCook4D):

HoloAssist

{video_name: {instant:{safety, tool_use}, short_term:{next_step_guidance, error_recovery, resource_reminder}}}. Each event has a time_window, an observation, and a paired dialogue. Built from HoloAssist Wrong Action annotations, then cleaned so that every error event maps to a real Wrong Action, no time window belongs to two error types, and time-adjacent segments describing the same mistake are merged (judged by an LLM). 191 of the 193 annotated videos carry at least one valid event.

CaptainCook4D

A list of recordings; each dialogue.items[] entry is one service event with step_id, error_tag, service_type {main, sub}, and a dialogue. Multiple errors on the same step are merged into one event; time-less Missing Step errors are removed; service types were re-checked for consistency. 87 recordings used in evaluation.

EgoLife (A1 / A4 / A5, DAY1–DAY5)

Per person, four files grouped by main type (instant, short_term, long_term, episodic); each file is a list of events with the sub-type stored in sub_type. Events whose trigger is purely verbal (supporting_source == "speakers_say"), or which lack a valid time window or assistant utterance, are excluded. correct-revised annotations are used when available.

Field notes

• time_window / current_time_window — when the service should trigger (HH:MM:SS.mmm-..., or DAY{n} HH:MM:SS-... for EgoLife).
• dialogue / proactive_dialogue — a short assistant↔user exchange; the assistant turn is the proactive message.
• observation / trigger_reason — the grounded evidence for the trigger.

Source datasets (videos)

• HoloAssist: https://holoassist.github.io/
• CaptainCook4D: https://captaincook4d.github.io/
• EgoLife: https://egolife-ai.github.io/

Citation

Please cite the EgoServe paper (see project page) and the three source datasets.

@inproceedings{gong2026vinci2,
  title     = {Vinci2: Providing Proactive Assistance in Continuous Egocentric Videos},
  author    = {Gong, Sitong and Yan, Tianyu and Kang, Caixin and Zheng, Bo and
               Ruan, Xiang and Lu, Huchuan and Zhang, Kaipeng and Sato, Yoichi and Huang, Yifei},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026}
}