EgoVITA: Learning to Plan and Verify for Egocentric Video Reasoning

ECCV 2026

Abstract

Egocentric video understanding requires procedural reasoning under partial observability and continuously shifting viewpoints. Current multimodal large language models (MLLMs) struggle with this setting, often generating plausible but visually inconsistent or weakly grounded responses. We introduce EgoVITA, a framework that decomposes egocentric video reasoning into a structured plan-then-verify process. The model first generates an egocentric plan: a causal sequence of anticipated actions from a first-person perspective. This plan is then evaluated by an exocentric verification stage that uses third-person reasoning over the same video to verify its spatiotemporal and logical consistency, without exocentric video input. This decomposition enables cross-perspective feedback without requiring paired ego-exo supervision. To train this reasoning process, we adopt Group Relative Policy Optimization (GRPO) with two dense reward signals: one that grounds anticipated actions in subsequent visual observations and another that reinforces consistent third-person verification. EgoVITA achieves state-of-the-art performance on egocentric reasoning benchmarks, outperforming Qwen2.5-VL-7B by +7.7 on EgoBlind and +4.4 on EgoOrient, while maintaining strong generalization on exocentric video tasks with only 52k training samples.

TODO

• Release paper.
• [] Release model checkpoints.
• Release inference code.
• Release eval scripts.
• Release training data.
• Release GRPO trainer (plan-then-verify with ACMG + confidence rewards).

Install

conda create -n egovita python=3.13
conda activate egovita

pip install torch==2.11.0 torchvision torchaudio
pip install flash-attn==2.8.3 --no-build-isolation
pip install transformers==5.12.1

pip install qwen-vl-utils torchcodec decord timm
pip install opencv-python pillow numpy tqdm filelock

MS-Swift Setup

cd ms-swift
pip install -e .

The ms-swift/ fork carries the EgoVITA patches (vision-merge guards, transformers-5 fps fix, kwarg/dtype handling) and the training entrypoints under ms-swift/egovita/.

Method

EgoVITA reasons in two turns:

1. Egocentric plan (<ego_plan>) — a first-person causal sequence of anticipated actions.
2. Exocentric verification (<exo_verify>) — third-person reasoning over the same video to check spatiotemporal/logical consistency, then the final <answer>.

Trained with GRPO and two dense rewards: ACMG (grounds anticipated actions in future frames) and a confidence reward (reinforces consistent third-person verification), plus format and answer rewards.

Inference

Minimal two-turn inference on a single video:

python single_video_infer.py    # edit MODEL / VIDEO / QUESTION at the top

Training

Data setup and download: see ms-swift/egovita/README.md and the dataset card. Scripts (per model family — Qwen3-VL, Qwen2.5-VL, Qwen2-VL, InternVL3.5):

# SFT (two-turn format)
bash ms-swift/egovita/sft_qwen3_vl.sh

# GRPO (plan-then-verify) — start the rollout server first (STEP 1 in the script), then the trainer
bash ms-swift/egovita/rl_qwen3_vl.sh

Key EgoVITA-specific env knobs (set in the RL scripts): EGOVITA_LAMBDA_ACMG, EGOVITA_GAMMA, EGOVITA_CONF_WARMUP, EGOVITA_ACMG_N, EGOVITA_TEACHER_CACHE, EGOVITA_AUX_K. See ms-swift/egovita/README.md for the full description and per-family notes.

Evaluation

Egocentric benchmark eval scripts in eval/ (paths hardcoded per script for now): egoplan.py, egothink.py, egoblind.py, ego_orient.py, eoc_bench.py. Each runs the EgoVITA two-turn flow and scores with the benchmark's native metric.

Citation

@inproceedings{kulkarni2026egovita,
  title={EgoVITA: Learning to Plan and Verify for Egocentric Video Reasoning},
  author={Kulkarni, Yogesh and Fazli, Pooyan},
  booktitle={European Conference on Computer Vision (ECCV)},
  year={2026}
}

Contact

For questions, contact Yogesh Kulkarni at ykulka10@asu.edu, or open an issue in this repository.