Watch, Remember, Reason: Human-View Video Understanding with MLLMs

Jiahao Meng · Yue Tan · Qi Xu · Kuan Gao · Weisong Liu · Yanwei Li · Jason Li
Lingdong Kong · Haochen Wang · Qianyu Zhou · Jiangning Zhang · Guangliang Cheng
Yunhai Tong · Lu Qi · Ming-Hsuan Yang

Overview

This repository accompanies our survey, which takes a human-view perspective on LLM/MLLM-based video understanding by decomposing it into three core cognitive abilities and reviewing how recent methods realize each:

• Watch — perceptual grounding of multimodal video streams: fine-grained spatio-temporal grounding, comprehensive captioning, audio-visual (omni-modal) perception, and efficient long-video processing.
• Remember — maintaining context over long or streaming inputs: offline memory (agentic vs. non-agent) and streaming memory.
• Reason — deriving grounded answers from perceived and retained evidence: text-only reasoning and the emerging thinking-with-videos paradigm, each split into agentic and non-agent approaches.

Beyond methods, the survey covers domain-specific subfields (egocentric, sports, instructional, medical, narrative videos), training datasets and evaluation benchmarks across major task types and capability dimensions, and open problems on the path to scalable, memory-aware, evidence-grounded video intelligence.

Table of Contents

• 1. Watch — How to Watch?
  • 1.1 Fine-grained Watching
  • 1.2 Comprehensive Watching
  • 1.3 Audio-Visual Watching
  • 1.4 Efficient Watching
• 2. Remember — How to Remember?
  • 2.1 Offline Memory — Agentic
  • 2.2 Offline Memory — Non-Agent
  • 2.3 Streaming Memory
• 3. Reason — How to Reason?
  • 3.1 Text-only Reasoning — Agentic
  • 3.2 Text-only Reasoning — Non-Agent
  • 3.3 Thinking with Videos — Agentic
  • 3.4 Thinking with Videos — Non-Agent
• 4. Subfields
  • 4.1 Egocentric Videos
  • 4.2 Sports Videos
  • 4.3 Instructional Videos
  • 4.4 Medical Videos
  • 4.5 Movie and Narrative Videos
• 5. Datasets and Benchmarks
  • 5.1 Training Datasets
  • 5.2 Evaluation Benchmarks
• 6. Future Directions
• Citation
• Contributing

---

1. Watch — How to Watch?

Watching corresponds to the perceptual stage where models transform raw multimodal inputs into structured representations. We organize methods along four complementary dimensions.

1.1 Fine-grained Watching

Precise spatio-temporal grounding: time representation, long-video efficiency, structured decoding, fine-grained perception architectures, and verifiable post-training. Includes both flagship methods (Table 2) and the broader family cited in §3.1.1 (temporal grounding, spatio-temporal grounding, video referring).

Temporal grounding & VTG-style methods

Spatio-temporal grounding & video referring

1.2 Comprehensive Watching

Whole-video, dense, and region-level captioning over visual-token sequences. Includes flagship methods (Table 2) and the broader family cited in §3.1.2.

Whole-video captioning

Dense video captioning

Region-level captioning & video referring

1.3 Audio-Visual Watching

Omni-modal perception unifying speech, environmental audio, and music with visual streams under a shared LLM backbone.

1.4 Efficient Watching

Reducing redundancy in long videos: frame-level selection, token-level compression / merging, and model-level efficient processing.

---

2. Remember — How to Remember?

Memory connects perception with higher-level understanding by retaining salient information over time. We split methods by storage paradigm.

2.1 Offline Memory — Agentic

LLMs/VLMs autonomously invoke memory tools through multi-round reasoning to construct and retrieve memory.

2.2 Offline Memory — Non-Agent

Deterministic pipelines where memory construction and retrieval are sequential, fixed stages.

2.3 Streaming Memory

Processing unbounded streams within fixed memory budgets via causal compression, KV pruning, sink reuse, or hierarchical caches.

---

3. Reason — How to Reason?

Reasoning operates on perceived and retained evidence. We separate models by what they reason over (text-only vs. interleaved with visual evidence) and how the reasoning is controlled (agentic tool loops vs. non-agent post-training).

3.1 Text-only Reasoning — Agentic

Agent loops with tools / memory / planning that orchestrate perception, retrieval, verification, and reflection.

3.2 Text-only Reasoning — Non-Agent

CoT-style supervised fine-tuning and RL/DPO post-training without tool-driven control loops.

3.3 Thinking with Videos — Agentic

Models that interleave reasoning with explicit visual grounding and re-inspection (o3-style "thinking with images" extended to video).

3.4 Thinking with Videos — Non-Agent

Models that natively emit grounded reasoning traces (timestamps, boxes, captions) without external tool calls.

---

4. Subfields

Domain-specific scenarios that stress different combinations of perception, memory, and reasoning.

4.1 Egocentric Videos

4.2 Sports Videos

4.3 Instructional Videos

4.4 Medical Videos

4.5 Movie and Narrative Videos

---

5. Datasets and Benchmarks

5.1 Training Datasets

5.1.1 Video QA

Covers both training corpora and widely used QA benchmarks (many pre-MLLM datasets are still used for evaluation).

5.1.2 Video Captioning

Includes classical clip / dense / narration-level corpora, MLLM-era recaptioned corpora, and grounded / omni-modal captioning data.

5.1.3 Video Temporal Grounding

Includes large-scale narrated-video pretraining corpora, MLLM-era instruction tuning corpora, and reasoning-/RL-oriented re-annotated data.

5.1.4 Long Video Memory

5.2 Evaluation Benchmarks

Categorized into six capability dimensions.

5.2.1 General Video Understanding

5.2.2 Temporal & Spatial Understanding

5.2.3 Complex Reasoning

5.2.4 Long-Context & Streaming Understanding

5.2.5 Domain-Specific Knowledge

5.2.6 Omnimodal Collaboration

---

6. Future Directions

Open problems highlighted by the survey:

• Spatial reasoning in video understanding — bridging object-level and scene-level spatial understanding for embodied / 3D-aware applications.
• Multi-video and multi-segment temporal grounding — set-based retrieval + refinement under verifiable rewards.
• Hour-scale video understanding with structured memory — multi-level memory with evidence pointers and learned write/forget policies.
• Efficient and verifiable video reasoning — budgeted evidence search jointly optimizing answer correctness, evidence alignment, and compactness.
• Streaming egocentric video understanding — stateful, goal-driven streaming memory with proactive timing for embodied agents.

---

Citation

If you find this survey useful, please consider citing:

@article{meng2026watch,
  title   = {Watch, Remember, Reason: Human-View Video Understanding with MLLMs},
  author  = {Meng, Jiahao and Tan, Yue and Xu, Qi and Gao, Kuan and Liu, Weisong and Li, Yanwei and Li, Jason and Kong, Lingdong and Wang, Haochen and Zhou, Qianyu and Zhang, Jiangning and Cheng, Guangliang and Tong, Yunhai and Qi, Lu and Yang, Minghsuan},
  journal = {arXiv preprint arXiv:2606.07433},
  year    = {2026}
}

Contributing

Contributions are welcome — please open a PR to add a missing paper, dataset, or benchmark. When adding an entry, place it in the most specific subsection that fits the taxonomy and follow the existing column order. Papers are listed roughly in chronological order within each table.