This repository accompanies the survey Video Understanding in the Era of Multimodal Large Language Models by Yiming Zhong, Chang Nie, Yan Yang, Xiaoyu Liu, and Caifeng Shan. It favors representative, reproducible work over an exhaustive paper dump.

✨ Highlights

• Architecture-first taxonomy — navigate by frame encoding, multimodal alignment, and language backbone.
• Task-aware resources — datasets and benchmarks are separated by the capability they measure.
• Curated, not collected — every entry has a one-line reason for inclusion.
• Community-maintained — consistent entries, automated link checks, and a lightweight PR template.

🧭 Start here

Legend and inclusion policy

• 📄 paper · 💻 code · 🤗 model/data · 🌐 project page
• Entries must be directly relevant to video understanding with foundation models or Video-LLMs.
• Preference is given to released papers with code, data, checkpoints, or lasting conceptual value.
• Ordering within a section is approximately chronological; this is not a leaderboard.

📚 Contents

• Survey
• Latest papers
• Milestones
• Video-LLMs
  • Basic encoding
  • Long-context encoding
  • Streaming encoding
  • Multimodal alignment
• Training resources
• Evaluation
• Challenges and frontiers
• Contributing

📝 Survey

Video Understanding in the Era of Multimodal Large Language Models

Yiming Zhong · Chang Nie · Yan Yang · Xiaoyu Liu · Caifeng Shan

The survey organizes modern Video-LLMs through three architectural decisions—frame encoding, multimodal alignment, and LLM selection—then connects them to training strategies, datasets, evaluation protocols, and open challenges.

📌 The paper link and BibTeX will be added here when the publisher activates the public article page and DOI.

🔥 Latest papers

Sorted by first public release, newest first. Star badges are live and update automatically from GitHub. A dash means that no official repository was available when the entry was added.

2026

2025

2024

2023

Foundations: 2019–2022

🕰️ Milestones

🎬 Video-LLMs

Basic encoding

Encode sampled frames independently with an image encoder, or model motion directly with a video-native encoder.

• CLIP — The standard image-text aligned visual backbone behind many early Video-LLMs. 📄 💻
• SigLIP — Replaces softmax contrastive learning with a sigmoid loss and scales efficiently. 📄 💻
• InternVideo — A video-native encoder that explicitly models temporal dynamics. 📄 💻
• InternVideo2 — Scales video foundation learning through masked token reconstruction and cross-modal contrastive objectives. 📄 💻

Long-context encoding

Streaming encoding

• Flash-VStream — Maintains a compact spatiotemporal memory for real-time video dialogue. 📄 💻
• StreamChat — Updates visual context incrementally with a streaming cache. 📄
• StreamingVLM — Preserves recent visual evidence while evicting redundant historical tokens. 📄
• VideoChat-Online — Uses hierarchical memories and dynamic frame eviction for online dialogue. 📄

Multimodal alignment

Selected resources:

• BLIP-2 — Introduces Q-Former, the canonical learnable-query alignment module. 📄 💻
• TimeChat — Uses a sliding video Q-Former for timestamp-aware long-video understanding. 📄 💻
• Qwen2-VL — Provides dynamic-resolution visual tokenization and multimodal rotary position encoding. 📄 💻

🧪 Training resources

Pre-training datasets

Instruction and preference data

• VideoInstruct-100K — Video instruction data introduced with VideoChatGPT. 📄 💻
• ShareGPT4Video — Detailed video captions and instruction data generated with an efficient captioner. 📄 💻
• LLaVA-Video-178K — Synthetic instruction data covering open-ended video understanding tasks. 📄 🤗
• VideoDPO — Preference optimization data for reducing video-language hallucination and improving alignment. 📄

Training paradigms

• Pre-training — Learn general visual and temporal representations from large-scale paired data.
• Instruction tuning — Convert perception into conversational, question-answering, captioning, and grounding capabilities.
• Preference alignment — Use DPO, RLAIF, PPO, or GRPO to improve helpfulness, reasoning, and faithfulness.
• Training-free systems — Compose frozen encoders, captioners, retrievers, memories, and LLM agents at inference time.

📊 Evaluation

General and long-video understanding

Temporal, streaming, and fine-grained evaluation

• TempCompass — Diagnoses temporal perception through counterfactual and caption-based tasks. 📄 💻
• StreamingBench — Measures perception, memory, and reasoning over streaming video. 📄 💻
• OVO-Bench — Evaluates online video understanding under temporal constraints. 📄
• MotionBench — Tests whether models truly perceive motion rather than rely on static cues. 📄

🔭 Challenges and frontiers

Video-LLMs
├── Perception       → fine detail, motion, audio, OCR, spatial grounding
├── Temporal memory  → long context, streaming, event boundaries
├── Reasoning        → causality, compositionality, multi-step inference
├── Reliability      → hallucination, calibration, evidence attribution
├── Efficiency       → token compression, adaptive compute, edge inference
└── Responsibility   → privacy, bias, safety, provenance

• Faithful generation — Ground every claim in visible or audible evidence and evaluate counterfactual robustness.
• Adaptive computation — Spend dense computation only on query-relevant moments without discarding brief events.
• True audio-visual reasoning — Align sounds to the frames and sources that produce them.
• Interactive streaming — Decide not only what to answer, but when to respond during an ongoing stream.
• Better data — Combine scalable model-generated annotations with targeted human verification.

🤝 Contributing

Contributions are welcome. Please read CONTRIBUTING.md before opening a pull request. A good addition includes a stable paper link, an official code or project link when available, and one sentence explaining why the work belongs in its category.

🙏 Acknowledgements

The presentation follows the curation principles of the Awesome manifesto and draws organizational inspiration from established multimodal research lists such as Awesome Multimodal Large Language Models.

📜 License

To the extent possible under law, the maintainers have waived copyright and related rights to this curated list under CC0 1.0. Individual papers, codebases, models, and datasets retain their original licenses.

Back to top ↑