Video-Language Understanding: A Survey from Model Architecture, Model Training, and Data Perspectives

This repository accompanies our ACL 2024 (Findings) survey paper, Video-Language Understanding: A Survey from Model Architecture, Model Training, and Data Perspectives. It curates leaderboards, datasets, and a structured paper list spanning the evolution of video-language understanding — from early RNN/CNN pipelines through transformer-based pre-training and into the current generation of large video–language models.

The list is actively maintained. Spotted an error or a missing paper? Please open an issue or submit a pull request — see Contributing below.

If our survey is useful for your research, please cite us:

@article{nguyen2024video,
  title   = {Video-Language Understanding: A Survey from Model Architecture, Model Training, and Data Perspectives},
  author  = {Nguyen, Thong and Bin, Yi and Xiao, Junbin and Qu, Leigang and Li, Yicong and
             Wu, Jay Zhangjie and Nguyen, Cong-Duy and Ng, See-Kiong and Tuan, Luu Anh},
  journal = {arXiv preprint arXiv:2406.05615},
  year    = {2024}
}

---

📑 Table of Contents

• What's New (2025–2026 Highlights)
• Leaderboards
  • Text-video retrieval
  • Video captioning
  • Video question answering
• Datasets and Benchmarks
• Paper List
  • Surveys
  • Model architecture perspective
    • Pre-transformer
    • Shared Transformer
    • Stack Transformer
    • Dual Transformer
    • LLM-augmented
    • Agentic and streaming Video-LLMs
  • Model training perspective
    • Pre-training
    • Fine-tuning
    • Reinforcement learning and video reasoning
  • Data perspective
    • Manual collection
    • Data augmentation
    • Manual annotation
    • Automatic generation
• Contributing

---

✨ What's New (2025–2026 Highlights)

The video-language landscape has shifted decisively toward large video–language models, long-form understanding, and reasoning. Some of the most notable threads from 2025–2026:

• Frontier Video-LLMs. Proprietary frontier models (Gemini 2.5 Pro / Gemini 3, GPT-4o, Claude 3.5 Sonnet) now natively ingest hour- to multi-hour-long videos at million-token context lengths, while open-source Qwen2.5-VL, InternVL3 / InternVL3.5, VideoLLaMA 3, LLaVA-Video, NVILA, Apollo, and Tarsier2 close much of the gap and are increasingly used as research baselines.
• Long-form, hour-scale evaluation. Benchmarks have moved past short clips to test models on hours of footage with referring QA, knowledge acquisition, and clue grounding (e.g., Video-MME, MLVU, LongVideoBench, CG-Bench, Video-MMMU, TemporalBench).
• Reasoning and reinforcement learning. Following DeepSeek-R1, a new wave of work — Video-R1, VideoChat-R1, Video-Thinker, VideoP2R — applies GRPO and rule-based rewards to imbue Video-LLMs with explicit chain-of-thought and temporal reasoning. New benchmarks (Video-Holmes, VCR-Bench, CG-Bench-Reasoning, VR-Bench) specifically target multi-step video reasoning.
• Streaming and agentic video understanding. VideoLLM-online, Flash-VStream, Dispider, and StreamChat move from offline clip-level reasoning to real-time, online video stream interaction, while VideoAgent (Stanford and PKU variants) explores tool-using LLM agents for hour-long content.

See the LLM-augmented, Agentic and streaming Video-LLMs, Reinforcement learning and video reasoning, and Modern Video-LLM evaluation benchmarks sections below for details and references.

---

🏆 Leaderboards

Text-video retrieval

Results reported on the MSRVTT 1k-A test split (zero-shot or fine-tuned, following each paper's original protocol). Higher is better for all metrics.

Video captioning

Results reported on MSRVTT test split.

Video question answering

Accuracy (%) on the MSRVTT-QA and MSVD-QA test splits.

💡 The latest LLM-augmented systems (Qwen2.5-VL, LLaVA-Video, VideoLLaMA 2/3, Tarsier2) are increasingly evaluated on long-video benchmarks such as Video-MME, MLVU, LongVideoBench, and MVBench rather than MSRVTT-QA / MSVD-QA. See the table below and Datasets and Benchmarks for the modern evaluation suites.

Long-video understanding (selected open Video-LLMs)

Accuracy (%) on the modern long-form video-LMM benchmarks. We report representative numbers from each paper; rigorous comparison should consult original protocols (frame count, resolution, prompt). "—" indicates the model did not officially report on that benchmark.

---

📚 Datasets and Benchmarks

Task abbreviations: TVR = text-video retrieval · VC = video captioning · VideoQA = video question answering · VMR = video moment retrieval · AL = action localization · AR = action recognition · AS = action segmentation · VG = video grounding · PT = pre-training.

Classic video-language datasets

Question answering and reasoning

Localization, grounding, and highlights

Action recognition

Large-scale pre-training corpora

Modern Video-LLM evaluation benchmarks

The shift toward general-purpose video-language models has produced a new wave of benchmarks emphasizing long-form, temporally rich, knowledge-grounded, and reasoning-oriented evaluation.

---

📖 Paper List

Surveys

1. Transformer-based video-language pre-training arXiv 2021 [Paper]
2. Self-supervised learning for videos: A survey ACM Computing Surveys 2022 [Paper] [Code]
3. Video question answering: Datasets, algorithms and challenges EMNLP 2022 [Paper] [Code]
4. Deep learning for video-text retrieval: a review IJMIR 2023 [Paper]
5. A review of deep learning for video captioning arXiv 2023 [Paper]
6. Video question answering: a survey of models and datasets Mobile Networks and Applications 2021 [Paper]
7. Video understanding with large language models: A survey TPAMI 2025 [Paper] [Code]
8. A survey on multimodal large language models arXiv 2024 [Paper] [Code]
9. A survey on visual understanding multimodal large language models OpenReview 2025 [Paper]
10. VideoLLM benchmarks and evaluation: A survey arXiv 2025 [Paper]
11. Vision language models: A survey of 26K papers (CVPR, ICLR, NeurIPS 2023–2025) arXiv 2025 [Paper]
12. A survey of multimodal models on language and vision: A unified modeling perspective DMML 2025 [Paper]

Model architecture perspective

Pre-transformer

1. Unifying visual-semantic embeddings with multimodal neural language models NeurIPS 2014 [Paper]
2. Describing videos by exploiting temporal structure ICCV 2015 [Paper] [Code]
3. Video paragraph captioning using hierarchical recurrent neural networks CVPR 2016 [Paper]
4. Convolutional two-stream network fusion for video action recognition CVPR 2016 [Paper] [Code]
5. Learning language-visual embedding for movie understanding with natural language arXiv 2016 [Paper]
6. Temporal tessellation for video annotation and summarization arXiv 2016 [Paper] [Code]
7. Video captioning with multi-faceted attention arXiv 2016 [Paper]
8. End-to-end concept word detection for video captioning, retrieval, and question answering CVPR 2017 [Paper]
9. Video question answering via attribute-augmented attention network learning SIGIR 2017 [Paper]
10. Hierarchical boundary-aware neural encoder for video captioning CVPR 2017 [Paper]
11. Localizing moments in video with natural language ICCV 2017 [Paper]
12. TALL: Temporal activity localization via language query ICCV 2017 [Paper] [Code]
13. Leveraging video descriptions to learn video question answering AAAI 2017 [Paper]
14. Tensor-train recurrent neural networks for video classification arXiv 2017 [Paper] [Code]
15. Two-stream RNN/CNN for action recognition in 3D videos IROS 2017 [Paper]
16. ConvNet architecture search for spatiotemporal feature learning arXiv 2017 [Paper]
17. A joint sequence fusion model for video question answering and retrieval ECCV 2018 [Paper]

Shared Transformer

1. Less is more: ClipBERT for video-and-language learning via sparse sampling CVPR 2021 [Paper] [Code]
2. VATT: Transformers for multimodal self-supervised learning from raw video, audio and text NeurIPS 2021 [Paper] [Code]
3. VIOLET: End-to-end video-language transformers with masked visual-token modeling arXiv 2021 [Paper] [Code]
4. LAVENDER: Unifying video-language understanding as masked language modeling arXiv 2022 [Paper] [Code]
5. All in one: Exploring unified video-language pre-training CVPR 2023 [Paper] [Code]
6. An empirical study of end-to-end video-language transformers with masked visual modeling CVPR 2023 [Paper] [Code]
7. VindLU: A recipe for effective video-and-language pre-training CVPR 2023 [Paper] [Code]

Stack Transformer

1. HERO: Hierarchical encoder for video+language omni-representation pre-training EMNLP 2020 [Paper] [Code]
2. UniVL: A unified video and language pre-training model for multimodal understanding and generation arXiv 2020 [Paper] [Code]
3. End-to-end generative pre-training for multimodal video captioning (MV-GPT) arXiv 2022 [Paper]
4. CLIP meets video captioning: Concept-aware representation learning does matter PRCV 2022 [Paper] [Code]
5. mPLUG-2: A modularized multimodal foundation model across text, image and video ICML 2023 [Paper] [Code]
6. VLAB: Enhancing video-language pre-training by feature adapting and blending arXiv 2023 [Paper]

Dual Transformer

1. Frozen in time: A joint video and image encoder for end-to-end retrieval ICCV 2021 [Paper] [Code]
2. CLIP4Clip: An empirical study of CLIP for end-to-end video clip retrieval and captioning arXiv 2021 [Paper] [Code]
3. VideoCLIP: Contrastive pre-training for zero-shot video-text understanding EMNLP 2021 [Paper] [Code]
4. ECLIPSE: Efficient long-range video retrieval using sight and sound ECCV 2022 [Paper] [Code]
5. A CLIP-Hitchhiker's guide to long video retrieval arXiv 2022 [Paper] [Code]
6. Video graph transformer for video question answering (VGT) ECCV 2022 [Paper] [Code]
7. CLIP-ViP: Adapting pre-trained image-text model to video-language representation alignment ICLR 2023 [Paper] [Code]
8. Contrastive video question answering via video graph transformer (CoVGT) TPAMI 2023 [Paper] [Code]
9. InternVideo2: Scaling foundation models for multimodal video understanding ECCV 2024 [Paper] [Code]

LLM-augmented

1. VideoChat: Chat-centric video understanding arXiv 2023 [Paper] [Code]
2. VideoLLM: Modeling video sequence with large language models arXiv 2023 [Paper] [Code]
3. Video-ChatGPT: Towards detailed video understanding via large vision and language models ACL 2024 [Paper] [Code]
4. Video-LLaMA: An instruction-tuned audio-visual language model for video understanding EMNLP 2023 [Paper] [Code]
5. Retrieving-to-answer: Zero-shot video question answering with frozen large language models arXiv 2023 [Paper]
6. MVBench: A comprehensive multi-modal video understanding benchmark (VideoChat2) CVPR 2024 [Paper] [Code]
7. Video-LLaVA: Learning united visual representation by alignment before projection EMNLP 2024 [Paper] [Code]
8. LLaMA-VID: An image is worth 2 tokens in large language models ECCV 2024 [Paper] [Code]
9. MovieChat: From dense token to sparse memory for long video understanding CVPR 2024 [Paper] [Code]
10. LongVA: Long context transfer from language to vision arXiv 2024 [Paper] [Code]
11. VideoLLaMA 2: Advancing spatial-temporal modeling and audio understanding in Video-LLMs arXiv 2024 [Paper] [Code]
12. Tarsier: Recipes for training and evaluating large video description models arXiv 2024 [Paper] [Code]
13. LLaVA-Video: Video instruction tuning with synthetic data arXiv 2024 [Paper] [Code]
14. LongVU: Spatiotemporal adaptive compression for long video-language understanding arXiv 2024 [Paper] [Code]
15. Qwen2-VL: Enhancing vision-language model's perception of the world at any resolution arXiv 2024 [Paper] [Code]
16. NVILA: Efficient frontier visual-language models arXiv 2024 [Paper] [Code]
17. Aria: An open multimodal native mixture-of-experts model arXiv 2024 [Paper] [Code]
18. Apollo: An exploration of video understanding in large multimodal models CVPR 2025 [Paper] [Code]
19. VideoLLaMA 3: Frontier multimodal foundation models for image and video understanding arXiv 2025 [Paper] [Code]
20. Tarsier2: Advancing large vision-language models from detailed video description to comprehensive video understanding arXiv 2025 [Paper] [Code]
21. Qwen2.5-VL Technical Report arXiv 2025 [Paper] [Code]
22. BOLT: Boost large vision-language model without training for long-form video understanding CVPR 2025 [Paper] [Code]
23. InternVL3.5: Advancing open-source multimodal models with cascade RL and visual resolution routing arXiv 2025 [Paper] [Code]
24. Temporal-oriented recipe for transferring large vision-language model to video understanding arXiv 2025 [Paper]
25. Video Panels for long video understanding arXiv 2025 [Paper]
26. BLIP3-o: A family of fully open unified multimodal models arXiv 2025 [Paper] [Code]
27. VideoP2R: Video understanding from perception to reasoning arXiv 2025 [Paper]

Agentic and streaming Video-LLMs

1. VideoLLM-online: Online video large language model for streaming video CVPR 2024 [Paper] [Code]
2. VideoAgent (Stanford): Long-form video understanding with large language model as agent ECCV 2024 [Paper] [Code]
3. VideoAgent (PKU): A memory-augmented multimodal agent for video understanding ECCV 2024 [Paper] [Code]
4. Flash-VStream: Memory-based real-time understanding for long video streams ICCV 2025 [Paper] [Code]
5. Dispider: Enabling video LLMs with active real-time interaction via disentangled perception, decision, and reaction arXiv 2025 [Paper] [Code]
6. VideoChat-Online & OVBench: Online video understanding benchmark and model arXiv 2025 [Paper] [Code]
7. StreamChat: Streaming video understanding and multi-round interaction with memory-enhanced knowledge ICLR 2025 [Paper]
8. StreamAgent: Towards anticipatory agents for streaming video understanding arXiv 2025 [Paper]
9. video-SALMONN S: Streaming audio-visual LLMs beyond length limits via memory arXiv 2025 [Paper]
10. Video-Thinker: Sparking "thinking with videos" via reinforcement learning arXiv 2025 [Paper]

Model training perspective

Pre-training

1. VideoBERT: A joint model for video and language representation learning ICCV 2019 [Paper] [Code]
2. Learning video representations using contrastive bidirectional transformer arXiv 2019 [Paper]
3. ActBERT: Learning global-local video-text representations CVPR 2020 [Paper] [Code]
4. CLIP2TV: Align, match and distill for video-text retrieval arXiv 2021 [Paper]
5. Understanding Chinese video and language via contrastive multimodal pre-training arXiv 2021 [Paper]
6. DeCEMBERT: Learning from noisy instructional videos via dense captions and entropy minimization NAACL 2021 [Paper] [Code]
7. MERLOT: Multimodal neural script knowledge models NeurIPS 2021 [Paper] [Code]
8. Revealing single frame bias for video-and-language learning arXiv 2022 [Paper] [Code]
9. VAST: A vision-audio-subtitle-text omni-modality foundation model and dataset NeurIPS 2023 [Paper] [Code]
10. VideoPrism: A foundational visual encoder for video understanding ICML 2024 [Paper]
11. LanguageBind: Extending video-language pretraining to N-modality by language-based semantic alignment ICLR 2024 [Paper] [Code]

Fine-tuning

1. Multilevel language and vision integration for text-to-clip retrieval AAAI 2019 [Paper] [Code]
2. ST-Adapter: Parameter-efficient image-to-video transfer learning NeurIPS 2022 [Paper] [Code]
3. Zero-shot video question answering via frozen bidirectional language models NeurIPS 2022 [Paper] [Code]
4. Attentive moment retrieval in videos SIGIR 2018 [Paper]
5. To find where you talk: Temporal sentence localization in video with attention based location regression arXiv 2018 [Paper]
6. Cross-modal adapter for text-video retrieval arXiv 2022 [Paper] [Code]
7. AIM: Adapting image models for efficient video action recognition ICLR 2023 [Paper] [Code]
8. Prompting visual-language models for efficient video understanding ECCV 2022 [Paper] [Code]
9. Multi-modal circulant fusion for video-to-language and backward IJCAI 2018 [Paper]
10. Long-term temporal convolutions for action recognition arXiv 2016 [Paper] [Code]
11. READ: Recurrent adapter with partial video-language alignment for parameter-efficient transfer learning in low-resource video-language modeling AAAI 2024 [Paper] [Code]
12. Video instruction tuning with synthetic data (LLaVA-Video) arXiv 2024 [Paper] [Code]

Reinforcement learning and video reasoning

A new paradigm following DeepSeek-R1: rule-based rewards and GRPO are used to elicit explicit chain-of-thought and temporal reasoning in Video-LLMs.

1. Visual-RFT: Visual reinforcement fine-tuning arXiv 2025 [Paper] [Code]
2. Vision-R1: Incentivizing reasoning capability in multimodal large language models arXiv 2025 [Paper] [Code]
3. Video-R1: Reinforcing video reasoning in MLLMs NeurIPS 2025 [Paper] [Code]
4. VideoChat-R1: Enhancing spatio-temporal perception via reinforcement fine-tuning NeurIPS 2025 [Paper] [Code]
5. TimeZero: Temporal video grounding with reasoning-guided LVLM arXiv 2025 [Paper] [Code]
6. DeepVideo-R1: Video reinforcement fine-tuning via difficulty-aware regressive GRPO arXiv 2025 [Paper]
7. Reinforcing video reasoning with focused thinking arXiv 2025 [Paper]
8. Veason-R1: Reinforcing video reasoning segmentation to think before it segments arXiv 2025 [Paper]
9. Video-Thinker: Sparking "thinking with videos" via reinforcement learning arXiv 2025 [Paper]
10. video-SALMONN-o1: Reasoning-enhanced audio-visual large language model ICML 2025 [Paper]
11. Reinforcing structured chain-of-thought for video understanding (SDRL) arXiv 2026 [Paper]
12. Answer-consistent chain-of-thought reinforcement learning for multimodal LLMs (ACRE) arXiv 2025 [Paper]
13. STEP: Enhancing Video-LLMs' compositional reasoning by spatio-temporal graph-guided self-training CVPR 2025 [Paper]
14. Video-TwG: Think with grounding for long video understanding arXiv 2026 [Paper]
15. VideoP2R: Video understanding from perception to reasoning arXiv 2025 [Paper]

Data perspective

Manual collection

1. Advancing high-resolution video-language representation with large-scale video transcriptions (HD-VILA) CVPR 2022 [Paper] [Code]
2. HowTo100M: Learning a text-video embedding by watching hundred million narrated video clips ICCV 2019 [Paper] [Code]
3. FIBER: Fill-in-the-blanks as a challenging video understanding evaluation framework ACL 2022 [Paper] [Code]
4. NExT-QA: Next phase of question-answering to explaining temporal actions CVPR 2021 [Paper] [Code]
5. The "Something Something" video database for learning and evaluating visual common sense arXiv 2017 [Paper] [Code]
6. Rescaling egocentric vision: Collection, pipeline and challenges for EPIC-Kitchens-100 IJCV 2020 [Paper] [Code]
7. From representation to reasoning: Towards both evidence and commonsense reasoning for video question answering CVPR 2022 [Paper] [Code]
8. Grounding action descriptions in videos TACL 2013 [Paper]
9. Multimodal pretraining for dense video captioning AACL-IJCNLP 2020 [Paper] [Code]
10. QVHighlights: Detecting moments and highlights in videos via natural language queries NeurIPS 2021 [Paper] [Code]
11. VATEX: A large-scale, high-quality multilingual dataset for video-and-language research ICCV 2019 [Paper] [Code]
12. EgoSchema: A diagnostic benchmark for very long-form video language understanding NeurIPS 2023 [Paper] [Code]
13. Video-MME: The first-ever comprehensive evaluation benchmark of multi-modal LLMs in video analysis arXiv 2024 [Paper] [Code]
14. MMBench-Video: A long-form multi-shot benchmark for holistic video understanding arXiv 2024 [Paper] [Code]
15. TemporalBench: Benchmarking fine-grained temporal understanding for multimodal video models arXiv 2024 [Paper] [Code]
16. CG-Bench: Clue-grounded question answering benchmark for long video understanding arXiv 2024 [Paper] [Code]
17. Video-MMMU: Evaluating knowledge acquisition from multi-discipline professional videos arXiv 2025 [Paper] [Code]
18. VCR-Bench: A comprehensive evaluation framework for video chain-of-thought reasoning arXiv 2025 [Paper]
19. Video-Holmes: Multi-step deductive reasoning over videos arXiv 2025 [Paper]
20. MT-Video-Bench: A holistic video understanding benchmark for multi-turn dialogues arXiv 2025 [Paper]
21. MVX-Bench: Multi-video understanding benchmark arXiv 2026 [Paper]

Data augmentation

1. SVFormer: Semi-supervised video transformer for action recognition CVPR 2023 [Paper] [Code]
2. Semi-supervised video paragraph grounding with contrastive encoder CVPR 2022 [Paper]
3. Learning temporal action proposals with fewer labels arXiv 2019 [Paper]
4. Self-supervised learning for semi-supervised temporal action proposal CVPR 2021 [Paper] [Code]
5. Semi-supervised action recognition with temporal contrastive learning CVPR 2021 [Paper] [Code]

Manual annotation

1. Collecting highly parallel data for paraphrase evaluation (MSVD) ACL 2011 [Paper] [Code]
2. MSR-VTT: A large video description dataset for bridging video and language CVPR 2016 [Paper] [Code]
3. TGIF-QA: Toward spatio-temporal reasoning in visual question answering CVPR 2017 [Paper] [Code]
4. Towards automatic learning of procedures from web instructional videos (YouCook2) arXiv 2018 [Paper] [Code]
5. HMDB: A large video database for human motion recognition ICCV 2011 [Paper] [Code]
6. The Kinetics human action video dataset arXiv 2017 [Paper] [Code]
7. TVSum: Summarizing web videos using titles CVPR 2015 [Paper] [Code]
8. TVR: A large-scale dataset for video-subtitle moment retrieval ECCV 2020 [Paper] [Code]
9. COIN: A large-scale dataset for comprehensive instructional video analysis CVPR 2019 [Paper] [Code]
10. Cross-task weakly supervised learning from instructional videos CVPR 2019 [Paper] [Code]
11. Moments in Time dataset: One million videos for event understanding CVPR 2019 [Paper] [Code]
12. MLVU: Benchmarking multi-task long video understanding CVPR 2025 [Paper] [Code]
13. LongVideoBench: A benchmark for long-context interleaved video-language understanding NeurIPS 2024 [Paper] [Code]

Automatic generation

1. MAMA: Meta-optimized angular margin contrastive framework for video-language representation learning ECCV 2024 [Paper] [Code]
2. Progressive graph attention network for video question answering ACMMM 2021 [Paper] [Code]
3. The StreetLearn environment and dataset arXiv 2019 [Paper] [Code]
4. VidChapters-7M: Video chapters at scale NeurIPS 2023 [Paper] [Code]
5. Learning audio-video modalities from image captions (VideoCC3M) ECCV 2022 [Paper]
6. InternVid: A large-scale video-text dataset for multimodal understanding and generation ICLR 2024 [Paper] [Code]
7. How2: A large-scale dataset for multimodal language understanding NeurIPS 2018 [Paper] [Code]
8. Learning video representations from textual web supervision (WTS70M) arXiv 2020 [Paper]
9. Panda-70M: Captioning 70M videos with multiple cross-modality teachers CVPR 2024 [Paper] [Code]
10. ShareGPT4Video: Improving video understanding and generation with better captions NeurIPS 2024 [Paper] [Code]
11. Video-R1-CoT-165k & Video-R1-260k: CoT and RL datasets for video reasoning NeurIPS 2025 [Paper] [Code]
12. Video-Thinker-10K: Autonomous tool-usage chain-of-thought dataset arXiv 2025 [Paper]
13. VL3-Syn7M: High-quality re-captioned image-text data for video LMMs arXiv 2025 [Paper] [Code]

---

🤝 Contributing

Contributions are very welcome! If you'd like to add a paper, dataset, or correct a leaderboard entry:

1. For new papers: open a PR adding the entry under the most appropriate section, following the existing formatting (Title Venue Year [Paper] [Code]).
2. For leaderboard updates: please cite the original paper and the specific evaluation split.
3. For errors: an issue with a brief description is enough — we'll patch it.

We aim to keep this list focused on video–language understanding (retrieval, captioning, QA, grounding, temporal localization, and instruction tuning). Pure video-generation, action recognition without language, or text-only NLP papers are typically out of scope.

---

📬 Contact

For questions about the survey itself, feel free to reach out to the authors via the corresponding emails listed in the paper.

---

_{Last updated: May 2026 · Maintained by the authors of the ACL 2024 Findings survey · 2025–2026 papers added}