Frontier Vision-Language Models: Architectural Evolution, Benchmarks, Applications, and Challenges

🌐 Language: English · 简体中文

A most Frontend Collection and survey of vision-language model papers, and models GitHub repository

---

🧭 The Evolution of VLM Architectures

VLM design has gone through four distinct architectural eras in just six years — and Era 3 has split into two parallel branches. Early models kept frozen vision and language towers, aligned contrastively (CLIP) or bridged by a learnable connector into a frozen LM (BLIP-2, Flamingo). The 2023–2025 generation made a pretrained LLM the trunk and treated vision as a bolt-on adapter (LLaVA, Qwen2.5-VL, GPT-4V). The 2025–2026 generation drops the bridge entirely and early-fuses all modalities into a single transformer — forking along the output axis — and in 2026 the trunk is becoming a world model that predicts and acts:

• Era 3a — Native Multimodal Input → Text Out. Image, video, and (sometimes) audio enter a single early-fused token stream, but generation is still autoregressive text. This is the design used by today's general-purpose flagships: Qwen3.5 / Qwen3.6, Gemma 4, Gemini 3, GPT-5.4, Phi-4-Reasoning-Vision, Claude Opus 4.6, Nemotron 3 Nano Omni.
• Era 3b — Omni-Modal Unified I/O. The same fused trunk plus dedicated image / video decoder (VAE / DiT / flow-matching) and/or audio codec decoder heads, so the model can also generate images, video, and speech — via autoregression or, increasingly, discrete diffusion / AR-Diffusion (LLaDA2.0-Uni, Mamoda2.5) — and the newest 3b models (SenseNova-U1.5, 09/2026) are dropping the VAE too, running understanding + reasoning + generation through one encoder-free and VAE-free mixture-of-transformers trunk. This is the design used by unified models: SenseNova-U1.5, BAGEL, Qwen3.5-Omni, InternVL-U, Emu3 / Emu3.5, Erin 5.0, DeepSeek-Janus-Pro, LLaDA2.0-Uni, Mamoda2.5. Generation-only specialists share the same decoder stack without the understanding half — Sora 2, Veo 3, Kling now generate video with synchronized audio, and they double as the substrate for Era 4 world models (DreamX-World builds on Wan, OmniDreams on Cosmos, SolarWM on Wan2.2 / LTX-2.5 / MiniMax-H3).
• Era 4 — World-Action Models (2026 →). The unified trunk adds action as a first-class modality and closes the loop with the environment: it predicts future observations, maintains persistent state and spatial memory, and emits actions — generator, perceiver, and policy in one network. Founding releases: Cosmos 3, Kairos, DreamX-World 1.0, OmniDreams; the 08–09/2026 wave: DreamX-Phi 1.0 (WorldArena 2.0 winner), SimWAM (91.5 PDMS NAVSIM), ZimaBlue (120k h video pretraining), G0.5 (one AR stream for reasoning+action), Puffin-World (native 3D world states), SolarWM (open 5B–33B, real-time minute-to-hour rollouts) (see §1.1 World Models).

Reading the diagram (left → right). Era 1 uses a two-tower design — aligned contrastively (CLIP: no generative decoder) or with a learnable cross-modal bridge (e.g. Q-Former) into a frozen LM — text-only output. Era 2 puts a pretrained LLM at the center; an MLP/Resampler projects visual tokens into the LLM's vocabulary, and the LLM does all the reasoning — still text-only output. Era 3a drops the bridge: image, video, audio, and text share a single tokenizer/embedding space and flow through one early-fused transformer — but the output is still autoregressive text. Era 3b keeps that fused trunk and adds decoder heads (image/video DiT, VAE, audio codec) so the model can natively output text, image, video, and/or speech — generation-only video models (Sora 2, Veo 3, Kling) reuse this decoder stack, now with synchronized audio. Era 4 adds an action token stream, persistent state, and a policy head, closing the observation → action → next-observation loop: the world-action model is generator, perceiver, and policy at once. Eras 3a, 3b, and 4 coexist; the choice is essentially "how much do you want the model to generate — and to act?"

---

🆕 What's in this repo

Below we compile awesome papers and model and github repositories that

• State-of-the-Art VLMs Collection of newest to oldest VLMs (we'll keep updating new models and benchmarks).
• Evaluate VLM benchmarks and corresponding link to the works
• Post-training/Alignment Newest related work for VLM alignment including RL, sft.
• Applications applications of VLMs in embodied AI, robotics, etc.
• Contribute surveys, perspectives, and datasets on the above topics.

📡 VLM Trends

VLM Trends is the live companion to this repository. This README records what exists; VLM Trends tracks what changed today — new model releases, papers, benchmarks, and datasets pulled daily from arXiv, Hugging Face, GitHub, and Semantic Scholar, scored on a published rubric, and grouped by topic and model family. It also makes this survey browsable, and charts how each research direction is moving over time.

Progressive research reports

We track new VLMs, benchmarks, and post-training methods that haven't yet been folded into the main tables in dated mini-surveys:

📂 Expand all 10 reports — latest: 2026-09-13, Era 4 fills out and the unified trunk drops the VAE (50 new entries)

• 📰 2026-09-13 — latest: Era 4 becomes a paradigm — six world-action systems ship in five weeks: DreamX-Phi 1.0 (WorldArena 2.0 winner), SimWAM (91.5 PDMS NAVSIM), ZimaBlue (120k h video pretraining, 30 Hz on RTX 4090), G0.5 (one AR stream for reasoning+action), Puffin-World (native 3D world states), SolarWM (open 5B–33B). Unified stack keeps shedding components — SenseNova-U1.5 ships 8B mixture-of-transformers encoder-free and VAE-free at native 4K. First VL foundation model for driving: Qwen-Drive-1.0 (Alibaba). Speech unification: AuK (Tencent Hunyuan, 4.5× via 4-step). First open >700B multimodal RL post-training stack: Miles v0.1 (GLM-5.2 744B-A40B, 64 GB300s, 263s/step). Frontier eval reopens on continuous settings — VideoGAIA puts GPT-5.5/Kimi K3 <60%; EgoMonth: 71.8% Gemini vs 94.2% human; StateSight probes latent state; WorldReward and PAWBench add reward/probabilistic-alignment axes for world models — 50 new entries since August 10.

• 📰 2026-08-10 — Evaluation stops asking whether a model sees — HumanCLAW (can a VLM act through a body?), GST-Bench (global spatial awareness from video), ChronoVision (temporal reasoning via latent state), WorldExam (reactivity over appearance); judging becomes its own research problem via OSReward, ConfBench, TruthLens. On the model side: Qwen3.8-Max (2.4T · 95B active, Vision Arena #2), DiffusionGemma (26B-A4B diffusion Gemma), Hunyuan3D-Buffalo 1.0 (unified 3D); plus N₀-VTLA (tactile VLA), Metis, Ego2Robot, VideoCoCo, OmniPack — 29 new entries since July 22.

• 📰 2026-07-22 — World models become the evaluators — GigaWorld-1 + WMBench, RoboWorld (r = 0.989 vs real-world), World-Action-Model tutorial; Gemma 4 tech report (encoder-free 12B), PRA-GRPO (93.2% V-Star at 4B), VRRL (trained self-reflection), LingBot-VLA 2.0 (60,000-hour corpus), ROSA (robot-factory serving), ISPA (50% KV-cache cut), OmniFocus, MoHallBench / LongVQUBench / SciVis-literacy benchmarks; plus the July frontier wave: Gemini 3.6 Flash, Kimi K3 (2.8T open MoE), GPT-5.5 / GPT-5.6 Sol, Grok 4.5, Qwen3.7-Plus — 19 new entries since June 27.

• 📰 2026-06-27 — World-model foundation releases — Cosmos 3 (NVIDIA omnimodal family: best open T2I/I2V + best RoboArena policy), Kairos (4B edge-real-time world-model stack, beats 14B), DreamX-World 1.0 (5B MIT-licensed interactive world model); "Persistent State Core" critique + Echo-Memory; ZPPO (teacher-in-prompts beats GRPO), Qwen-RobotManip (38,100-hour corpus), Supervise What Survives, VisCritic (visual process rewards for GUI), HPP (long video), IMCBench (medical dialogue safety) — 11 new entries since June 23.

• 📰 2026-06-23 — World Models focus — NVIDIA OmniDreams (real-time closed-loop driving world model), Mirage (latent spatial memory), Reward-as-Agent (GRPO for world models), WorldOlympiad & LongSpace-Bench (world-model benchmarks); plus PP-OCRv6 (34.5M beats 235B VLMs on OCR), Occ-VLM (3D-grounded), discrete-diffusion RL reasoning, VLA layer pruning, Hy-Embodied-0.5-VLA, RT-VLA (44.8× faster driving), VLA language steering — 12 new entries since June 2.

• 📰 2026-06-02 — Mamoda2.5 (AR-Diffusion DiT-MoE, 95.9× faster editing), VLM3 (native 3D learners), AlphaGRPO (RL for unified-model generation), Stage-wise Preference Optimization, FastOCR / WindowQuant (KV-cache efficiency), Fast-dDrive / CLOVER / CoWorld-VLA (driving VLA), Lost in Fog (reasoning-as-safety-signal), LiteGUI (SFT-free GUI agents), Health-Conditioned VLA, POLAR, TOC-Bench / VGenST-Bench (video), HalluCXR (medical) — 16 new entries since May 16.

• 📰 2026-05-16 — LensVLM (Apple), Nemotron 3 Nano Omni (NVIDIA), LLaDA2.0-Uni, PLaMo 2.1-VL, S-GRPO / Faithful GRPO / GRPO-TTA / OpenSearch-VL, MindVLA-U1 (surpasses human driving), VLADriver-RAG, Green-VLA, Anticipation-VLA, VLA Foundry, LAMO, ScreenExplorer, VideoZeroBench, Video-Oasis, MedThinkVQA, data curation at 87× less compute — 34 new entries since April 28.

• 📰 2026-04-28 — Qwen3.6-27B & Qwen3.6-35B-A3B, Claude Mythos (gated), S1-VL, GLM-5V-Turbo, FreshPER / GMPO / ARPO / GRPO-VPS, QUOTA, Fast-dVLM, VLA-World, SpanVLA, VLA-Forget, R-VLM, UILoop, WebForge, WorldMark, Video-MME-v2, CrossMath, BabyVision, SlowBA — 30 new entries since April 13.

• 📰 2026-04-13 — LFM2.5-VL-450M, EXAONE 4.5, Gemma 4, Granite 4.0 3B Vision, InternVL-U, GLM-4.6V, Vero, MolmoWeb, UniDriveVLA, QAPruner, Firebolt-VL, CoME-VL, and more.

• 📰 2026-03-25 — GPT-5.4, Phi-4-Reasoning-Vision-15B, Gemini 3.0, Qwen3.5, Claude Opus 4.6, Molmo2, and more.

Welcome to contribute and discuss!

---

🤩 Papers marked with a ⭐️ are contributed by the maintainers of this repository. If you find them useful, we would greatly appreciate it if you could give the repository a star or cite our paper.

---

Table of Contents

• 📄 Paper Link/⛑️ Citation

• 1. 📚 SoTA VLMs
  • 1.1. 🌍 World Models
• 2. 🗂️ Dataset and Evaluation
  • 2.1. Large Scale Pre-Training & Post-Training Dataset
  • 2.2. Datasets and Evaluation for VLM
  • 2.3. Benchmark Datasets, Simulators and Generative Models for Embodied VLM

• 3. 🔥 Post-Training/Alignment/prompt engineering 🔥

  • 3.1. RL Alignment for VLM
  • 3.2. Regular finetuning (SFT)
  • 3.3. VLM Alignment Github
  • 3.4. Prompt Engineering
• 4. ⚒️ Applications
  • 4.1. Embodied VLM agents
  • 4.2. Generative Visual Media Applications
  • 4.3. Robotics and Embodied AI
    • 4.3.1. Manipulation
    • 4.3.2. Navigation
    • 4.3.3. Human-robot Interaction
    • 4.3.4. Autonomous Driving
  • 4.4. Human-Centered AI
    • 4.4.1. Web Agent
    • 4.4.2. Accessibility
    • 4.4.3. Medical and Healthcare
    • 4.4.4. Social Goodness
• 5. ⛑️ Challenges
  • 5.1. Hallucination
  • 5.2. Safety
  • 5.3. Fairness
  • 5.4. Alignment
    • 5.4.1. Multi-modality Alignment
      • 5.4.2. Commonsense and Physics Alignment
  • 5.5. Efficient Training and Fine-Tuning
  • 5.6. Scarce of High-quality Dataset

0. Citation

@InProceedings{Li_2025_CVPR,
    author    = {Li, Zongxia and Wu, Xiyang and Du, Hongyang and Liu, Fuxiao and Nghiem, Huy and Shi, Guangyao},
    title     = {A Survey of State of the Art Large Vision Language Models: Benchmark Evaluations and Challenges},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
    month     = {June},
    year      = {2025},
    pages     = {1587-1606}
}

---

1. 📚 SoTA VLMs

1.1. 🌍 World Models

World models moved from research demos to release-grade artifacts in mid-2026: unified trunks that serve as generator, perceiver, and policy at once. Foundation releases first, then memory/state analyses and benchmarks.

Benchmarks & analyses

See also the earlier world-model entries in §2.3 (UniSim, GAIA-1, LWM, Genesis, RoboGen) and the 2026-06-23 / 2026-06-27 / 2026-09-13 progressive reports.

2. 🗂️ Benchmarks and Evaluation

2.1. Datasets for Training VLMs

2.2. Datasets and Evaluation for VLM

🧮 Visual Math (+ Visual Math Reasoning)

💬 Benchmark for Unified Models

🎞️ Video Understanding

💬 Multimodal Conversation

🧠 Multimodal General Intelligence

🔎 Visual Reasoning / VQA (+ Multilingual & OCR)

📝 Visual Text / Document Understanding (+ Charts)

🌄 Text‑to‑Image Generation

🚨 Hallucination Detection / Control

2.3. Benchmark Datasets, Simulators, and Generative Models for Embodied VLM

3. ⚒️ Post-Training

3.1. RL Alignment for VLM

3.2. Finetuning for VLM

3.3. VLM Alignment github

3.4. Prompt Optimization

4. ⚒️ Applications

4.1 Embodied VLM Agents

4.2. Generative Visual Media Applications

4.3. Robotics and Embodied AI

4.3.1. Manipulation

4.3.2. Navigation

4.3.3. Human-robot Interaction

4.3.4. Autonomous Driving

4.4. Human-Centered AI

4.4.1. Web Agent

4.4.2. Accessibility

4.4.3. Healthcare

4.4.4. Social Goodness

5. Challenges

5.1 Hallucination

5.2 Safety

5.3 Fairness

5.4 Alignment

5.4.1 Multi-modality Alignment

5.4.2 Commonsense and Physics Alignment

5.5 Efficient Training and Fine-Tuning

5.6 Scarce of High-quality Dataset