@misc{aydogan2024awesomevlmarchitectures,
  author       = {Gökay Aydoğan},
  title        = {Awesome VLM Architectures},
  year         = {2024},
  howpublished = {\url{https://github.com/gokayfem/awesome-vlm-architectures}},
  note         = {GitHub repository, fal.ai},
  url          = {https://github.com/gokayfem/awesome-vlm-architectures}
}

2026: MODUS | Argus-Unified | Kimi K3 | Mage-VL | Inkling | Hy-Embodied-VLM | MonkeyOCRv2 | MiniMax M3 | InternVideo3 | Keye-VL 2.0 | Zamba2-VL | Cosmos 3 | Lance | ZAYA1-VL | Falcon Perception | GLM-5V-Turbo | PLaMo 2.1-VL | EXAONE 4.5 | BidirLM and BidirLM-Omni | Gemma 4 | Penguin-VL | Phi-4-Reasoning-Vision | V-SONAR and V-LCM | Qwen3.5 | Youtu-VL | Kimi K2.5 and K2.6 | Step3-VL-10B

2025: ERNIE 5.0 | DeepSeek-OCR | PaddleOCR-VL | Qwen3-VL | Step3 | GLM-4.1V-Thinking | ERNIE 4.5-VL | MiMo-VL | BAGEL | Seed1.5-VL | InternVL3 and InternVL3.5 | Kimi-VL | Llama 4 Scout and Maverick | Qwen2.5-Omni | Gemma 3 | Aya Vision | Phi-4-multimodal | SigLIP 2 | EVEv2 | Qwen2.5-VL | VideoLLaMA 3 | UI-TARS | MiniMax-01 | MiniCPM-o-2.6 | Eagle 2 | Sa2VA

2024: VideoChat-Flash | OmniVLM | Apollo | DeepSeek-VL2 | Maya | InternVL 2.5 | PaliGemma 2 | ShowUI | SmolVLM | AIMv2 | LLaVA-CoT | LLM2CLIP | Tarsier2 | Janus and Janus-Pro | ARIA | Emu3 | Molmo and PixMo | Llama 3.2-Vision | NVLM | Pixtral 12B | VILA-U | Qwen2-VL | EAGLE | Show-o | Idefics3-8B | Transfusion | mPLUG-Owl3 | VITA | LLaVA-OneVision | VILA² | INF-LLaVA | SlowFast-LLaVA | EVLM | InternLM-XComposer-2.5 | OMG-LLaVA | Cambrian-1 | EVE | Ovis | Parrot | ConvLLaVA | Phi-3-Vision and Phi-3.5-Vision | CogVLM2 | Chameleon | PaliGemma | xGen-MM (BLIP-3) | MANTIS | Moondream-next | Idefics2 | InternLM-XComposer2-4KHD | MM1 | DeepSeek-VL | AnyGPT | SPHINX-X | LLaVA 1.6 | MiniCPM-V | MouSi | InternLM-XComposer2 | MoE-LLaVA | moondream1 and moondream2 | FireLLaVA | COSMO

2023: TinyGPT-V | MobileVLM | Alpha-CLIP | Nous-Hermes-2-Vision - Mistral 7B | SPHINX | Florence-2 | u-LLaVA | LLaVA-Plus | OtterHD | CoVLM | GLaMM | Fuyu-8B | PaLI-3 Vision Language Models | MiniGPT-v2 | BakLLaVA | Ferret | LLaVA 1.5 | CogVLM | MetaCLIP | Qwen-VL | IDEFICS | BLIVA | KOSMOS-2 | LaVIN | InstructBLIP | ImageBind | LLaVA | MiniGPT-4 | SigLIP | OpenFlamingo | PaLM-E | KOSMOS-1 | BLIP-2

2022: MULTIINSTRUCT | PaLI | Flamingo | BLIP

2021: GLIP | FROZEN | CLIP

2020: ViT

MODUS is a decoder-only any-to-any model that represents diverse modalities inside one autoregressive architecture. Unlike encoder-decoder or diffusion systems assembled around modality-specific output paths, the same model predicts any supported modality from any combination of the others. This makes intermediate-modality chains and self-scoring through a second generated modality native behaviors rather than external workflows.

The design deliberately reuses strong pretrained decoder-only priors instead of training a bespoke multimodal stack from scratch. A single checkpoint is evaluated across heterogeneous tasks and modalities, making MODUS most notable as a general architectural formulation rather than a narrowly optimized VLM endpoint.

Argus-Unified resolves the conflicting visual representations required by comprehension and synthesis through hybrid visual tokens. Continuous encoder features preserve semantic information for understanding, while a learned quantizer produces discrete tokens for image generation from the same frozen visual encoder.

Training first learns the quantizer and image decoder, then initializes the language component from a pretrained VLM and trains unified multimodal prediction. The reported recipe uses 15.6 million examples and roughly $2,000 of compute, positioning the model as a reproducible baseline for understanding-and-generation research rather than a scale demonstration.

Kimi K3 has 2.8T total and 104B active parameters across 93 layers. Sixty-nine layers use gated delta-rule linear attention and 24 use gated Multi-head Latent Attention; learned Attention Residuals create structured information paths across depth. Stable LatentMoE activates 16 of 896 routed experts plus two shared experts, making the model unusually sparse for its scale.

A 401M-parameter MoonViT-V2 encoder supplies native image and video input. The model supports a 1,048,576-token context and receives multimodal, long-context, agentic, reasoning, and tool-use training. Quantization-aware pretraining targets MXFP4 weights and MXFP8 activations rather than treating low-precision deployment as an afterthought.

Mage-ViT replaces uniform frame sampling with selective 16×16-patch encoding guided by motion vectors and residual energy across anchor and predicted frames. This codec-native tokenization reduces visual-token use by more than 75% while retaining spatiotemporal context. The encoder is trained from scratch on roughly 560M unlabeled images and 100M unlabeled video frames.

The complete VLM uses a dual-system design: a small event gate decides when a stream warrants deeper processing, and a causal decoder performs contextual reasoning. This turns streaming perception into an event-driven process and yields up to a reported 3.5× wall-clock speedup while retaining static-image and spatial reasoning ability.

Inkling contains 975B total and 41B active parameters. Its MoE layers activate six of 256 routed experts plus two shared experts using sigmoid routing and auxiliary-loss-free balancing. Attention alternates five local sliding-window layers with one global layer and uses learned relative-position representations rather than RoPE; short convolutions further refine key, value, attention, and MLP pathways.

Pretraining spans 45T text, image, audio, and video tokens. Post-training begins with a small synthetic supervised bootstrap and scales asynchronous reinforcement learning across multimodal understanding, tools, coding, mathematics, conversation, and safety. The released weights support controllable reasoning effort and contexts up to one million tokens.

Hy-Embodied-VLM-1.0 connects Hy-ViT2 to a roughly 30B-total, 3B-active language model. Each sparse-MoE layer selects eight of 128 routed experts plus one shared expert. It accepts as many as 128 images in a 32K context and exposes direct-response and explicit-thinking modes through the same checkpoint.

Its curriculum moves from action-relevant state understanding to transition reasoning and then sequential, adaptive reasoning. A self-evolving post-training loop alternates reinforcement learning with rejection-sampling fine-tuning, separately optimizing geometric precision and higher-level planning before policy fusion.

MonkeyOCRv2 is a visual-text foundation encoder specialized for the properties that natural-image pretraining often misses: dense text, small glyphs, fine strokes, and layout. Its two complementary objectives align images with their transcribed content while forcing the encoder to retain pixel-level document structure.

The MonkeyDoc v2 corpus contains 113M document images in 17 languages. The frozen encoder improves recognition, detection, tamper analysis, segmentation, parsing, and understanding; paired with a lightweight language model, it forms a 0.7B document parser. Public weights were announced July 11, two days before the arXiv report, so the timeline follows the model release rather than the paper date.

MiniMax Sparse Attention adds a lightweight index branch that scores KV blocks independently for each grouped-query-attention group. The main branch performs exact attention only over selected blocks, while a KV-outer kernel groups queries retrieving the same block to improve memory locality and reuse.

The released 109B-parameter model is trained natively across modalities and evaluated at contexts up to one million tokens. The sparse-attention report focuses on architecture, kernel design, and scaling behavior; it does not publish a complete source-level training-data inventory.

Multimodal Contextual Reasoning maintains one evolving context containing observations, instructions, intermediate reasoning, tool actions, and memory. This closed loop lets the model accumulate and verify evidence across long videos rather than treating video QA as a single static prompt.

Multimodal Multi-head Latent Attention reparameterizes and compresses KV-cache states while preserving the complete visual-token sequence. Training combines continued pretraining, short-to-long supervised tuning, rule-based reinforcement learning, and on-policy distillation; a retrieval-equipped video-agent implementation demonstrates the intended tool-using setting.

Keye-VL-2.0-30B-A3B is the first reported adaptation of DeepSeek Sparse Attention to grouped-query multimodal attention. Only 3B of 30B parameters activate per token, while the sparse retrieval mechanism keeps critical frames and long-range dependencies accessible across a 256K context. Heterogeneous ViT-language-model parallelism and custom sparse-attention kernels address the systems cost of hour-long video.

Cross-Modal Multi-Teacher On-Policy Distillation feeds dense token-level guidance from specialist teachers back into on-policy trajectories. Context-RL and Video-RL then target long-context reasoning, temporal localization, code, search, tools, and multimodal self-correction without collapsing the multi-task mixture.

Zamba2-VL extends the Zamba hybrid backbone to vision-language inputs in 1.2B, 2.7B, and 7B variants. Most sequence processing occurs in Mamba-2 state-space layers, with a small set of shared attention blocks supplying global token interaction. This preserves transformer-like multimodal reasoning while moving more computation toward near-linear prefill and bounded recurrent state.

The architecture is particularly relevant for edge and long-context deployment: visual sequences can be large even when the language model is small, so reducing quadratic attention changes time-to-first-token and memory behavior more materially than ordinary parameter pruning.

Cosmos 3 uses a Mixture-of-Transformers with two communicating towers. An autoregressive transformer reasons over language, vision, and action sequences; a diffusion transformer generates continuous images, video, and world trajectories. Their shared representation lets the family serve as a VLM, forward or inverse dynamics model, simulator, generator, or policy without separate task pipelines.

Nano configurations combine 8B-class components and Super configurations combine 32B-class components. Training spans text, image, video, audio, simulated physical interaction, action trajectories, robot demonstrations, driving, and synthetic worlds. The July 20 Cosmos 3 Edge release preserves the same two-tower design in a smaller real-time deployment branch and is folded into this family.

Lance interleaves text, image, and video in a shared 3B-parameter sequence model while routing semantic understanding and visual synthesis through dedicated experts. Semantic ViT tokens coexist with clean or noised VAE latents, and generalized 3D causal attention plus modality-aware positional encoding preserve spatial and temporal relationships.

One checkpoint supports visual understanding, text-to-image and text-to-video generation, image-to-video generation, and editing. Its training-from-scratch recipe uses no more than 128 A100 GPUs, making the contribution as much about an attainable unified architecture as headline generation quality.

ZAYA1-VL-8B joins a Qwen2.5-VL visual encoder to ZAYA1's MoE decoder. Vision-only LoRA and Compressed Convolutional Attention parameters activate for visual tokens without duplicating the language backbone, while bidirectional attention within image-token spans improves spatial integration.

The model is trained on roughly 140B vision-language tokens assembled from open data and released under Apache 2.0. Its importance is the clean separation of visual specialization from shared language capacity in a compact sparse model.

Falcon Perception is a dense early-fusion Transformer in which image patches and text interact from the first layer. Its attention mask is bidirectional over image tokens and causal over prediction tokens. Variable-length instance tokens identify multiple objects, while lightweight continuous heads produce masks without turning segmentation into an unwieldy text sequence.

The family includes a 600M perception model and a 300M OCR model. The paper was submitted March 28, but the timeline uses the documented May 3 public model launch, following this repository's release-date policy.

GLM-5V-Turbo is organized around end-to-end multimodal agency across images, video, webpages, documents, GUIs, code, and external tools. Its contribution is the integration of perception with the agent loop: observations remain available during planning and execution, and outcomes can be visually verified rather than handed off to a detached captioning stage.

The report describes coordinated changes to model design, multimodal training, hierarchical optimization, reinforcement learning, toolchains, and agent-framework integration. Because this is a new native agent foundation model rather than a latency-only checkpoint, it is kept separate from the GLM-4.1V family.

PLaMo 2.1-VL is designed around a constrained deployment envelope rather than frontier scale. Both sizes support VQA and bounding-box grounding, with evaluations emphasizing Japanese language operation and applications such as factory tool recognition and infrastructure anomaly detection.

A large synthetic-data pipeline supplies Japanese multimodal instructions, grounding examples, and domain-oriented supervision. The entry broadens the catalog's language and edge coverage; its contribution is a deployable bilingual VQA-and-grounding family rather than a new attention primitive.

EXAONE 4.5 adds a dedicated vision encoder to the EXAONE 4.0 framework and trains the resulting stack natively over text and visual inputs. Its data mixture is deliberately document-heavy, targeting enterprise documents, OCR-rich layouts, and Korean contextual reasoning while retaining general language capability.

Context extends to 256K tokens for long documents and enterprise workflows. The combination of open weights, native multimodal training, long context, and focused document/Korean coverage makes it a distinct public VLM release rather than a post-training adapter.

BidirLM removes the causal mask and adapts a decoder with masked next-token prediction and contrastive learning. Weight merging and mixed-domain training preserve generative knowledge while recovering the full-context interaction needed by retrieval and embedding tasks.

BidirLM-Omni combines Qwen3 text, Qwen3-VL, and Qwen3-ASR specialist backbones with frozen visual and audio projectors, aligning all three modalities in one contrastive space. The entry belongs beside CLIP, SigLIP, and ImageBind as an encoder architecture, despite originating from decoder checkpoints.

Gemma 4 is a suite of related architectures rather than one uniform network. E2B and E4B target mobile and edge deployment; 31B is a dense server-class decoder; and 26B-A4B is a sparse Mixture-of-Experts model with roughly 4B active parameters. These branches accept images through improved visual processing, while the mobile models additionally support audio. The distinctive Gemma 4 12B Unified removes separate vision and audio encoders: raw image patches and audio features enter through direct linear projections, reducing duplicated encoder memory and latency. Context windows reach 128K or 256K depending on the variant.

Pretraining jointly develops language, code, reasoning, and multimodal understanding, followed by instruction tuning, safety alignment, and reasoning post-training. A configurable thinking mode lets the instruction-tuned models emit a reasoning trace before answering. The family also ships dedicated multi-token-prediction draft models for speculative decoding, while per-layer embeddings and shared key-value caching improve inference efficiency in applicable variants.

Google describes a large multilingual mixture containing web text, documents, code, mathematics, image-text pairs, OCR and document examples, audio, and video-derived supervision. The complete record-level training corpus is not released, so benchmark collections should not be mistaken for an exhaustive dataset list. Image support includes variable resolutions and aspect ratios; native audio and video coverage depends on the specific branch.

Penguin-VL builds its Penguin-Encoder by initializing a vision backbone from a text-only Qwen3-0.6B model. Causal attention is converted to bidirectional attention, and 2D rotary position embeddings permit variable-resolution visual tokenization. The encoder is connected to compact language decoders to form 2B- and 8B-class VLMs. For video, Temporal Redundancy-Aware compression assigns larger token budgets to informative key frames and compresses redundant intermediate frames, allowing longer sequences within a fixed visual-token budget.

Training begins with mixed-supervision encoder adaptation. A teacher vision encoder supplies amplitude, direction, and relational distillation targets while image reconstruction-style supervision stabilizes the LLM-to-vision conversion. A low-to-high-resolution curriculum then aligns the encoder for fine-grained perception, followed by unified image and video multimodal pretraining and instruction tuning. This recipe is designed to preserve OCR, spatial, temporal, and dense-captioning signals that category-oriented contrastive objectives can suppress.

The project releases Penguin-Recap-I, a reconstructed high-quality image training set, and Penguin-Recap-V, video annotations at dense timestamp, paragraph, and whole-video granularities. These resources complement broader image-text, document, OCR, mathematical reasoning, and video instruction mixtures described in the report. The authors' comparisons attribute the compact models' gains primarily to visual representation quality and token allocation rather than parameter scaling alone.

Phi-4-Reasoning-Vision-15B uses a mid-fusion architecture built from the Phi-4-Reasoning language backbone and a SigLIP 2 vision encoder. Visual features are projected into the language embedding space and inserted into the decoder sequence. Dynamic-resolution processing produces up to 3,600 visual tokens, supporting fine-grained documents and interface screenshots. Bidirectional attention is restricted to tokens within each image, improving spatial interaction while avoiding the overfitting Microsoft observed with broader bidirectional attention.

The model is trained through supervised fine-tuning on a deliberately mixed curriculum of reasoning and non-reasoning examples. Explicit <think> and <nothink> modes allow one checkpoint to use extended reasoning for mathematics and science or answer directly for perception-heavy tasks such as captioning, OCR, detection, and grounding. Microsoft reports four days of training on 240 NVIDIA B200 GPUs, followed by safety SFT and automated red-team evaluation.

Training data primarily comprises heavily filtered and corrected open-source vision-language datasets, augmented with synthetic examples, internally produced domain data, and targeted acquisitions. Microsoft emphasizes filtering, error correction, and synthetic augmentation as larger contributors than raw scale, but does not publish a complete source-level dataset inventory. Public evaluations span scientific diagrams, charts, mathematical reasoning, OCR, and GUI grounding; those benchmarks are evaluation references, not a claimed exhaustive training list.

V-SONAR starts with Meta's Perception Encoder for image and video features and learns a lightweight projector into the pre-existing SONAR embedding space. Positional embeddings and a temporal-attention layer preserve frame order, while attention pooling produces a modality-level representation compatible with SONAR text concepts. Because the target space is shared with OMNISONAR, the representation can be decoded into text across SONAR's multilingual coverage. A text-only Large Concept Model can also consume these aligned visual embeddings without vision-specific pretraining.

V-SONAR is trained with a coarse-to-fine post-hoc alignment curriculum rather than retraining the foundation encoders from scratch. The alignment uses paired visual-caption supervision and SONAR text embeddings, with staged optimization and asynchronous learning rates for the newly initialized projector and pretrained encoder. V-LCM then concatenates V-SONAR visual concepts with SONAR language concepts and retains LCM's two-tower contextualizer and denoiser architecture. It predicts the next continuous concept embedding with the same latent-diffusion objective used during text-only LCM pretraining.

Vision-language instruction tuning uses M3IT, which spans image and video inputs, eight task categories, and 80 languages. Retrieval and captioning experiments additionally use VATEX, DREAM-1K, PE-Video, and VideoXum. The resulting system separates perception, multilingual semantic alignment, and generative concept modeling instead of converting every modality into a single discrete token vocabulary.

Qwen3.5 is a natively multimodal decoder family rather than a separate text model with a later VL edition. Its flagship 397B-A17B checkpoint uses sparse Mixture-of-Experts routing, activating about 17B parameters per token, and combines Gated DeltaNet linear-attention layers with standard softmax-attention layers. Visual inputs are handled inside the same model interface as text, supporting image and video reasoning, tool use, coding, and long-context agent workflows. Smaller dense and MoE variants extend the design across deployment scales.

The training pipeline combines large-scale multimodal pretraining with supervised fine-tuning and reinforcement learning. Qwen describes a scalable asynchronous RL system spanning text, multimodal, and multi-turn interaction, enabling a single checkpoint to switch between thinking and non-thinking behavior. The data mixture covers natural language, code, mathematics, images, video, documents, GUI interactions, tool calls, and multi-turn agent trajectories; the full source-level corpus is not publicly enumerated.

Qwen3.6 is best treated as a continuation of this architecture rather than a separate VLM family. Its open dense and sparse checkpoints retain native multimodality and unified thinking and non-thinking modes while prioritizing stability and agentic coding. Folding these releases into the Qwen3.5 entry keeps the catalog focused on architectural changes rather than model-version churn.

Vision-Language Unified Autoregressive Supervision learns a visual codebook whose symbols expand the text vocabulary into one multimodal vocabulary. Visual tokens are not passive conditioning embeddings: the decoder predicts them alongside text through joint visual-token and text reconstruction.

Because dense outputs are serialized in the same vocabulary, a standard autoregressive VLM can produce segmentation, monocular depth, human pose, and object detections without separate heads. This turns dense perception into native generation while retaining ordinary visual question answering and instruction following.

Kimi K2.5 connects MoonViT to a 1T-total, 32B-active decoder with 61 layers, Multi-head Latent Attention, and 384 routed experts. Eight routed experts plus shared capacity activate per token. Native multimodal training supports images, video, reasoning, code, tools, and GUI interaction within the same model.

Released April 20, Kimi K2.6 retains the K2.5 architecture while extending context to 256K and strengthening coding, research, and multimodal-agent post-training. It is folded here rather than receiving a duplicate timeline row; Kimi K3 is separate because its attention, residual, sparsity, scale, and vision stack change materially.

Step3-VL-10B is architecturally distinct from the much larger 2025 Step3 sparse-MoE system. Its 1.8B perception encoder is aligned to language during pretraining, then a projector with two stride-2 stages reduces the spatial sequence by 16× before feeding a Qwen3-8B decoder.

Images combine one 728×728 global view with 504×504 local crops, treated as independent batch items. Explicit newline tokens mark patch rows while standard one-dimensional RoPE is retained. The PaCoRe parallel proposer-and-synthesizer reasoning method is a post-training and inference addition rather than the defining visual architecture.

ERNIE 5.0 maps language, images, video, and audio into one autoregressive framework using Next-Group-of-Tokens Prediction. Text uses next-token and multi-token prediction; visual generation predicts the next frame and scale; audio generation predicts codec tokens depth-wise. Modality-agnostic expert routing activates less than 3% of the 2.4T-parameter network.

Elastic depth, width, and top-k sparsity train a super-network from which smaller deployment configurations can be extracted. Baidu first unveiled ERNIE 5.0 at Baidu World on November 13, 2025, formally released it in January 2026, and published the technical report on February 4; the timeline therefore uses the earliest documented public disclosure.

DeepSeek-OCR consists of DeepEncoder and a DeepSeek-3B-MoE decoder with roughly 570M active parameters. DeepEncoder combines a SAM-derived local-perception pathway, a CLIP-derived semantic pathway, convolutional compression, and windowed and global attention to turn high-resolution pages into as few as 64 to 800 visual tokens. Multiple resolution modes trade recognition fidelity for compression.

Training first develops the encoder's visual-text representation and then jointly optimizes document decoding with the MoE language model. Data covers multilingual text pages, natural images, formulas, tables, charts, diagrams, and structured document conversion, supplemented with synthetic renderings. The work frames OCR as an experiment in optical compression for future long-context memory, rather than only a document benchmark model.

DeepSeek-OCR 2, released January 27, 2026, replaces the fixed raster-scan assumption with DeepEncoder V2 and Visual Causal Flow. The encoder dynamically reorders visual tokens according to document semantics before language decoding, exploring whether cascaded one-dimensional causal reasoning can better represent complex two-dimensional layouts. Paper · Repository

PaddleOCR-VL-0.9B connects a NaViT-style dynamic-resolution visual encoder to the compact ERNIE-4.5-0.3B language model. Variable-resolution packing preserves page detail and aspect ratio without forcing every document through a fixed square canvas. The decoder emits text and structured representations for paragraphs, tables, formulas, charts, and other document elements across 109 languages.

Its curriculum combines element-level recognition with page-level document parsing, followed by instruction tuning on complex layouts and structured outputs. Training data includes multilingual OCR, synthetic and scanned documents, tables, mathematical expressions, charts, reading-order annotations, and layout-rich pages. PaddleOCR-VL-1.5 (January 2026) adds robust physical-document distortions, seal recognition, and text spotting; 1.6 (June 2026) adds region-aware data optimization and progressive reinforcement-learning post-training without changing the core architecture.

Qwen3-VL combines a native-dynamic-resolution Vision Transformer with Qwen3 language backbones in dense 2B, 4B, 8B, and 32B configurations and sparse 30B-A3B and 235B-A22B Mixture-of-Experts configurations. Its DeepStack mechanism injects features from multiple ViT depths into corresponding language-model layers instead of relying only on the visual encoder's final layer. Interleaved-MRoPE distributes temporal, height, and width position information across rotary-embedding frequencies, while explicit text-timestamp alignment improves event localization in video. The family supports interleaved text, images, and video in a native 256K-token context, with documented extrapolation for longer contexts.

Training proceeds through large-scale multimodal pretraining followed by supervised instruction tuning and reasoning-oriented post-training. The released Instruct checkpoints target direct response and agent interaction, whereas Thinking checkpoints generate extended reasoning for visual mathematics, spatial analysis, and other difficult multimodal tasks. The curriculum emphasizes recognition, multilingual OCR, document structure, 2D and 3D grounding, long-video understanding, GUI operation, and code generation from visual inputs.

The technical report describes a broad mixture of text, image-text, document, OCR, grounding, chart, multi-image, video, and agent-interaction data. Qwen does not publish a reproducible itemized list of every pretraining source, so individual benchmark datasets should not be presented as the complete training corpus.

Step3 uses a 321B-parameter sparse-MoE decoder with approximately 38B parameters active per token. The language stack contains dense lower layers followed by routed expert layers, while its vision pathway supports images and video for multimodal reasoning. The model is designed together with an Attention-FFN Disaggregation serving architecture, separating attention and expert computation so hardware placement and communication topology match their different workloads.

Training combines large-scale text and multimodal pretraining with instruction tuning and reasoning-oriented reinforcement learning. The data mixture spans language, code, mathematics, image-text, OCR, documents, charts, multi-image, video, and multimodal reasoning. The system report discloses architecture and deployment mechanisms more clearly than the source-level training corpus, which remains categorically described.

GLM-4.1V-9B-Thinking joins an AIMv2-Huge-initialized vision encoder, an MLP projector, and the GLM-4-9B-0414 language decoder. The vision tower replaces two-dimensional patch convolutions with 3D convolutions, temporally downsampling video while duplicating single images for consistent processing. Interpolated absolute position embeddings and 2D RoPE support native resolutions and extreme aspect ratios; the language decoder applies 3D RoPE to multimodal tokens. Explicit timestamp tokens after video frames supply temporal position and distance cues.

Training begins with broad multimodal pretraining, followed by continual training on video, higher-resolution imagery, and sequences up to 32K tokens. Supervised fine-tuning then teaches a standardized long-reasoning format with separate <think> and <answer> spans. The final stage introduces Reinforcement Learning with Curriculum Sampling (RLCS). Rather than sampling a fixed task mixture, RLCS estimates changing model competence and prioritizes informative problems of suitable difficulty. Domain-specific rule, model, and hybrid rewards cover both verifiable and open-ended tasks.

Pretraining draws on curated caption pairs, knowledge-rich web and academic-book interleaving, 220M OCR images, natural-image and GUI grounding, instructional video, and text replay. SFT and RL span STEM reasoning, charts, documents, video, grounding, coding, GUI agents, and general visual instruction following. The later GLM-4.5V and GLM-4.6V families scale and extend this framework; they are successors, not separate architectures in this catalog.

ERNIE 4.5-VL includes 424B-total/47B-active and 28B-total/3B-active multimodal MoE models, plus dense members of the broader ERNIE 4.5 family. Its heterogeneous modality MoE shares some experts across text and visual tokens while reserving other experts for individual modalities. Modality-isolated routing, router orthogonality loss, and multimodal token-balancing loss reduce competition between modalities.

The models are jointly pretrained on text, images, and video, then post-trained in both direct-response and thinking modes. PaddlePaddle infrastructure uses heterogeneous hybrid parallelism, hierarchical load balancing, FP8 mixed precision, and fine-grained recomputation to train and serve the sparse models efficiently. Baidu describes broad text, visual-knowledge, OCR, document, chart, video, and reasoning mixtures but does not provide a complete dataset manifest.

MiMo-VL-7B joins a high-resolution vision encoder and projector to a 7B language decoder and is released as supervised and reinforcement-learned checkpoints. Its primary contribution is the training system rather than an exotic connector: long-chain-of-thought examples are introduced during pretraining, not only during post-training, so multimodal reasoning is developed alongside perception and language.

Four pretraining stages consume approximately 2.4T tokens and progressively align vision, general multimodal knowledge, high-quality reasoning, and long-context capabilities. Mixed On-Policy Reinforcement Learning then trains one checkpoint across general QA, mathematics, grounding, and GUI tasks using domain-appropriate rule and model rewards. The mixture includes text replay, image-text, OCR, document, chart, video, spatial-grounding, GUI, and reasoning data; a complete source-level inventory is not published.

BAGEL is a unified decoder-only model with 14B total and 7B active parameters. Its Mixture-of-Transformer-Experts architecture duplicates the Qwen2.5-initialized transformer into understanding and generation experts: text and SigLIP 2 ViT tokens use the understanding parameters, while FLUX-VAE latent tokens use the generation parameters. Both experts operate on one interleaved sequence through shared self-attention at every layer, avoiding a small connector bottleneck between comprehension and synthesis. Text is learned with next-token prediction; images are generated with rectified flow.

A generalized causal-attention scheme lets later text or images attend to earlier clean visual representations while preventing access to noised generation targets. Visual understanding uses a native-aspect-ratio-adapted SigLIP2-So400m encoder and MLP connector; generation uses a frozen FLUX VAE. Training progresses through alignment, large-scale pretraining, continued training, and supervised fine-tuning, jointly optimizing cross-entropy and flow-matching objectives.

BAGEL is pretrained on trillions of tokens assembled from text-only, paired image-text, and interleaved multimodal sources. The reported inventory includes 400M text samples, 500M understanding pairs, 1.6B generation pairs, 100M interleaved understanding samples, 45M video-derived sequences, and 20M web documents. OCR, charts, grounding, editing sets, and 500K reasoning-augmented generation and manipulation examples support document reading, spatial control, image editing, and long-context multimodal reasoning.

Seed1.5-VL uses a compact vision tower and a large sparse-MoE language model, maintaining 20B active decoder parameters while scaling total capacity substantially higher. Its visual pathway supports native-aspect-ratio images, multiple images, and video, with explicit support for spatial grounding and GUI control. The same model can provide short direct answers or extended multimodal reasoning.

Training progresses through vision-language pretraining, instruction tuning, long-chain-of-thought cold start, and reinforcement learning. The report emphasizes careful balancing of general text, visual knowledge, OCR and documents, charts, grounding, video, 3D understanding, GUI trajectories, games, and verifiable multimodal-reasoning problems. Source-level corpus provenance and all mixture proportions are not publicly reproducible. The arXiv v1 date is used because an earlier generally accessible official launch date is not unambiguously documented.

InternVL3 retains the InternViT-MLP-LLM structure and dynamic image tiling of earlier InternVL releases, but jointly learns language and multimodal knowledge during native pretraining instead of treating vision alignment primarily as a later adaptation stage. The family spans dense and sparse-MoE language backbones. Its training recipe combines multimodal pretraining, supervised fine-tuning, Mixed Preference Optimization, and test-time scaling for visual reasoning.

Released on August 26, 2025, InternVL3.5 is part of the same evolving family. It introduces Cascade RL, using offline RL for stable coarse alignment followed by online RL for refinement. A Visual Resolution Router selects visual-token resolution according to input complexity, while Decoupled Vision-Language Deployment places the vision encoder and language model on different devices to balance inference load. Training covers image, multi-image, document, chart, video, GUI, grounding, multilingual, and reasoning data; full pretraining provenance is not enumerated.

Kimi-VL-A3B consists of MoonViT, a two-layer MLP projector, and a Moonlight-derived MoE language model with 16B total and 2.8B activated decoder parameters. MoonViT packs variable-sized patch sequences without tiling images into separately processed crops. It combines interpolated SigLIP-SO400M position embeddings with two-dimensional RoPE, preserving pretrained visual knowledge while representing fine spatial positions. A 2×2 pixel-shuffle operation compresses its output before projection into the language model.

After the language checkpoint's 5.2T-token text-only phase, Kimi-VL undergoes standalone vision training, vision-language alignment, joint pretraining, a high-quality cooldown, and long-context activation. These stages consume 4.4T additional tokens, including 2T for MoonViT training, 0.1T for alignment, and 2.3T across joint stages. Context is progressively extended from 8K to 128K. The instruct model receives joint supervised fine-tuning. Kimi-VL-Thinking adds long-chain-of-thought SFT and reinforcement learning; the later Thinking-2506 checkpoint is folded into this family.

The multimodal mixture is organized into six categories: caption, interleaved image-text, OCR, knowledge, video, and agent data. MoonViT learns from alt text, synthetic captions, grounding boxes, and OCR targets using both SigLIP-style contrastive and captioning losses. Long-video, long-document, academic visual QA, GUI-grounding, and text replay data provide long-context and agent capabilities while preserving language performance.

Llama 4 Scout and Llama 4 Maverick are natively multimodal, autoregressive MoE models rather than text decoders retrofitted only during instruction tuning. Scout has 109B total parameters across 16 experts and Maverick approximately 400B across 128 experts; both activate about 17B parameters per token. Early fusion places image and multilingual text tokens in a shared model sequence. Scout supports a 10M-token context, whereas Maverick supports 1M tokens.

Scout was trained on roughly 40T tokens and Maverick on roughly 22T. Meta describes a mixture of public data, licensed data, information from its products and services, and interactions with Meta AI, without releasing a complete corpus manifest. Post-training combines supervised fine-tuning, online reinforcement learning, and direct preference optimization. Maverick was co-distilled from the unreleased Llama 4 Behemoth teacher; Scout also uses distillation during training.

Qwen2.5-Omni is an end-to-end any-input model whose block-wise audio and vision encoders accept streaming speech, images, and video alongside text. Time-aligned Multimodal RoPE (TMRoPE) interleaves audio and video representations on a synchronized timeline. Its Thinker-Talker design separates semantic reasoning from speech realization: the Thinker produces text and hidden representations, while a dual-track autoregressive Talker conditions on those states to generate speech tokens concurrently.

A sliding-window diffusion transformer converts audio tokens into waveform output without waiting for a complete response, reducing first-packet latency. Training jointly aligns text, vision, video, audio, and speech-generation objectives, followed by multimodal instruction tuning. The report describes caption, OCR, video, audio-transcription, speech, and general text mixtures but does not publish a reproducible source-level inventory.

Gemma 3 is a family of 1B, 4B, 12B, and 27B open-weight models; the 4B, 12B, and 27B variants accept images as well as text. A shared, frozen 400M-parameter SigLIP vision encoder processes images at 896×896 resolution. Its embeddings are average-pooled into a fixed sequence of 256 soft visual tokens, while a Pan-and-Scan preprocessing method handles non-square and high-resolution inputs through crops. The 1B model remains text-only.

For efficient long-context inference, Gemma 3 changes the decoder from Gemma 2's balanced attention schedule to five local sliding-window layers for every global-attention layer. Local attention uses a 1,024-token window, substantially limiting KV-cache growth; global layers retain access to the complete context. The larger models are pretrained at 32K context and extended to 128K near the end of pretraining through RoPE rescaling.

Training relies heavily on knowledge distillation from a larger teacher, updating the language model while keeping the vision encoder frozen. Post-training combines supervised fine-tuning with preference-based reinforcement learning. The pretraining mixture contains text, code, mathematics, multilingual material, and multimodal examples; the report states token totals of 2T, 4T, 12T, and 14T for the four model sizes but does not publish a complete dataset inventory.

Aya Vision is released in 8B and 32B configurations supporting 23 languages. It connects a vision encoder to Aya Expanse-derived language models and addresses the loss of text-only capability that often occurs when vision is introduced. Its principal modeling contribution is cross-modal model merging, which combines separately learned text and multimodal strengths rather than relying only on sequential fine-tuning.

Training uses a synthetic annotation pipeline that generates and filters culturally and linguistically diverse multimodal instructions, avoiding dependence on literal machine translation alone. The mixture covers multilingual visual question answering, captions, OCR, image-grounded generation, and text-and-image translation. The full pretraining corpus is not released; the paper's reproducible contributions center on its annotation, merging, and evaluation recipes.

Phi-4-multimodal is a 5.6-billion-parameter multimodal model built around the Phi-4-Mini language model. Separate vision and speech encoders transform non-text inputs, while modality projectors map their representations into the language model's embedding space. Its defining mechanism is a Mixture-of-LoRAs: modality-specific LoRA parameters and routers adapt the shared decoder for vision-language, speech-language, and joint vision-speech inference. The design isolates modality extensions more effectively than fully merging every adaptation into the base weights; the speech and audio LoRA component itself contains 460 million parameters.

The underlying language model uses grouped-query attention, a 200K-token vocabulary, and supports a 128K-token context. Multimodal training first aligns the modality encoders and projectors, then jointly optimizes modality-specific adaptations. The released instruct checkpoint receives supervised fine-tuning followed by direct preference optimization and reinforcement learning from human feedback for instruction following and safety.

Its data recipe builds on the language, vision, and speech research used for Phi-3.5 and Phi-4. Vision supervision covers captioning, OCR, charts, diagrams, document understanding, visual question answering, and multi-image reasoning. Speech data spans automatic speech recognition, speech translation, and summarization across multiple languages and acoustic conditions. The model card provides a data summary rather than a complete sample-level corpus inventory, so undisclosed sources should not be inferred from benchmark coverage.

SigLIP 2 retains the dual-encoder Vision Transformer design and pairwise sigmoid image-text objective of SigLIP, but broadens the representation learned by the vision tower. Its unified recipe combines contrastive image-text learning with captioning-based pretraining, masked prediction, self-distillation, and online data curation. These additions improve not only zero-shot classification and retrieval, but also localization and dense tasks such as segmentation, depth estimation, and surface-normal prediction. Models are released at ViT-B, ViT-L, So400m, and ViT-g scales.

Alongside conventional fixed-resolution checkpoints, the NaFlex variants use patch-and-pack processing to retain an image's native aspect ratio and accept different sequence lengths with a single checkpoint. This avoids distortive square resizing and lets users trade visual-token count for inference cost, which is especially useful for documents, screenshots, and OCR-heavy inputs. Standard fixed-resolution models additionally receive a self-distillation stage.

Training uses WebLI, comprising 10 billion images and 12 billion alt-texts across 109 languages. The reported mixture draws 90% of image-text pairs from English pages and 10% from non-English pages, while debiasing filters target representation and association biases involving sensitive attributes. SigLIP 2 is positioned both as a standalone multilingual image-text encoder and as a stronger frozen vision tower for downstream VLMs.

EVEv2 departs from the traditional encoder-based VLM approach. Instead of relying on a pre-trained vision encoder (like CLIP), it builds visual perception directly within a decoder-only Large Language Model (LLM). Key architectural features include: Divide-and-Conquer: This is the core innovation. Instead of mixing visual and textual information throughout the entire LLM, EVEv2 introduces modality-specific components. This means separate attention matrices (query, key, value), Layer Normalization layers, and Feed-Forward Networks for visual and textual tokens. This reduces interference and allows for more efficient learning. It's a fully sparse, decoder-only architecture. Patch Embedding Layer: A minimalist patch embedding layer is learned from scratch. This avoids the inductive biases of pre-trained vision encoders. It uses two convolutional layers (Conv1 and Conv2) to process image patches. Lossless Encoding: Unlike some encoder-free models that use discrete tokenization (which can lose information), EVEv2 aims for lossless encoding of visual information. LLM Adaptation: The architecture is designed for seamless adaptation to existing LLMs. The paper experiments with Vicuna-7B and Qwen2-7B. Multi-Stage Training: A four-stage training process is used: LLM-guided Pre-aligning: Only the patch embedding layer is trained, using re-captioned web data (EVE-recap-10M). The LLM is frozen. This establishes a basic alignment between visual and textual representations. Vision Perception Learning: Vision layers within the LLM are trained, using progressively larger datasets and image resolutions. The LLM weights are still frozen. Vision-Text Fully alligning: The entire network is update. Supervised Fine-tuning (SFT): The entire model is fine-tuned on question-answering and instruction-following datasets. DenseFusion++: A new, efficient captioning engine is introduced to generate high-quality image-text pairs for training. This is crucial for building strong visual perception from scratch. It leverages multiple vision experts. Data Efficiency: A key focus of the research is demonstrating that EVEv2 can achieve strong performance with less data than comparable encoder-based models, thanks to its efficient architecture.

Qwen2.5-VL builds upon its predecessor, Qwen2-VL, with substantial improvements in perception of temporal and spatial scales, as well as a simplified network structure for increased efficiency. World-wide Image Recognition: Expanded recognition capabilities covering a vast array of categories, including landmarks, objects, and even film/TV IPs. Precise Object Grounding: Uses bounding boxes and point-based representations for object localization, with standardized JSON output for coordinates and attributes, enabling hierarchical positioning. Enhanced Text Recognition (OCR): Improved multi-scenario, multi-language, and multi-orientation text recognition and localization, with enhanced information extraction for applications like document processing. Powerful Document Parsing: Introduces "QwenVL HTML" format, leveraging HTML for layout information extraction from documents, magazines, research papers, web pages, and mobile screenshots. Enhanced Video Comprehension: Supports understanding of ultra-long videos (hourly scale) with dynamic frame rate (FPS) training and absolute time encoding. Enables second-level event localization and key point summarization. Visual Agent Capabilities: Can function as a visual agent for computer and phone use, capable of reasoning and dynamically directing tools. Capable of tasks like booking flights. Time and Image Size Perception In the spatial dimension, the model is capable of adapting varying image sizes into tokens and directly represents coordinates by detection boxes. In the temporal dimension, the model can comprehend the pace of time through temporal dimension. Visual Encoder A native dynamic resolution ViT is trained from scratch. Window Attention is used to minimize computational load. The model comes in three sizes (3B, 7B, and 72B parameters), with both base and instruct-tuned versions available. The 72B-Instruct model achieves competitive performance on various benchmarks, excelling in document and diagram understanding. Smaller models also demonstrate strong performance, with the 7B-Instruct model outperforming GPT-4o-mini in several tasks and the 3B model exceeding the performance of the previous Qwen2-VL 7B model. The models is trained with 18 trillion tokens. Future developments aim to further enhance problem-solving, reasoning, and multi-modality integration.

VideoLLaMA3 introduces a vision-centric approach in both its training paradigm and framework design, focusing on enhancing image and video understanding capabilities. The core architecture incorporates a pre-trained vision encoder (SigLIP), a video compressor (DiffFP), a projector, and a large language model (LLM - Qwen2.5). The model employs a four-stage training process: 1) Vision Encoder Adaptation, where the vision encoder is adapted to accept images of variable resolutions using scene images, document data, and scene text images; 2) Vision-Language Alignment, which jointly tunes the vision encoder, projector, and LLM using large-scale image-text data (including detailed captions, documents, and charts) and a small amount of text-only data; 3) Multi-task Fine-tuning, incorporating image-text data for downstream tasks and general video caption data; and 4) Video-centric Fine-tuning, using general videos, streaming videos, temporally grounded videos, image-only, and text-only data. A key innovation is Any-resolution Vision Tokenization (AVT), which allows the vision encoder to process images and videos of any resolution by replacing fixed positional embeddings with Rotary Position Embedding (RoPE). This enables handling images with variable shapes and minimal information loss. For video inputs, Differential Frame Pruner (DiffFP) acts as a video compressor, reducing the number of vision tokens by comparing the 1-norm distance between temporally consecutive patches in pixel space and pruning redundant patches. This makes video representations more compact and precise. The training data mixture is carefully curated for each stage, emphasizing high-quality image-text data. The Vision Encoder Adaptation stage uses datasets like VL3-Syn7M-short, LLaVA-Pretrain-558k, and document datasets. The Vision-Language Alignment stage expands on this with detailed captions, OCR data, and fine-grained data with bounding boxes. The Multi-task Fine-tuning stage adds question-answering data and general video caption data. Finally, the Video-centric Fine-tuning stage includes general videos, streaming videos, and temporal grounding data. This "vision-centric" approach, prioritizing image understanding as a foundation for video understanding, along with AVT and DiffFP, allows VideoLLaMA3 to achieve strong performance on both image and video benchmarks.

UI-TARS leverages several key innovations: (1) Enhanced Perception, utilizing a large-scale GUI screenshot dataset for context-aware understanding and precise captioning of UI elements; (2) Unified Action Modeling, standardizing actions into a unified space across platforms and achieving precise grounding through large-scale action traces; (3) System-2 Reasoning, incorporating deliberate reasoning for multi-step decision-making, including task decomposition, reflection, and milestone recognition; and (4) Iterative Training with Reflective Online Traces, addressing the data bottleneck by automatically collecting, filtering, and refining interaction traces on hundreds of virtual machines. The model is trained iteratively and tuned via reflection, continuously learning from mistakes and adapting to new situations with minimal human intervention. The architecture takes screenshots as input and uses a Vision-Language Model (VLM), specifically Qwen-2-VL 7B and 72B, to process visual information and generate actions. The action space is unified across platforms (mobile, desktop, web) and includes actions like click, type, scroll, and drag. Reasoning is infused by generating explicit "thoughts" before each action, inspired by reasoning-and-acting frameworks. These thoughts are generated through a combination of curated GUI tutorials and augmented action traces, incorporating patterns like task decomposition, long-term consistency, milestone recognition, trial and error, and reflection. The training process involves multiple stages, starting with perception enhancement using a curated dataset of GUI screenshots and associated metadata. This dataset supports tasks like element description, dense captioning, state transition captioning, question answering, and set-of-mark prompting. Action modeling is improved by creating a large-scale dataset of action traces and using grounding data to pair element descriptions with spatial coordinates. The model is trained using a combination of supervised fine-tuning (SFT) and Direct Preference Optimization (DPO) with reflection tuning to learn from errors.

Hybrid Attention: The core innovation is the hybrid attention mechanism. It primarily uses "lightning attention" (an I/O-aware implementation of TransNormer linear attention) for efficiency. However, to maintain strong retrieval capabilities, it strategically inserts a standard transformer block with softmax attention after every seven transnormer blocks (with lightning attention). This is a key differentiator from purely linear attention models, which often struggle with retrieval tasks. Mixture of Experts (MoE): To scale the model efficiently, MiniMax-01 employs a Mixture of Experts (MoE) architecture in the feed-forward layers. It has a massive 456 billion total parameters, but only 45.9 billion are activated for each token, using 32 experts with a top-2 routing strategy. This allows for a large model capacity without a corresponding increase in computational cost per token. Vision-Language Model (MiniMax-VL-01): The vision-language model (MiniMax-VL-01) builds upon MiniMax-Text-01 by integrating a lightweight Vision Transformer (ViT) module. It uses a dynamic resolution strategy, resizing input images to various sizes (from 336x336 to 2016x2016) and concatenating features from both resized patches and a standard thumbnail. It does not use pooling or downsampling on the visual features, relying instead on the long-context capabilities of the architecture. Demonstrates the viability of linear attention at a massive scale, achieving performance comparable to top-tier models while significantly extending the context window. Long-Context Capability: Supports context inputs of up to 4 million tokens, with strong performance in long-context evaluations. Efficient Training and Inference Framework: Introduces several novel algorithmic and engineering optimizations to handle the hybrid architecture, MoE, and long contexts efficiently. Pre-training: A curated corpus incorporating academic literature, books, web content, and programming code. Vision-Language Pre-training (VL-01): A substantial image-caption dataset (694 million unique pairs) and a dataset of 100 million images with fine-grained descriptions. Vision-Language Instruction Data (VL-01): A diverse instruction-based dataset synthesized from a wide array of image-related tasks. Hybrid Attention: The core fusion method combines the efficiency of lightning attention (linear) with the retrieval capabilities of softmax attention. MoE Routing: The top-2 routing strategy selectively activates experts, enhancing model capacity without increasing compute per token; a global router supports load balancing. Vision-Language Fusion (VL-01): Visual features from the ViT are projected into the LLM embedding space using a two-layer MLP. The raw, high-dimensional visual features are used without pooling or downsampling, leveraging the architecture's long-context capabilities. Varlen Ring Attention and LASP+: These algorithms enable efficient handling of long, variable-length sequences and data packing during training and inference.

MiniCPM-o-2.6 employs an end-to-end omni-modal architecture. It integrates several pre-trained components: Vision Encoder: SigLip-400M Audio Encoder: Whisper-medium-300M Text-to-Speech (TTS): ChatTTS-200M Large Language Model (LLM): Qwen2.5-7B. These components are connected and trained end-to-end. A key innovation is the "Omni-modal Live Streaming Mechanism." This involves: Online Modality Encoders/Decoders: The offline encoders and decoders are transformed into online versions to handle streaming inputs and outputs. Time-Division Multiplexing (TDM): A TDM mechanism within the LLM backbone processes omni-modal streams. It divides parallel streams (video, audio) into sequential information within short time slices. Configurable Speech Modeling: A multimodal system prompt (including text and audio prompts) allows for flexible voice configuration during inference, enabling voice cloning and description-based voice creation.

MiniCPM-o 4.5, released in April 2026, evolves this family toward real-time full-duplex omni-modal interaction. It retains streaming vision, speech, and text while improving simultaneous perception and generation; the update belongs here because it extends the same time-multiplexed, end-to-end omni-modal lineage. Paper · Repository

Eagle 2 adopts a "diversity first, then quality" data strategy, beginning with a large, diverse pool of over 180 data sources, followed by rigorous filtering and selection. The architecture uses a tiled mixture of vision encoders (MoVE), specifically SigLIP and ConvNeXt-XXLarge, with image tiling to handle high resolutions. Each image tile is encoded by channel-concatenated MoVE. The vision encoder outputs are concatenated and aligned with the LLM (Qwen2.5) via a simple MLP connector. A three-stage training recipe is used: Stage 1 trains the connector to align modalities; Stage 1.5 trains the full model on a large, diverse dataset; and Stage 2 fine-tunes on a high-quality instruction-tuning dataset. Crucially, all available visual instruction data is used in Stage 1.5, not just captioning/knowledge data. Balanced data packing addresses limitations in existing open-source frameworks. The core contribution is the detailed data strategy. This involves: (1) Data Collection: Building a highly diverse data pool (180+ sources) through both passive gathering (monitoring arXiv and Hugging Face) and proactive searching (addressing "bucket effect" via error analysis). (2) Data Filtering: Removing low-quality samples based on criteria like mismatched question-answer pairs, irrelevant image-question pairs, repeated text, and numeric formatting issues. (3) Data Selection: Choosing optimal subsets based on data source diversity, distribution, and K-means clustering on SSCD image embeddings to ensure balance across types (especially useful for chart data, etc.). (4) Data Augmentation: Mining information from input images through techniques like Chain-of-Thought (CoT) explanation generation, rule-based QA generation, and expanding short answers into longer ones. (5) Data Formating: remove unnecessary decorations. Training uses a three-stage approach: Stage 1: Aligns language and image modalities by training the MLP connector. Stage 1.5: Trains the full model using a large-scale, diverse dataset (21.6M samples). All available visual instruction data is used here, unlike common two-stage approaches, leading to substantial improvements. Stage 2: Fine-tunes the full model on a carefully curated, high-quality visual instruction tuning dataset (4.6M samples). The model is trained with AdamW. Eagle 2 demonstrates strong performance across a wide range of multimodal benchmarks, matching or outperforming frontier open-source and some closed-source VLMs.

Sa2VA leverages a pre-trained LLaVA-like model (containing a visual encoder, visual projection layer, and LLM) and appends SAM-2 alongside it. Crucially, it uses a decoupled design, where SAM-2's decoder and memory module are frozen. This preserves SAM-2's perception and tracking capabilities and allows Sa2VA to be a plug-and-play module, updatable with newer MLLMs. The connection between the LLM and SAM-2 is a special [SEG] token. The LLM generates this token, and its hidden states act as a spatial-temporal prompt for SAM-2's decoder, which produces segmentation masks. The model is trained end-to-end, demonstrating scalability. The training uses a unified instruction-tuning format for various tasks: referring segmentation, visual question answering (VQA), and grounded conversation generation (GCG) for both images and videos. It treats all images, videos and prompts as visual tokens. A key aspect is the co-training with multiple datasets, including image and video data. The authors introduce Ref-SAV, an auto-labeled dataset with over 72,000 object expressions in complex video scenes, and manually validate 2,000 video objects in Ref-SAV for benchmarking referring video object segmentation. A simple mask tracking method re-utilizes SAM-2's knowledge. The model formulates all tasks as a single instruction-tuning process. Datasets used for co-training are: LLAVA 1.5 (665K), RefCOCO (17K), RefCOCO+ (17K), RefCOCOg (22K), Grand-f (214K), ChatUniVi (100K). Ref-YTVOS (3.5K), MeVIS (0.6K), ReVOS (1.7K) and Ref-SAV (37K).

Hierarchical visual token Compression (HiCo): This is the core innovation. It compresses video information at two levels: Clip-level Compression: The video is divided into clips. A visual encoder (UMT-L) processes each clip, and a compressor (token merging with MLP) reduces the number of visual tokens. This exploits inter-frame redundancy. Video-level Compression: During the LLM (Qwen2-7B) interaction, visual tokens are further reduced using a progressive visual dropout strategy. This leverages the idea that the LLM focuses on the entire context at shallow layers and specific details at deeper layers. It combines uniform dropout (shallow layers) and text-guided selection (deep layers). Visual Encoder: UMT-L@224 (a video encoder, shown to be more efficient than image encoders like SigLIP). Visual-Language Connector: A compressor (token merging) followed by an MLP projection. Large Language Model (LLM): Qwen2-7B. Multi-stage Short-to-Long Learning: This is a crucial training strategy: Stage 1: Video-Language Alignment: Train the compressor and MLP with image-text and short video-text pairs (0.5M each). Stage 2: Short Video Pre-training: Enhance visual understanding with more images (3.5M) and short videos (2.5M). Stage 3: Joint Short & Long Video Instruction Tuning: Fine-tune on a mix of images (1.1M), short videos (1.7M), and long videos (0.7M) with instruction-following data. Stage 4: Efficient High-Resolution Post-finetuning: Adapt to higher resolutions (224 to 448) by fine-tuning the video encoder on a subset (25%) of Stage 3 data.Dynamic Video Sampling: Uses a dual sampling strategy: dense sampling (15 fps) for short videos (capturing motion) and sparse sampling (1 fps) for long videos (capturing events). Timestamp-aware Prompt: Uses a text prompt to provide timestamp information to the model. LongVid: A new large-scale long video instruction-tuning dataset introduced in the paper. It contains 114,228 long videos and 3,444,849 question-answer pairs across five task types. It leverages existing datasets (Ego4D, HowTo100M, HD-Vila, MiraData) and generates dense event labels. Mixed Training Data: Uses a combination of short and long videos during training. NIAH (Needle In A video Haystack). A newly created dataset for testing models capabilities for understanding long contexts.

OmniVLM addresses the challenges of deploying vision-language models (VLMs) on resource-constrained edge devices. It achieves this through a novel token compression mechanism and a multi-stage training pipeline. The core innovation is the image token compression, which transforms the embedding dimensions from [batch_size, 729, hidden_size] to [batch_size, 81, hidden_size] within the projection layer. This 9x reduction in token count is achieved through reshaping, chosen after empirical comparison against convolution-based methods. The model architecture (Figure 1) builds upon the LLaVA framework, employing Google's SigLIP-400M as the vision encoder, Qwen2.5-0.5B-Instruct as the base language model, and a Multi-Layer Perceptron (MLP) as the projection layer. The training pipeline consists of three stages: (1) Pretraining on large-scale image-caption pairs (primarily from the LLaVA pretraining dataset) to learn visual-linguistic alignments, training only the projection layer; (2) Supervised Fine-Tuning (SFT) on a mix of datasets (LLaVA, UnimmChat, and internal data) to improve contextual understanding and conversational coherence, training the projector and LLM while freezing the vision encoder; and (3) Minimal-Edit Direct Preference Optimization (DPO), using a teacher model to create minimally edited corrections to the base model's outputs, forming chosen-rejected pairs for preference learning, again freezing the vision encoder and training the projector and LLM. The DPO process leverages GPT-4V to generate synthetic training pairs. Extensive experiments show that the 81-token configuration provides the optimal balance between computational efficiency and model performance. OmniVLM outperforms nanoLLAVA on benchmarks like ScienceQA, POPE, and MMMU, demonstrating improved reasoning, multimodal comprehension, and generalization. Crucially, it achieves significantly faster inference speeds (9.1x faster time-to-first-token and 1.5x higher decoding speed compared to nanoLLAVA on a laptop, and 8x faster TTFT on a mobile device), making it suitable for deployment on edge devices like smartphones and laptops.

Apollo leverages the Qwen2.5 series of Large Language Models (LLMs) with 1.5B, 3B, and 7B parameters. The key architectural innovation is the combination of a SigLIP-SO400M image encoder and an InternVideo2 video encoder. Features from both encoders are interpolated and concatenated channel-wise before being fed into a Perceiver Resampler, which outputs 32 tokens per frame. This combination was empirically found to be superior to other encoder choices. The model uses a 3-stage training approach. Critically, the paper introduces the concept of "Scaling Consistency," demonstrating that design decisions made on smaller models and datasets (up to a critical size) effectively transfer to larger models. This allows for more efficient experimentation. The paper also advocates for frames-per-second (fps) sampling during training, as opposed to uniform frame sampling, and demonstrates its superiority. The optimal number of tokens is 8-32 per frame. It also includes a curated benchmark, ApolloBench, that reduces evaluation time by 41x compared to existing benchmarks while maintaining high correlation and focusing on temporal reasoning and perception. The exploration also includes Token Resampling showing that Perceiver resampling has a good performace. Token Integration is also discussed: Adding tokens (text, learned, etc.) between the video tokens derived from different frames or clips is sufficient for efficient token integration. Training Stages is also disscussed, concluding that progressively unfreezing the different components in different stages leads to superior model training dynamics. Finally, training the Video Encoder is discussed. The paper concludes that Finetuning video encoders on only video data further improves overall performance, especially on reasoning and domain-specific tasks. Data Composition is also studied. It concludes that Data mixture matters, and including a moderate amount of text data and maintaining a slight video-heavy mix leads to optimal performance.

DeepSeek-VL2 builds upon a LLaVA-style architecture. it consists of three core modules: (1) a vision encoder, (2) a vision-language adaptor, and (3) a Mixture-of-Experts language model. It introduces two major enhancements: a dynamic tiling strategy and it uses DeepSeekMOE language model that has Multi-head Latent Attention. The dynamic tiling strategy addresses the limitations of fixed-resolution encoders by splitting high-resolution images into tiles. It uses a single SigLIP-SO400M-384 vision encoder. A set of candidate resolutions CR = {(m· 384, η · 384) | m∈ N, n ∈ N, 1 ≤ m, n,mn ≤ 9} is defined, representing different aspect ratios. For an input image, the optimal resolution from CR that minimizes padding is selected. The resized image is then divided into m₁ × n₁ local tiles of 384 × 384 pixels, plus one global thumbnail tile. The SigLIP-SO400M-384 processes all (1 + m¡ × n₁) tiles, yielding 729 visual embeddings (27x27) of 1152 dimensions per tile. Dynamic tiling is disabled for multiple (>2) images for efficiency. A 2x2 pixel shuffle compresses each tile's visual tokens to 14x14 (196 tokens). Special tokens are added: 14 <tile_newline> tokens at the end of each row in the global thumbnail (total 210 tokens); m₁ · 14 <tile_newline> tokens at the end of the final column of the local tiles; and a <view_separator> token between the global thumbnail and local tiles. The total visual sequence length is 210 + 1 + m₁ · 14 × (nį · 14 + 1). This sequence is projected into the LLM's embedding space by a two-layer MLP. The language model utilizes DeepSeekMoE, featuring Multi-head Latent Attention (MLA) to compress the Key-Value (KV) cache, improving inference speed and throughput. The MoE architecture further enhances efficiency. A global bias term is used during MoE training for load balancing. DeepSeek-VL2 comes in three variants (Tiny, Small, and Base) with 1.0B, 2.8B, and 4.5B activated parameters, respectively. The training data is constructed in three stages: (1) VL alignment, (2) VL pretraining, and (3) supervised fine-tuning (SFT). The alignment stage uses ShareGPT4V (1.2M samples). Pretraining data combines VL and text-only data (70/30 ratio), including interleaved image-text data (WIT, WikiHow, OBELICS, Wanjuan, and in-house data), image captioning data (various open-source datasets with quality enhancements and filtering), OCR data (LaTeX OCR, 12M RenderedText, and in-house data), general VQA data, table/chart/document understanding data (PubTabNet, FinTabNet, Docmatix), web-to-code and plot-to-Python data (Websight, and Python plots), QA with visual prompts, visual grounding data and grounded conversation data. SFT data includes enhanced general visual question-answering data, cleaned OCR and document understanding data, enhanced table and chart understanding data, improved reasoning/logic/math data, textbook/academic questions, and expanded web-to-code and plot-to-Python data, visual grounding data, grounded conversation data. Text only datasets were used during SFT stage. The training methodology involves a three-stage pipeline. Stage 1 trains the vision encoder and vision-language adaptor MLP, keeping the language model fixed, using image-text paired data. Stage 2 performs vision-language pre-training with all parameters unlocked, using ~800B image-text tokens. Stage 3 conducts supervised fine-tuning. Visual understanding is emphasized, and the loss is computed only on text tokens. Unlike previous work, the fixed-resolution vision encoder is adapted during training.

Architecture: Maya builds upon the LLaVA 1.5 framework. It uses the Aya-23 8B model as its Large Language Model (LLM) due to Aya's strong multilingual capabilities (trained on 23 languages). Critically, it replaces the CLIP vision encoder used in LLaVA with SigLIP. This is motivated by SigLIP's superior performance, multilingual support, and ability to handle variable-length image patches (allowing for more flexible input sizes). The visual features from SigLIP (Zv = g(Xv)) are passed through a trainable projection matrix (W, a 2-layer MLP with GELU activation) to align them with the LLM's embedding space, producing visual features Hv. The architecture is fairly standard for this type of model, concatenating visual and textual features for input to the LLM. The training process involves two main phases: pretraining and finetuning. Pretraining: The model is pretrained on a newly created multilingual image-text dataset. This dataset is derived from the English-only LLaVA pretraining dataset (558k image-text pairs) and translated into seven additional languages (Chinese, French, Spanish, Russian, Hindi, Japanese, and Arabic) using a sophisticated translation pipeline. This pipeline uses the Aya 35B model, optimized prompt engineering (determined empirically using BLEU and N-gram scores), and a batch processing approach with quality checks. Crucially, this dataset undergoes toxicity filtering. LLaVAGuard and Toxic-BERT are used to identify and remove toxic image-caption pairs, creating a "toxicity-free" version of the dataset (removing 7,531 toxic images). The pretraining uses a learning rate of 1e-3 and a cosine scheduler. Only the projection matrix is trained during pretraining. Finetuning: The pretrained model is then instruction-tuned using the PALO 150K instruction-tuning dataset (which covers 10 languages). Full finetuning is performed (as opposed to LoRA), with frozen vision encoder and LLM. The core alignment technique is the trainable projection matrix (the 2-layer MLP) that maps the SigLIP visual features into the embedding space of the Aya-23 LLM. This is a simple but effective method, common in many VLMs. The paper explicitly states they did not use more complex alignment techniques like gated soft-attention (Flamingo) or Q-Former (BLIP-2) in this phase, reserving those for future work. Pretraining Dataset: A new multilingual dataset created by translating and filtering the LLaVA pretraining dataset. This dataset is a key contribution of the paper. The translation process and toxicity filtering are described in detail. Instruction Tuning Dataset: PALO 150K instruction-tuning dataset. Evaluation Datasets: PALO multilingual evalution, VizWiz, GQA, ScienceQA, TextVQA, POPE, MMBench, MM-Vet, MME. Multilingual Image-Text Pretraining Dataset: A new dataset of 558,000 images in eight languages. Toxicity Analysis and Mitigation: A thorough analysis of toxicity in the original LLaVA dataset and the creation of a toxicity-free version. This is a significant and novel aspect. Multilingual Model: A model (Maya) that shows improved performance in understanding cultural and linguistic nuances, especially in comparison to models trained primarily on English data. The results show that Maya performs comparably to or better than models of similar size (LLaVA-7B) and often approaches the performance of larger models (PALO-13B) on multilingual benchmarks. The toxicity filtering has a minimal impact on overall performance, suggesting that valuable information isn't lost by removing toxic content. The paper includes both quantitative benchmark results and qualitative examples demonstrating the model's capabilities.

InternVL 2.5 retains the "ViT-MLP-LLM" architecture of its predecessors, combining a pre-trained InternViT (either InternViT-6B or InternViT-300M) with LLMs of varying sizes (InternLM 2.5, Qwen 2.5) via a 2-layer MLP projector. A key feature is the pixel unshuffle operation, reducing visual tokens from 1024 to 256 per 448x448 image tile, improving scalability for high-resolution processing. The model architecture supports dynamic resolution, adapting to image aspect ratios by dividing images into 448x448 tiles. Crucially, InternVL 2.0 and 2.5 incorporate multi-image and video data, in addition to single-image and text-only data. The training strategy involves a three-stage pipeline: (1) MLP warmup, where only the MLP projector is trained, (2) optional ViT incremental learning, where the vision encoder and MLP are trained to enhance visual feature extraction, particularly for domains rare in web-scale data, and (3) full model instruction tuning, where the entire model is trained on high-quality multimodal instruction datasets. A progressive scaling strategy is employed, starting with smaller LLMs and scaling up, allowing for efficient alignment of the vision encoder with larger LLMs. Training enhancements include random JPEG compression (for robustness to real-world image quality) and loss reweighting (to balance contributions from responses of different lengths). Data organization is optimized using parameters like nmax (maximum tile number) and a repeat factor (r) to control data sampling frequency. A data-packing strategy concatenates multiple samples into longer sequences to improve GPU utilization. A significant contribution is a data filtering pipeline to remove low-quality samples, particularly those with repetitive patterns, mitigating the risk of repetitive generation, a common issue in MLLMs. The data mixture includes a wide range of tasks (captioning, general QA, mathematics, charts, OCR, etc.) and modalities (single-image, multi-image, video, text). The model was evaluated comprehensively on diverse benchmarks including multi-discipline reasoning (MMMU, MMMU-Pro), document understanding (DocVQA), multi-image/video understanding, real-world comprehension, multimodal hallucination detection, visual grounding, multilingual capabilities, and pure language processing.

PaliGemma 2 closely follows the architecture of its predecessor, PaliGemma. It uses a pre-trained SigLIP-So400m vision encoder. The embeddings from this encoder are mapped to the input space of the Gemma 2 language model using a linear projection. The combined visual and text embeddings are then fed into the Gemma 2 model, which autoregressively generates the output. The model comes in three size variants (2B, 9B, and 27B parameters in the Gemma 2 component, corresponding to 3B, 10B, and 28B total parameters) and is trained at three resolutions (224x224, 448x448, and 896x896 pixels). This allows for analysis of the interplay between model size, resolution, and transfer performance. The input image gets concatenated with the input text tokes and Gemma 2 autoregressively completes this prefix with an answer. PaliGemma 2's training follows a three-stage approach, similar to the original PaliGemma: Stage 1: The pre-trained SigLIP-So400m and Gemma 2 checkpoints are combined and trained jointly on a multimodal task mixture of 1 billion examples. The image resolution is 224px². Stage 2: Training continues for 50 million examples at 448px² resolution, then for 10 million examples at 896px². Tasks benefiting from higher resolution are upweighted. Stage 3: Fine-tuning the checkpoints from stage 1 or 2 on the target tasks. The training data mixture includes captioning, grounded captioning, OCR, visual question answering (VQA), detection, and instance segmentation. Notably, the training data relies heavily on machine-generated labels from publicly available specialist models, avoiding the use of large commercial VLMs for label generation. Gemma 2 Language Models: The core upgrade is the use of the more recent and capable Gemma 2 family of language models, replacing the original Gemma model in PaliGemma. Resolution and Model Size Scaling: PaliGemma 2 systematically explores the impact of both image resolution and language model size on transfer performance. This is a key contribution, as most prior work did not jointly study these factors with consistent training recipes.

ShowUI adapts Qwen2-VL-2B into an end-to-end GUI agent that predicts textual responses and executable actions from pixels. Its UI-Guided Visual Token Selection models screenshot patches as a connected graph, identifies visually redundant regions, and retains representative tokens during selected self-attention blocks. This exploits the repeated colors and structures common in interfaces: the paper reports removing about 33% of visual tokens during training and a 1.4× speedup without sacrificing grounding accuracy.

Training uses Interleaved Vision-Language-Action Streaming to express multiple GUI tasks in one causal format. For navigation, screenshots, instructions, and actions are arranged as temporal trajectories so the model can use visual-action history. For grounding, multiple query-action pairs can share a screenshot, improving data reuse and training efficiency. A resampling strategy prevents abundant task or interface categories from overwhelming smaller ones. The resulting model is trained on approximately 256K curated examples.

The instruction mixture covers web, mobile, and desktop interfaces and combines screenshot grounding with sequential navigation data. It incorporates carefully filtered public GUI sources and recaptioned annotations, then evaluates transfer on ScreenSpot-style grounding and on Mind2Web, AITW, and MiniWob navigation environments. The repository releases model weights, datasets, training and evaluation code, and the implementation of UI-guided token selection.

SmolVLM builds upon the architecture of Idefics3, leveraging a similar implementation in transformers but with key differences to enhance efficiency. It replaces the Llama 3.1 8B language backbone with the smaller SmolLM2 1.7B model. A more aggressive image compression strategy is employed, using a pixel shuffle strategy that reduces visual information by a factor of 9 (compared to 4x in Idefics3). This allows for 384x384 patches, and a shape-optimized SigLIP is used as the vision backbone with 14x14 inner patches. The model demonstrates superior memory usage compared to other VLMs in transformers, enabling efficient on-device inference. For instance, encoding a single image and prompt requires only 1.2k tokens, significantly less than models like Qwen2-VL. This efficiency translates to faster prefill and generation throughputs. SmolVLM achieves strong performance on benchmarks such as MMMU, MathVista, MMStar, DocVQA, and TextVQA. It also shows promising results in basic video analysis, leveraging its long context capabilities. Training involved extending the context window of SmolLM2 to 16k tokens using techniques like RoPE base value adjustment and fine-tuning on a mixture of long- and short-context datasets. A curated training dataset, largely based on The Cauldron and Docmatix, was used for the VLM training. Checkpoint selectio