Awesome paper for multi-modal llm with grounding ability
21
25 commits
updated Oct 11, 2025
README
Awesome-Multimodal-Large-Language-Models-With-Grounding
A curated list of Multimodal Large Language Models (or Large Vision Language Model) with grounding ability.
Table of Contents
🔥 Large Vision-Language Model
Grounding
| Format | Desc | Paper |
|---|---|---|
| Decoder on latent | leverage a decoder to ground | PerceptionGPT, NExT-Chat, PSALM, PixelLM, u-LLaVA, GSVA, ChatterBox, GLaMM |
| Output numerical coordinates | direct output numerical tokens | Shikra, VisionLLM, Ferret, Ferret2, CogVLM |
| Output token coordinates | output new tokens added to refer positions | Kosmos-2 |
| Pixel space | output in discrete pixel space encoded by VQGAN | Unified-IO, Unified-IO 2 |
| Proposal retrieval | retrieval from region candidates | LLM-Seg, Kosmos-2, GROUNDHOG |
Referring
| Format | Desc | Paper |
|---|---|---|
| Pooling | Leverage Mask Pooling / RoI Pooling / RoI Align to obtain features from the im encoder output | Groma, GPT4RoI, Osprey, PSALM, GROUNDHOG, Ferret, Ferret2, PVIT, ChatterBox, GLaMM |
| Numerical coordinates | Leverage numerical coordinates for referring (bbox / sampled points in mask) | Shikra, PerceptionGPT (w/ encoder), NExT-Chat (w/ encoder), CogVLM |
| Token coordinates | Add new tokens to vocab to present spatial positions | Kosmos-2 |
- w/ encoder: refers to using a encoder to encode the input coordinates.
Training Dataset
| Dataset | Source | Data Source | Quantity | Cnstruction Method |
|---|---|---|---|---|
| GRIT | Ferret | COYO-700M, LAION-2B | - | |
| Shikra-RD | Shikra | Flickr30K Entities | 5,922 QA pairs | ChatGPT4 ==> Referential Dialogue (CoT dialogues with grounding & referring) |
| CB-300K | ChatterBox | VG | 717,075 QA pairs | 4 subsets. |
| GranD | GLaMM | SA-1B | 11M images with 7.5M unique concepts and 810M regions. | Automated annotation pipeline with SAM for dense pixel-wise grounding. Used for pretraining. |
| GranD-f | GLaMM | GranD (refined), Flickr30K, RefCOCOg, and PSG | ~214K image-grounded text pairs | Refined subset of GranD for fine-tuning, with 1000 images held out for human-annotated evaluation |
Training Recipe
| Model | Recipe |
|---|---|
| Ferret | |
| Ferret2 | |
| ChatterBox | Trainable: LoRA and location decoder |
| GPT4RoI | |
| GLaMM |
Evaluation Dataset
| Dataset | Source | Data Source | Quantity | Cnstruction Method |
|---|---|---|---|---|
| Ferret Bench | Ferret | COCO validation set | 120 |
Paper List
GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest
- propose referring for mllm by replacing placeholder <region_i> by feature obtained by mask pooling
Osprey: Pixel Understanding with Visual Instruction Tuning
- similar to GPT4RoI, Osprey also use mask representation to refer to entities in images.
- It uses mask pooling to extract semantic features from image encoder and combines with a location extractor to process the mask and output spatial token.
VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks
- unified interface for vision and vl tasks: points for detection, sample points for instance seg ==> instruction format for training
- extra tokens & output-format-as-query to decode (faster)
Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks
- creates a unified IO for all sorts of vision and vl task (into discrete tokens)
- using t5-like encoder-decoder arch
Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action
- following Unified-IO v1, creates a unified IO for all sorts of modalities including image, masks, bboxes, audios (into discrete tokens)
- dense masks are all binary, unlike v1 which specifies the color in text instruction (model struggles to follow)
- propose 2D Rotary Embedding, QK Normalization and Scaled Cosine Attention to stabilize training and scaling
- Mixture of Denoisers taining objectives
- instruction tuning of 220 tasks drawn from over 120 external datasets
PixelLM: Pixel Reasoning with Large Multimodal Model
- learnable seg tokens + light-weight decoder
- a bunch of tricks:
- N x L seg tokens for L level multi-scale vision features. N tokens within each group for better modeling
- reweighted loss on regions with overlapping predictions
PSALM: Pixelwise SegmentAtion with Large Multi-Modal Model
- new paradigm: first generate mask proposal, then genereate mask and classification (following mask2former)
- instruction prompt + conditional prompt + candidate masks token
- three types of conditional prompt: classes, sentence (ref seg) and visual cues (point, scribbles, boxes, etc)
- conditional prompt => condition embed, candidate masks token => mask embed.
- condition embed +mask embed + image feature => mask2former decoder => bipartite matching loss + query-based decoding
LLM-Seg: Bridging Image Segmentation and Large Language Model Reasoning
- Use SAM to generate mask candidates, then fomulate the problem as mask selection (mask classification)
- promote LLM-Seg40K dataset, by using LLaVA to generate caption, then GPT4 to generate question-answer pair.
GROUNDHOG: Grounding Large Language Models to Holistic Segmentation
- disantengle grounding with referring
- grounding as mask selection and train a mask2former+ to generate mask candidates
- referring by mask pooling on feature
- promote 2.5M M3G2 dataset
DetGPT: Detect What You Need via Reasoning
- Follow LLaVA to tune VLM and for vqa
- Use grouding DINO to ground response generated by VLM to detect the relevantg entities.
Ferret: Refer and Ground Anything Anywhere at Any Granularity
- propose hybrid region representation for referring : region name + coordinates + mask pooled feature by Spatial-aware visual sampler
- grounding through bbox
Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Models
- propose a bunch of improvements on Ferret v1
- including any-resolution (patches) for larger resolution
- DINOv2 Encoder for local feature extraction
- and High-resolution Dense Alignment stage between SFT and instruction turning.
u-LLaVA: Unifying Multi-Modal Tasks via Large Language Model
- propose to use different decoder for grounding (SAM for segmentation, Grounding DINO for detection)
GSVA: Generalized Segmentation via Multimodal Large Language Models
- propose to Generalized Referring Expression Segmentation (GRES) in grounding LLM
- multiple object to ground
- need to reject null target
- propose to use multple [SEG] token to ground multiple objects (indicted by the texts before the [SEG] token), and [REJ] token to rej null target
NExT-Chat: An LMM for Chat, Detection and Segmentation
- propose box encoder-decoder for referring and grounding
- for grounding, use token to indicate the presence of a grounding output and input the latent embedding to the box decoder (mask decoder e.g. SAM) for box (mask) generation
- for referring, use boxes to represent referred region and use box encoder to encode the referred boxes into features, which is input to LLM.
- propose a cycle consistency loss for regularization of box encoder-decoder
PerceptionGPT: Effectively Fusing Visual Perception into LLM
- similar to NExT-Chat, propose box encoder-decoder to encode and decode boxes, but seems to only focus on grounding without referring
- One possible intriguing point: grounding output indicator <vis> is used to indicate the presence of grounding output (as usual) but the is replaced by the encoder's output feature in the LLM input.
Kosmos-2: Grounding Multimodal Large Language Models to the World
- build a web-scale grounding dataset by web-scale data (COYO-700M & LAION-2B etc) and vision detector (GLIP)
- following pix2seq, divide the image into PxP grids and introduce PxP new tokens to represent
- Use <box></box> to represent a bbox, with <delim> to separate multiple boxes (if there are multiple boxes)
- Use markdown-like grammar to reference grounded text with <p> </p>
e.g.
Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic
- propose to use normalized boxes for unified grounding and referring
- Use texts to represent all normalized boxes (directly tokenized by text tokenizer) and input to LLM
Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Models
- Propose to ground and refer with a set of proposed regions.
- Change a Deformable DETR detection head into binary classifier to propose ROI and use AlignROI pooling to get the region feature
LISA: Reasoning Segmentation via Large Language Model
- Introduce the reasoning segmentation task and establish a reasoning segmentation benchmark.
- Propose LISA model, which represents the segmentation mask as an embedding and incorporates new segmentation capabilities.
GLaMM: Pixel Grounding Large Multimodal Model
- Introduces Grounded Conversation Generation (GCG) task combining phrase grounding, referring expression segmentation, and vision-language conversations.
- Proposes a scalable pipeline to curate GranD (Grounding-anything Dataset) with 7.5M unique concepts grounded in 810M regions with segmentation masks, 214k GranDf and a ～1000 evaluation set.
- Architecture: Global Image Encoder for holistic understanding, Region Encoder with RoI pooling for regions referring, LLM generates responses with grounding tokens , and use Pixel Decoder (SAM) to decode segmentation masks from tokens' latent
🔥 Multi-modality
Awesome paper for multi-modal llm with grounding ability
21
25 commits
updated Oct 11, 2025
README
Awesome-Multimodal-Large-Language-Models-With-Grounding
A curated list of Multimodal Large Language Models (or Large Vision Language Model) with grounding ability.
Table of Contents
🔥 Large Vision-Language Model
Grounding
| Format | Desc | Paper |
|---|---|---|
| Decoder on latent | leverage a decoder to ground | PerceptionGPT, NExT-Chat, PSALM, PixelLM, u-LLaVA, GSVA, ChatterBox, GLaMM |
| Output numerical coordinates | direct output numerical tokens | Shikra, VisionLLM, Ferret, Ferret2, CogVLM |
| Output token coordinates | output new tokens added to refer positions | Kosmos-2 |
| Pixel space | output in discrete pixel space encoded by VQGAN | Unified-IO, Unified-IO 2 |
| Proposal retrieval | retrieval from region candidates | LLM-Seg, Kosmos-2, GROUNDHOG |
Referring
| Format | Desc | Paper |
|---|---|---|
| Pooling | Leverage Mask Pooling / RoI Pooling / RoI Align to obtain features from the im encoder output | Groma, GPT4RoI, Osprey, PSALM, GROUNDHOG, Ferret, Ferret2, PVIT, ChatterBox, GLaMM |
| Numerical coordinates | Leverage numerical coordinates for referring (bbox / sampled points in mask) | Shikra, PerceptionGPT (w/ encoder), NExT-Chat (w/ encoder), CogVLM |
| Token coordinates | Add new tokens to vocab to present spatial positions | Kosmos-2 |
- w/ encoder: refers to using a encoder to encode the input coordinates.
Training Dataset
| Dataset | Source | Data Source | Quantity | Cnstruction Method |
|---|---|---|---|---|
| GRIT | Ferret | COYO-700M, LAION-2B | - | |
| Shikra-RD | Shikra | Flickr30K Entities | 5,922 QA pairs | ChatGPT4 ==> Referential Dialogue (CoT dialogues with grounding & referring) |
| CB-300K | ChatterBox | VG | 717,075 QA pairs | 4 subsets. |
| GranD | GLaMM | SA-1B | 11M images with 7.5M unique concepts and 810M regions. | Automated annotation pipeline with SAM for dense pixel-wise grounding. Used for pretraining. |
| GranD-f | GLaMM | GranD (refined), Flickr30K, RefCOCOg, and PSG | ~214K image-grounded text pairs | Refined subset of GranD for fine-tuning, with 1000 images held out for human-annotated evaluation |
Training Recipe
| Model | Recipe |
|---|---|
| Ferret | |
| Ferret2 | |
| ChatterBox | Trainable: LoRA and location decoder |
| GPT4RoI | |
| GLaMM |
Evaluation Dataset
| Dataset | Source | Data Source | Quantity | Cnstruction Method |
|---|---|---|---|---|
| Ferret Bench | Ferret | COCO validation set | 120 |
Paper List
GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest
- propose referring for mllm by replacing placeholder <region_i> by feature obtained by mask pooling
Osprey: Pixel Understanding with Visual Instruction Tuning
- similar to GPT4RoI, Osprey also use mask representation to refer to entities in images.
- It uses mask pooling to extract semantic features from image encoder and combines with a location extractor to process the mask and output spatial token.
VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks
- unified interface for vision and vl tasks: points for detection, sample points for instance seg ==> instruction format for training
- extra tokens & output-format-as-query to decode (faster)
Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks
- creates a unified IO for all sorts of vision and vl task (into discrete tokens)
- using t5-like encoder-decoder arch
Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action
- following Unified-IO v1, creates a unified IO for all sorts of modalities including image, masks, bboxes, audios (into discrete tokens)
- dense masks are all binary, unlike v1 which specifies the color in text instruction (model struggles to follow)
- propose 2D Rotary Embedding, QK Normalization and Scaled Cosine Attention to stabilize training and scaling
- Mixture of Denoisers taining objectives
- instruction tuning of 220 tasks drawn from over 120 external datasets
PixelLM: Pixel Reasoning with Large Multimodal Model
- learnable seg tokens + light-weight decoder
- a bunch of tricks:
- N x L seg tokens for L level multi-scale vision features. N tokens within each group for better modeling
- reweighted loss on regions with overlapping predictions
PSALM: Pixelwise SegmentAtion with Large Multi-Modal Model
- new paradigm: first generate mask proposal, then genereate mask and classification (following mask2former)
- instruction prompt + conditional prompt + candidate masks token
- three types of conditional prompt: classes, sentence (ref seg) and visual cues (point, scribbles, boxes, etc)
- conditional prompt => condition embed, candidate masks token => mask embed.
- condition embed +mask embed + image feature => mask2former decoder => bipartite matching loss + query-based decoding
LLM-Seg: Bridging Image Segmentation and Large Language Model Reasoning
- Use SAM to generate mask candidates, then fomulate the problem as mask selection (mask classification)
- promote LLM-Seg40K dataset, by using LLaVA to generate caption, then GPT4 to generate question-answer pair.
GROUNDHOG: Grounding Large Language Models to Holistic Segmentation
- disantengle grounding with referring
- grounding as mask selection and train a mask2former+ to generate mask candidates
- referring by mask pooling on feature
- promote 2.5M M3G2 dataset
DetGPT: Detect What You Need via Reasoning
- Follow LLaVA to tune VLM and for vqa
- Use grouding DINO to ground response generated by VLM to detect the relevantg entities.
Ferret: Refer and Ground Anything Anywhere at Any Granularity
- propose hybrid region representation for referring : region name + coordinates + mask pooled feature by Spatial-aware visual sampler
- grounding through bbox
Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Models
- propose a bunch of improvements on Ferret v1
- including any-resolution (patches) for larger resolution
- DINOv2 Encoder for local feature extraction
- and High-resolution Dense Alignment stage between SFT and instruction turning.
u-LLaVA: Unifying Multi-Modal Tasks via Large Language Model
- propose to use different decoder for grounding (SAM for segmentation, Grounding DINO for detection)
GSVA: Generalized Segmentation via Multimodal Large Language Models
- propose to Generalized Referring Expression Segmentation (GRES) in grounding LLM
- multiple object to ground
- need to reject null target
- propose to use multple [SEG] token to ground multiple objects (indicted by the texts before the [SEG] token), and [REJ] token to rej null target
NExT-Chat: An LMM for Chat, Detection and Segmentation
- propose box encoder-decoder for referring and grounding
- for grounding, use token to indicate the presence of a grounding output and input the latent embedding to the box decoder (mask decoder e.g. SAM) for box (mask) generation
- for referring, use boxes to represent referred region and use box encoder to encode the referred boxes into features, which is input to LLM.
- propose a cycle consistency loss for regularization of box encoder-decoder
PerceptionGPT: Effectively Fusing Visual Perception into LLM
- similar to NExT-Chat, propose box encoder-decoder to encode and decode boxes, but seems to only focus on grounding without referring
- One possible intriguing point: grounding output indicator <vis> is used to indicate the presence of grounding output (as usual) but the is replaced by the encoder's output feature in the LLM input.
Kosmos-2: Grounding Multimodal Large Language Models to the World
- build a web-scale grounding dataset by web-scale data (COYO-700M & LAION-2B etc) and vision detector (GLIP)
- following pix2seq, divide the image into PxP grids and introduce PxP new tokens to represent
- Use <box></box> to represent a bbox, with <delim> to separate multiple boxes (if there are multiple boxes)
- Use markdown-like grammar to reference grounded text with <p> </p>
e.g.
Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic
- propose to use normalized boxes for unified grounding and referring
- Use texts to represent all normalized boxes (directly tokenized by text tokenizer) and input to LLM
Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Models
- Propose to ground and refer with a set of proposed regions.
- Change a Deformable DETR detection head into binary classifier to propose ROI and use AlignROI pooling to get the region feature
LISA: Reasoning Segmentation via Large Language Model
- Introduce the reasoning segmentation task and establish a reasoning segmentation benchmark.
- Propose LISA model, which represents the segmentation mask as an embedding and incorporates new segmentation capabilities.
GLaMM: Pixel Grounding Large Multimodal Model
- Introduces Grounded Conversation Generation (GCG) task combining phrase grounding, referring expression segmentation, and vision-language conversations.
- Proposes a scalable pipeline to curate GranD (Grounding-anything Dataset) with 7.5M unique concepts grounded in 810M regions with segmentation masks, 214k GranDf and a ～1000 evaluation set.
- Architecture: Global Image Encoder for holistic understanding, Region Encoder with RoI pooling for regions referring, LLM generates responses with grounding tokens , and use Pixel Decoder (SAM) to decode segmentation masks from tokens' latent