Awesome MLLM Uncertainty

:star::star::star: If you find this repo useful, please star it!

Discussion

🏄🏄🏄 Welcome to join our MLLM uncertainty discussion group (the left QR code)! Or add my WeChat (the right QR code) to enter the group if the group QR code expires~

🔥 News

• 2025.3: 🔥🔥🔥 Check out our latest work: Uncertainty-o, unveiling uncertainty in Large Multimodal Models (LMMs) in a model-agnostic manner, supporting both Large Comprehension Models and Large Generation Models.
• 2025.1: 🔥🔥🔥 Check out our VL-Uncertainty, leveraging semantic-equivalent perturbation for refined LVLM uncertainty estimation!

Awesome List

• PC-SGG Conformal Prediction and MLLM aided Uncertainty Quantification in Scene Graph Generation (18 Mar 2025, CVPR 2025)

• SRICE Seeing and Reasoning with Confidence: Supercharging Multimodal LLMs with an Uncertainty-Aware Agentic Framework (11 Mar 2025)

• Uncertainty-o Uncertainty-o: One Model-agnostic Framework for Unveiling Epistemic Uncertainty in Large Multimodal Models (11 Mar 2025)

• HEIE HEIE: MLLM-Based Hierarchical Explainable AIGC Image Implausibility Evaluator (26 Nov 2024)

• Calibration-MLLM Unveiling Uncertainty: A Deep Dive into Calibration and Performance of Multimodal Large Language Models (19 Dec 2024, COLING 2025)

• LAP LAP, Using Action Feasibility for Improved Uncertainty Alignment of Large Language Model Planners (9 Dec 2024)

• DropoutDecoding From Uncertainty to Trust: Enhancing Reliability in Vision-Language Models with Uncertainty-Guided Dropout Decoding (9 Dec 2024)

• IDK I Don't Know: Explicit Modeling of Uncertainty with an [IDK] Token (9 Dec 2024, NeurIPS 2024)

• BayesVLM Post-hoc Probabilistic Vision-Language Models (8 Dec 2024)

• Verb Mirage Verb Mirage: Unveiling and Assessing Verb Concept Hallucinations in Multimodal Large Language Models (6 Dec 2024)

• PUNC Towards Understanding and Quantifying Uncertainty for Text-to-Image Generation (4 Dec 2024)

• UA-CLM Enhancing Trust in Large Language Models with Uncertainty-Aware Fine-Tuning (3 Dec 2024)

• VL-Uncertainty VL-Uncertainty: Detecting Hallucination in Large Vision-Language Model via Uncertainty Estimation (18 Nov 2024)

• 4DGS 4D Gaussian Splatting in the Wild with Uncertainty-Aware Regularization (13 Nov 2024, NeurIPS 2024)

• SUM Uncertainty-aware Fine-tuning of Segmentation Foundation Models (6 Nov 2024, NeurIPS 2024)

• MUB Exploring Response Uncertainty in MLLMs: An Empirical Evaluation under Misleading Scenarios (5 Nov 2024)

• CrossPred-LVLM Can We Predict Performance of Large Models across Vision-Language Tasks? (14 Oct 2024)

• TRON Sample then Identify: A General Framework for Risk Control and Assessment in Multimodal Large Language Models (10 Oct 2024)

• Reference-free Hallucination Detection for Large Vision-Language Models (11 Aug 2024, EMNLP 2024 Findings)

• MLLM-CompBench MLLM-CompBench: A Comparative Reasoning Benchmark for Multimodal LLMs (23 Jul 2024, NeurIPS 2024)

• Semantic Entropy Detecting Hallucinations in Large Language Models Using Semantic Entropy (19 Jun 2024, Nature)

• UAL Uncertainty Aware Learning for Language Model Alignment (7 Jun 2024, ACL 2024)

• MOMBO Deterministic Uncertainty Propagation for Improved Model-Based Offline Reinforcement Learning (6 Jun 2024, NeurIPS 2024)

• HIO Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced Optimization (24 May 2024, NeurIPS 2024)

• Overconfidence is Key: Verbalized Uncertainty Evaluation in Large Language and Vision-Language Models (5 May 2024, TrustNLP 2024))

• Consistency and Uncertainty Consistency and Uncertainty: Identifying Unreliable Responses From Black-Box Vision-Language Models for Selective Visual Question Answering (16 Apr 2024, CVPR 2024)

• UPD Unsolvable Problem Detection: Evaluating Trustworthiness of Vision Language Models (29 Mar 2024)

• ICD Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding (27 Mar 2024, ACL 2024 Findings)

• The First to Know The First to Know: How Token Distributions Reveal Hidden Knowledge in Large Vision-Language Models? (14 Mar 2024, ECCV 2024)

• VLM-Uncertainty-Bench Uncertainty-Aware Evaluation for Vision-Language Models (22 Feb 2024)

• LogicCheckGPT Logical Closed Loop: Uncovering Object Hallucinations in Large Vision-Language Models (18 Feb 2024, ACL 2024 Findings)

• UQ_ICL Uncertainty Quantification for In-Context Learning of Large Language Models (15 Feb 2024, NAACL 2024)

• IntroPlan Introspective Planning: Aligning Robots' Uncertainty with Inherent Task Ambiguity (9 Feb 2024, NeurIPS 2024)

• LLM-Uncertainty-Bench Benchmarking LLMs via Uncertainty Quantification (23 Jan 2024, NeurIPS 2024 Datasets & Benchmarks)

• CD-CCA Cloud-Device Collaborative Learning for Multimodal Large Language Models (26 Dec 2023, CVPR 2024)

• VCD Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding (28 Nov 2023, CVPR 2024 Highlight)

• LURE Analyzing and Mitigating Object Hallucination in Large Vision-Language Models (1 Oct 2023, ICLR 2024)

• PAU Prototype-based Aleatoric Uncertainty Quantification for Cross-modal Retrieval (29 Sep 2023, NeurIPS 2023)

• KnowNo Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners (4 Jul 2023, CoRL 2023, Best Student Paper)

• ProbVLM ProbVLM: Probabilistic Adapter for Frozen Vision-Language Models (1 Jul 2023, ICCV 2023)

• GAVIE Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning (26 Jun 2023, ICLR 2024)

• Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs (22 Jun 2023, ICLR 2024)

• POPE Evaluating Object Hallucination in Large Vision-Language Models (17 May 2023, EMNLP 2023)

• UQ-NLG Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models (30 May 2023, TMLR)

• Semantic Uncertainty Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation (19 Feb 2023, ICLR 2023 Spotlight)

• MAP MAP: Multimodal Uncertainty-Aware Vision-Language Pre-training Model (11 Oct 2022, CVPR 2023)