MLLM-Semantic-Hallucination [NeurIPS 2025]

📄arXiv • 🤗HFPaper • 🤗TextHalu-Bench •

This repository provides the official PyTorch implementation of the following paper:

When Semantics Mislead Vision:Mitigating Large Multimodal Models Hallucinationsin Scene Text Spotting and Understanding
Yan Shu¹, Hangui Lin², Yexin Liu³, Yan Zhang^4,5, Gangyan Zeng⁶, Yan Li³, Yu Zhou⁷, Ser-Nam Lim⁸, Harry Yang², Nicu Sebe^1
1University of Trento (UNITN), ²University of International Relations (UIR), ³The Hong Kong University of Science and Technology (HKUST), ⁴Institute of Information Engineering, Chinese Academy of Sciences (IIE, CAS), ⁵University of Chinese Academy of Sciences (UCAS), ⁶Nanjing University of Science and Technology (NJUST), ⁷Nankai University (NKU), ⁸University of Central Florida (UCF)

Overview

Large Multimodal Models (LMMs) have achieved impressive progress in visualperception and reasoning. However, when confronted with visually ambiguous ornon-semantic scene text, they often struggle to accurately spot and understand thecontent, frequently generating semantically plausible yet visually incorrect answers,which we refer to as semantic hallucination. In this work, we investigate the un-derlying causes of semantic hallucination and identify a key finding: Transformerlayers in LLM with stronger attention focus on scene text regions are less prone to producing semantic hallucinations. Thus, we propose a training-free semantic hal-lucination mitigation framework comprising two key components: (1) ZoomText,a coarse-to-fine strategy that identifies potential text regions without external detec-tors; and (2) Grounded Layer Correction, which adaptively leverages the internalrepresentations from layers less prone to hallucination to guide decoding, correct-ing hallucinated outputs for non-semantic samples while preserving the semanticsof meaningful ones. To enable rigorous evaluation, we introduce TextHalu-Bench,a benchmark of over 1,730 samples spanning both semantic and non-semanticcases, with manually curated question–answer pairs designed to probe model hallu-cinations. Extensive experiments demonstrate that our method not only effectivelymitigates semantic hallucination but also achieves strong performance on publicbenchmarks for scene text spotting and understanding.

Setup

Qwen2.5-VL

We follow the official inplement of Qwen2.5-VL and replace the transformer use following code.

cd MLLM-Semantic-Hallucination/Qwen
pip install .

MiniMonkey

We follow the official inplement of MiniMonkey and download their official weight. Then we replace their code modeling_internlm2.py and modeling_minimonkey_chat.py with code here

LLaVA-NeXT

We follow the steps below for inplement of LLaVA-NeXT .

cd llava-next
conda create -n llava python=3.10 -y
conda activate llava
pip install --upgrade pip  # Enable PEP 660 support.
pip install -e ".[train]"
pip install .

Quickstart

After setup the environment, you can directly use our method on MLLMs model by:

Qwen2.5-VL

from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
from qwen_vl_utils import process_vision_info


model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    "/scqian/Qwen2.5-VL-3B-Instruct", torch_dtype="auto", device_map="auto"
)


processor = AutoProcessor.from_pretrained("/scqian/Qwen2.5-VL-3B-Instruct")
image_path="your/path/to/image"
question ="your question"
messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": image_path,
            },
            {"type": "text", "text": question},
        ],
    }
]

# Preparation for inference
text = processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
    text=[text],
    images=image_inputs,
    videos=video_inputs,
    padding=True,
    return_tensors="pt",
)
inputs = inputs.to(model.device)


# Inference: Generation of the output

image_token_id = 151655
idx = (inputs.input_ids == image_token_id).nonzero(as_tuple=True)
first_token_idx = idx[1][0].item()
last_token_idx = idx[1][-1].item()
generated_ids = model.generate(
    **inputs,
    max_new_tokens=128,
    img_token_idx=first_token_idx,
    qs_token_idx=last_token_idx
)


generated_ids_trimmed = [
    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]

output_text = processor.batch_decode(
    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text)

MiniMonkey

We use their original official demo.

LLaVA-NeXT

We use their original official demo.

Evaluation

TextHalu-Bench

• We have applied the test code to VLMEvalKit.You can test our benchmark using following code：

git clone https://github.com/open-compass/VLMEvalKit.git
conda activate your-env
cd VLMEvalKit
pip install -e .
python run.py  --data  TextHaluBench  --model Qwen2.5-VL-3B-Instruct  --verbose

STVQA

• Download the STVQA annotations file in here and evaluate it using the your inference outcome file:

python eval/eval_stvqa.py

TextVQA OCRVQA SEEDBench AI2D

We also use VLMEvalKit to evaluate the TextVQA OCRVQA SEEDBench AI2D.

Experiment's Results

Citation

If you find this work useful for your research, please cite our paper:

@article{shu2025semantics,
  title={When Semantics Mislead Vision: Mitigating Large Multimodal Models Hallucinations in Scene Text Spotting and Understanding},
  author={Shu, Yan and Lin, Hangui and Liu, Yexin and Zhang, Yan and Zeng, Gangyan and Li, Yan and Zhou, Yu and Lim, Ser-Nam and Yang, Harry and Sebe, Nicu},
  journal={arXiv preprint arXiv:2506.05551},
  year={2025},
}