AnomalyThink-LLaVA-OneVision-7B-SFT

The plain supervised baseline of this family. LLaVA-OneVision-7B-SI fine-tuned on 6,000 teacher-distilled reasoning traces, nothing else.

A LLaVA-OneVision-7B-SI model fine-tuned for explainable industrial anomaly detection (IAD). Given one product image it writes a reasoning trace in <think>, then a defect <location> and <type> when it sees a defect, and finally a binary <answer>. Research artefact from the MSc thesis Reasoning-Enhanced Vision-Language Models for Explainable Industrial Anomaly Detection (TU Delft, 2026).

This checkpoint is the starting point for the other two models in the family. It is the GRPO initialisation, and it produced the rollouts that seeded the KCR corpus. Publish it mostly so those two are reproducible from a fixed init.

Results

Metric is balanced accuracy, (TPR + TNR) / 2, in percent.

Reference rows measured on the exact same harness:

Evaluation protocol. One shared harness for every row above. The DS-MVTec and VisA subsets of MMAD, single image per prompt, the same instruction the model was trained on, greedy decoding at temperature 0, at most 1024 new tokens, images capped at 262,144 pixels. Answers are parsed from the <answer> tag. This particular row was generated through the plain transformers generate path. The vLLM 0.10.2 path used for the sibling models agreed with it on 99 percent of a probe set. Nothing here is a re-scored or best-of-N number.

Strict scoring: a generation with no parsable verdict counts as wrong, as in the thesis. Six of the 2,141 VisA generations of the SFT then GRPO model have no verdict, which gives 72.38; over the 2,135 parsable outputs it is 72.58. No ordering changes.

Contamination note, please read this before you compare DS-MVTec numbers

The public LLaVA-OneVision training mixture (lmms-lab/LLaVA-OneVision-Data, config vision_flan(filtered)) contains 426 rows whose id matches %MVTecAD%. The base model has therefore seen MVTec-AD material during its own instruction tuning. Every DS-MVTec number for any LLaVA-OneVision derived model carries that caveat, including the 85.91 above, and including the IAD-R1 row. We do not know how much of the gap is real capability and how much is recall.

VisA is not affected. The same query over the mixture returns 0 rows for VisA. So the 68.26 on VisA is the clean number and it is the one to trust for a cross-model comparison.

Training

• Base model: llava-hf/llava-onevision-qwen2-7b-si-hf.
• Corpus: anomalythink_6k/combined_6k_train.json, 6,000 AnomalyThink traces distilled from Gemini 2.5-Flash on Real-IAD images. This is the same corpus as the Qwen2.5-VL headline SFT model, which is the point. It isolates the backbone as the only thing that changed.
• Recipe: supervised fine-tuning only. No reinforcement learning stage. The SigLIP vision tower is frozen, the multimodal projector and the language model are trained. Learning rate 1e-5, cosine schedule, warmup ratio 0.03, weight decay 0.1, effective batch size 32, context cutoff 8,192 tokens, bf16, 4 epochs.
• Epoch: this is epoch 1 of 4 (step 188), and it is the best of the four saved epochs by a clear margin. The full epoch curve on DS-MVTec / VisA was 85.91 / 68.26, 81.95 / 64.35, 83.13 / 66.94, and 82.78 / 67.44. Longer training on 6,000 traces makes this backbone worse, not better.

The KCR corpus in this family is LLaVA native, not borrowed from Qwen

This model is not trained on a KCR corpus, it is trained on the Gemini 2.5-Flash traces. The note still belongs here, because this checkpoint is where the KCR loop starts.

KCR stands for keep, correct, rewrite. The KCR corpus used by the sibling KCR model was built from LLaVA-OneVision's own rollouts, not from the Qwen rollouts used elsewhere in the thesis. GRPO was run on this checkpoint, that GRPO model produced 10,236 rollouts on Real-IAD, the rollouts were bucketed into keep, correct, and rewrite, and a Gemini 2.5-Flash judge then repaired the failing ones. The result is on-policy for this backbone, and that matters. The control, which is the Qwen derived KCR corpus trained on this same backbone, peaked at 87.70 on DS-MVTec and 73.52 on VisA in different epochs. The native corpus reaches 88.45 and 74.25 in one checkpoint.

All three arms of that corpus are published at llava_kcr/: sft_llava_A_kept.json, sft_llava_B_kept_corrected.json, and sft_llava_C_train.json.

Tested transformers versions

The weights and configs in this repo were written by transformers 4.51.3. Loading, the processor, and the full evaluation were verified under transformers 4.57.1, both through the plain HF generate path and through vLLM 0.10.2.

One warning for anyone rebuilding this pipeline. Checkpoints saved by transformers 5.0 write the rope settings under text_config.rope_parameters. Transformers 4.x does not read that key and silently falls back to rope_theta = 10000, which is 100 times too small. The model then stays fluent but goes blind and answers "no" to nearly everything, which looks like a collapsed run rather than a loading bug. This repo was written by 4.51.3 so it was never affected, and it carries a plain text_config.rope_theta = 1000000.0 like the rest of the family.

Usage

import torch
from PIL import Image
from transformers import AutoProcessor, LlavaOnevisionForConditionalGeneration

repo = "aacudad/AnomalyThink-LLaVA-OneVision-7B-SFT"
model = LlavaOnevisionForConditionalGeneration.from_pretrained(
    repo, torch_dtype=torch.bfloat16, device_map="auto")
processor = AutoProcessor.from_pretrained(repo)

image = Image.open("part.png").convert("RGB")
product = "tile"          # the product category, it goes into the prompt

question = (
    f"Analyze the provided image of the {product}. "
    "Determine if there are any anomalies present. "
    "If an anomaly is detected, specify its type and location, "
    "and provide a detailed reasoning for your conclusion."
)
messages = [{"role": "user", "content": [
    {"type": "image"},
    {"type": "text", "text": question},
]}]
prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(images=image, text=prompt, return_tensors="pt").to(
    model.device, torch.bfloat16)

out = model.generate(**inputs, max_new_tokens=1024, do_sample=False)
print(processor.decode(out[0][inputs["input_ids"].shape[1]:],
                       skip_special_tokens=True))

Use this exact instruction. The model was trained on it and it degrades on a different phrasing.

Expected output on a defective part:

<think>
I am inspecting a tile with a speckled, grayish-white surface. ... In the center of the
tile, I detect a triangular, translucent plastic fragment. ...
</think>
<location>center</location>
<type>Contamination</type>
<answer>Yes</answer>

On a normal part the model emits <think> and then <answer>No</answer>, with no <location> or <type> tag.

Intended use and limitations

Research on explainable industrial anomaly detection. This is a thesis artefact, not a production inspection system. If you only want the best detector, take the KCR model instead.

Known limitations:

• The DS-MVTec contamination caveat above.
• Recall is the weak side of this checkpoint. On VisA it misses far more defects than it false-alarms, which is what the two later models fix.
• The model can write a confident and well argued trace for a defect that is not there.
• The <type> label is coarse and the model over-uses "missing parts" for any loss of material, including chips and gouges.
• It was trained on Real-IAD style single-object images on plain backgrounds. Cluttered scenes, multiple parts per image, and very different lighting are out of distribution.
• Reasoning traces were distilled from a teacher model. A fluent trace is not proof that the model looked at the right pixels.

Citation

@mastersthesis{acudad2026reasoning,
  author  = {Acudad, A.},
  title   = {Reasoning-Enhanced Vision-Language Models for Explainable Industrial Anomaly Detection},
  school  = {Delft University of Technology},
  year    = {2026},
  type    = {Master's thesis},
  url     = {https://resolver.tudelft.nl/uuid:65c62420-79c0-447f-b095-7fb11d4474fc}
}

Thesis: https://resolver.tudelft.nl/uuid:65c62420-79c0-447f-b095-7fb11d4474fc. Code and evaluation files: https://github.com/aacudad/IAD-VLMs. The training data is at aacudad/AnomalyThink.

Related models

• AnomalyThink-LLaVA-OneVision-7B-KCR
• AnomalyThink-LLaVA-OneVision-7B-SFT-GRPO
• AnomalyThink-Qwen2.5-VL-7B-SFT, the same corpus and recipe on the Qwen2.5-VL backbone

License

Apache-2.0, inherited from the LLaVA-OneVision-7B-SI base. Trained on Real-IAD images, which are not redistributed here, so cite Real-IAD separately. Reasoning traces were distilled from Gemini 2.5-Flash.