ActDiff

The Illusion of Expertise: Narrow Finetuning Induces Persistent Domain Priors in Vision–Language Models

Narrow-domain finetuning makes a vision–language model sound like a domain expert while quietly making it less attentive to the image. ActDiff diagnoses, repairs, and prevents the persistent domain prior behind this illusion of expertise — for any base VLM and its finetuned counterpart.

What it does

Point ActDiff at a base model and its narrowly finetuned version and get the full loop:

• Diagnose — extract the persistent domain-prior trace (the mean base→finetuned shift in assistant-token hidden states), and check that it survives blank / noise images.
• Measure — quantify the illusion as an image-grounding sensitivity gap: how much less the finetuned model is disrupted than the base when the image is destroyed.
• Repair — subtract the trace direction at inference to restore visual grounding, with a peak-and-collapse dose response.
• Prevent — add a one-line first-moment regulariser at training time.

Install

conda env create -f environment.yml && conda activate actdiff
pip install -e .                    # or: pip install -e ".[reproduction]" for the paper pipeline

Version note. transformers >= 5.5 requires huggingface_hub >= 1.5 (older versions drop is_offline_mode and transformers fails to import). requirements.txt pins this; if you hit the error, run pip install -U "huggingface_hub>=1.5".

Use it on your own model

from actdiff import ActDiff

ad = ActDiff.from_pretrained(
    base="Qwen/Qwen2.5-VL-3B-Instruct",
    finetuned="AdaptLLM/biomed-Qwen2.5-VL-3B-Instruct",   # id / path / LoRA adapter
)

samples = [{"image": img, "prompt": "...", "response": "..."}, ...]   # your own data

trace = ad.extract_trace(samples)          # diagnose
print(trace.summary())                     #   -> peak layer, persistence under blank/noise
print(ad.sensitivity_gap(samples))         # measure the illusion (nats)
print(ad.dose_response(samples, trace))    # causal repair dose-response curve

with ad.repair(trace, alpha=0.6):          # repair at inference (no retraining)
    print(ad.generate("What is shown?", img))

Command line:

actdiff diagnose --base <id> --finetuned <id> --data data.jsonl --out trace.pt
actdiff gap      --base <id> --finetuned <id> --data data.jsonl
actdiff dose     --base <id> --finetuned <id> --data data.jsonl --trace trace.pt

Training-time prevention is a drop-in transformers.Trainer subclass:

from actdiff.train import MuShiftTrainer
trainer = MuShiftTrainer(..., base_model=base, anchor_layer=L, lambda_mu=1.0)

What the paper found

• A persistent, image-agnostic prior. The base→finetuned trace is compact, late-layer, early-position, and just as strong on a blank image as on the real one.
• The illusion of expertise. On a corruption-sensitivity probe the finetuned model is less disturbed by a destroyed image than the base — a 0.323 ± 0.036 nat gap on Biomed.
• A causal handle, no retraining. Subtracting the trace recovers 54% of the Biomed MMStar collapse, with a peak-and-collapse dose response that random / wrong-domain controls do not produce; it transfers to a second VLM family with the signature and fails on a third without it.
• A one-line mitigation. A single first-moment regulariser shrinks the trace by 20–64× across three AdaptLLM domains while improving generic NLL.

Models: Qwen2.5-VL-3B (main) + Qwen2-VL-2B, LLaVA-NeXT-Llama3-8B, Llama-3.2-11B-Vision; domains: AdaptLLM Biomed, Food, Remote Sensing.

Repository

src/actdiff/     the ActDiff tool (diagnose / measure / repair / train + CLI)
src/diffing/     the engine — a fork of diffing-toolkit with our
                 methods/vlm_activation_difference_lens/ (native-multimodal ADL)
examples/        end-to-end "diagnose your own model" scripts
tests/           unit tests
reproduction/    the paper: experiment runners + result CSVs
                 -> reproduction/REPRODUCE.md (every experiment's exact command)
                 -> reproduction/DATA.md      (checkpoints / datasets to download)

Citation

The paper is accepted to EMNLP 2026 (Main). A BibTeX entry will be added once it is published in the proceedings (or on arXiv).

Acknowledgments

Built on the diffing-toolkit by Julian Minder and Clément Dumas (science-of-finetuning), MIT-licensed: https://github.com/science-of-finetuning/diffing-toolkit. ActDiff extends their Activation Difference Lens to the multimodal setting; if you use this code, please also cite Narrow Finetuning Leaves Clearly Readable Traces in Activation Differences (arXiv:2510.13900).

License

MIT — see LICENSE.