---

CADIF is a placebo-controlled control-task battery for instruction-following retrieval. Benchmarks such as FollowIR credit an embedder for demoting the documents a narrowed instruction makes irrelevant, scored by a paired metric (p-MRR). But instruction-tuned need not mean instruction-following: because the instruction is concatenated query-side, a gain can come from two confounds rather than comprehension, its topical words inflating an nDCG-style gain (lexical expansion), or its demotion rewarding topic drift.

CADIF removes both with one within-model contrast: the correct instruction scored against its own word-shuffled twin.

Delta_comp(q) = pMRR_real(q) - pMRR_placebo(q)

A bag-of-words ranker is invariant to the shuffle, so BM25 scores Delta_comp = 0 exactly. That algebraic zero anchors a floor no lexical or drift confound can lift. A system whose Delta_comp clears the floor has demonstrably used word order, not just the instruction's vocabulary.

The result in one figure

Every system on two axes: horizontal, the demotion score a leaderboard sees; vertical, the de-confounded contrast that survives the placebo. They come apart. A model-free 200-line keyword rule (green) sits at the same high leaderboard score as the strongest trained follower yet at Delta_comp = 0: it games the metric but cannot pass the control.

On FollowIR's 104-query census, floor-clearance coincides with instance-level instruction training: four systems from three independent lineages (two Promptriever bi-encoders, FollowIR-7B, and the ContextualAI reranker at 1B/2B/6B) clear the floor, while six general-purpose instruction embedders sit inside it.

What is here

Results and per-query arrays (every arm, contrast, and checkpoint for every model) live in the companion Hugging Face dataset, not in this repository: tardellirs/cadif-followir-results. The base FollowIR and mFollowIR corpora are loaded from their original sources, not redistributed here.

Quickstart

pip install -r requirements.txt

# Run the de-confounded battery for one model over the FollowIR English collections.
python cloud/run_cadif.py \
  --model intfloat/multilingual-e5-large-instruct \
  --collections robust04,news21,core17 \
  --max-pool 1000 --control none --checkpoint --outdir results

# The FLIP meaning-inversion crossover for a cross-encoder reranker (N=27 audited pairs):
PYTHONPATH=. python cloud/run_flip_reranker.py \
  --model ContextualAI/ctxl-rerank-v2-instruct-multilingual-2b \
  --flips-file followir_flips_curated.json --max-pool 100 --outdir results

Two models in the architecture panel (NV-Embed-v2, gte-Qwen2) need a transformers==4.44.2 environment; see the note in requirements.txt.

Reproducing the headline numbers

Every arm, contrast, CI, p-value, and per-query array in the paper is produced by cloud/run_cadif.py (battery) and cloud/run_flip_reranker.py (FLIP) and released on the Hugging Face dataset above. A model's verdict is mean(real_minus_placebo) over the released per_query arrays; cadif/stats.py reproduces the bootstrap CI, randomization p, TOST, and power from them:

from huggingface_hub import hf_hub_download
import json, numpy as np

f = hf_hub_download("tardellirs/cadif-followir-results",
                    "ctxl-rerank-v2-instruct-multilingual-6b_marginal_robust04-news21-core17_pool1000.json",
                    repo_type="dataset")
dcomp = np.array([q["real_minus_placebo"] for q in json.load(open(f))["per_query"]])
print(f"Delta_comp = {dcomp.mean():+.4f}")   # +0.0720  ->  clears the floor

Citation

Until the accompanying paper is publicly available, please cite this software archive. The per-query results have their own DOI (10.57967/hf/9547). Both entries will be superseded by the article citation once it appears.

@software{cadif2026,
  title     = {CADIF: A Composition Assay for De-confounding Instruction-Following},
  author    = {Stekel, Tardelli},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.21297047},
  url       = {https://doi.org/10.5281/zenodo.21297047}
}

License

Code is released under the MIT License (LICENSE). The author-constructed data in data/ is released under CC-BY-4.0. Third-party model weights and the FollowIR corpora retain their own licenses (the ContextualAI reranker, for example, is CC-BY-NC-SA-4.0).