CADIF Results: De-confounded Instruction-Following Retrieval

Per-query results, contrasts, and run checkpoints for CADIF (a Composition Assay for De-confounding Instruction-Following), a control-task battery for instruction-following retrieval. A paper describing CADIF is in preparation.

CADIF asks whether an instruction-conditioned retriever actually uses an instruction's composition, or only exploits the topical words it adds. The headline readout is a within-model contrast, the correct instruction scored against its own word-shuffled twin:

Delta_comp(q) = pMRR_real(q) - pMRR_placebo(q)

A bag-of-words ranker is invariant to the shuffle, so BM25 scores Delta_comp = 0 exactly. That algebraic zero anchors a floor that neither lexical expansion nor topic drift can lift. This dataset releases the per-query arrays behind every number in the paper, so each verdict is checkable rather than trusted.

Code: the battery that produced these files is at github.com/tardellirs/cadif, archived on Zenodo (10.5281/zenodo.21297047). Base corpora: the FollowIR and mFollowIR query/document pairs are not redistributed here; they are loaded from their original sources (Weller et al., 2024, 2025).

What is in here

Per-query schema (*_marginal_*.json)

{
  "model": "...", "pool": 1000, "n": 104,
  "aggregate": { "pmrr": [...], "follow_gain": [...] },
  "per_query": [
    {
      "id": "robust04:305", "n_pool": 100, "n_cut": 26,
      "pmrr_real": 0.0093,      // correct-instruction demotion
      "pmrr_placebo": ...,      // word-shuffled twin
      "pmrr_swap": ...,         // wrong-instruction (drift confound)
      "real_minus_placebo": ..., // Delta_comp  <-- the headline
      "real_minus_swap": ...,
      "ndcg_og": ..., "ndcg_changed": ...
      // graded runs add: pmrr_surface, real_minus_surface
    }
  ]
}

The paper's verdict for a model is mean(real_minus_placebo) with its paired bootstrap CI, sign-flip randomization p, TOST equivalence, and power. All of those are reproduced from the per_query arrays by cadif/stats.py in the code repository.

Loading

from huggingface_hub import hf_hub_download
import json, numpy as np

repo = "tardellirs/cadif-followir-results"
f = hf_hub_download(repo, "ctxl-rerank-v2-instruct-multilingual-6b_marginal_robust04-news21-core17_pool1000.json",
                    repo_type="dataset")
d = json.load(open(f))
dcomp = np.array([q["real_minus_placebo"] for q in d["per_query"]])
print(f"Delta_comp = {dcomp.mean():+.4f}  (n={len(dcomp)})")   # +0.0720  -> clears the floor

Coverage

• Collections: FollowIR English (Robust04, News21, Core17; 104-query census) and mFollowIR (Chinese, Russian, Persian). The excluir_cadif_*.json files hold the cross-benchmark replication on ExcluIR (Zhang et al., 2024; 1036 exclusionary Wikipedia-QA items), where both a bi-encoder and a cross-encoder follower clear the same algebraic floor (BM25 Delta_comp = 0 exact) on a corpus and construction independent of FollowIR.
• Pool depths: 50 / 100 / 1000.
• Systems: the four trained followers (two Promptriever backbones, FollowIR-7B, and the ContextualAI reranker at 1B / 2B / 6B), six general-purpose instruction embedders, the matched non-instruction bases, BM25, and the model-free symbolic follower.

License and citation

Released under CC-BY-4.0. The underlying model weights and the FollowIR/mFollowIR corpora retain their own licenses.

Until the accompanying paper is publicly available, please cite this dataset (this entry will be superseded by the article citation once it appears):

@misc{cadif2026,
  title        = {CADIF: A Composition Assay for De-confounding Instruction-Following},
  author       = {Stekel, Tardelli},
  year         = {2026},
  publisher    = {Hugging Face},
  doi          = {10.57967/hf/9547},
  url          = {https://doi.org/10.57967/hf/9547}
}