query_form_experiment

The code and the evidence behind Your Queries Can Reorder the Leaderboard.

I explored what happens to a retrieval leaderboard when you change only the form of the queries, holding the corpus and the relevance labels fixed. This repository is the harness that ran the experiment and the per-query score files it produced. If you have just read the post and want to see where a number came from, the table below is the index. Every number recomputes from a clone, offline, with nothing paid:

uv sync
uv run python scripts/05_analyze.py        # the reproduction gate, model by model
uv run python tools/transform_analysis.py  # reshuffle tables, controls, levelling

Where the post's numbers come from

Figures live in docs/figures/. Regenerate them with:

uv run python tools/chart_data.py chart_data.json
uv run python tools/make_post_figures.py chart_data.json docs/figures

What was run

Three datasets: ChatDoctor marks one relevant document per query, CUREv1 about forty, and TREC-COVID several hundred, graded 0/1/2. Each dataset's queries were rewritten three ways (paraphrase, terse, verbose) by four LLMs from four families. The rewriters worked independently, so if only one of them produces an effect it shows up as such.

Twenty-one embedding models. All 21 ran on CUREv1 and TREC-COVID. 17 of them ran on ChatDoctor, which is why its board is shorter. Retrieval is turbopuffer, dense by default, with BM25 and hybrid RRF available as controls.

The harness also carries AILAStatutes and HumanEval as pilots. They were just used to explore before moving on to the larger corpora, and their runs are still in results/, but the post does not use them.

The reproduction gate

The first task was to tune the harness so that it could reproduce each model's published NDCG@10 on the original queries from each same dataset.

81 of 86 pairs match within 0.01. The three TREC-COVID misses are all bge-*-en-v1.5 prefix models and they all miss low by a similar amount, so the ordering survives - my TREC-COVID board reproduces the published one at τ = 1.000, an exact order match. The five prefix-free models on that dataset land within 0.003. I think the bge gap is a prompting difference in how the published run handled those models, but I have not closed it and I don't want to explain it away, so I'll keep it on the books as a miss.

Leaderboard reshuffle

Kendall's τ between each transformed-query leaderboard and the same dataset's original human-query leaderboard, dense retrieval, averaged over the four generators. This is the post's headline table:

Averaging hides whether the generators agree, so the per-generator numbers follow, with RBO (p = 0.9) alongside. Board size differs by dataset because not every model has a published anchor on every one.

ChatDoctor, 17-model board, 5,591 queries:

CUREv1, 21-model board, 2,000 queries:

TREC-COVID, 21-model board, 50 topics. With only 50 queries, τ carries a seeded 1,000-resample query-level bootstrap CI, and I only count a τ below 0.9 if its interval excludes 0.9:

I see a pattern across the three tables. Verbose is the gentlest condition on ChatDoctor and the harshest on the two densely-labelled datasets, and all four generators agree on that ordering, so I don't read it as one rewriter's quirk. This isn't quite a controlled ablation though since the datasets differ in more than their label density.

The board reordered by elaboration

TREC-COVID, 21 models, ranked by the benchmark's own questions and then by the generator-averaged verbose rewrite of those same questions. This is Figure 1:

Three models gain three or more places and four lose three or more. qwen3-embedding-8b goes 15th to 1st. Five of the original top six move down.

Human phrasings: the control on LLM style

Every rewrite above was written by an LLM, so the obvious objection is that the result is about how language models write. Since each TREC-COVID topic also ships both of a human-written keyword query and a paragraph-length narrative alongside the question the benchmark scores, I can use these as a check without involving LLMs.

τ against the question board, over the 20 models that have all three human phrasings (bge-m3 has no human-field run):

A human-written paragraph reorders the board at τ = 0.726 with no change in meaning and no LLM involved. The LLM rewrites are each somewhat harsher than the human form they imitate, so my read is that they overstate the effect a little without creating it. The two extremes, keyword against narrative, sit at +0.7158, no further apart than either is from the middle form, which makes me doubt that length alone explains this.

The LLM τ values above are the mean of the four per-generator τ values on that 20-model board, which is why they differ slightly from the 21-model figures in the reshuffle tables.

Comparing boards is one thing. Matching the shifts model by model is stricter:

Keyword compression is well mimicked. Elaboration is only partly mimicked - sign agreement near chance means the LLM elaboration and a human's both reshuffle hard but land the per-model effect differently. That is a caveat on the verbose numbers specifically and it does not touch the terse ones.

Levelling

Elaboration costs the strongest models the most and lifts several of the weakest. Going the other way, compressing narrative back to question improves 19 of the 20 models with human phrasings, by an average of +0.1098 NDCG@10.

Correlating a change score against the same baseline it was computed from invites regression to the mean, so I also measured strength on a separate dataset and report both readings:

From what I can see, some of the effect is that artifact and most of it isn't, at least on TREC-COVID, where the independent measure holds at -0.779. On CUREv1 it doesn't reach significance either way. So I wouldn't state this as a general law - it is one dataset's result, with a second dataset declining to confirm it.

Controls not in the post

Two further checks, both out of tools/transform_analysis.py, on whether the verbose result is a property of the queries or an artifact of something else.

The first is BM25, which isolates surface-word change since a lexical scorer only sees tokens. ΔNDCG@10 against the human-query BM25 run, averaged over the four generators:

Verbose leaves BM25 near flat on all three while moving the dense board hard. I read that as semantic broadening - the tokens barely change, but the meaning the dense models see does. Terse on TREC-COVID is the opposite case. BM25 climbs 0.097 there because the rewrite really is more keyword-like.

The second is a cross-lineage LLM audit, with the judge never drawn from the rewriter's own family, flagging rewrites that changed the underlying information need. Pooled drift rate against a 15% threshold, n = 120 sampled queries per cell (50 on TREC-COVID):

Verbose breaches the threshold, so drift is a live worry, but it is generator-split. gpt-5.1 and llama drift hard on verbose (52.5% and 56.7% on CUREv1) while claude (6.0-7.5%) and gemini (12.6-14.9%) elaborate faithfully. The faithful lineages still move the dense board about as far as the drifting ones - claude's verbose gives τ 0.857 on CUREv1 and 0.695 on TREC-COVID, against 0.962 and 0.905 for its own paraphrase. So far as I can tell the effect survives the drift control.

Per-generator rates are in results/audit__{dataset}__{condition}.json.

Data

results/ contains 1,386 files - every number from the experiment recomputes without re-running any pipelines. The dense__* and bm25__* files carry run metadata and a per-query map of qid to NDCG@10. The queries__* files carry the generated rewrites themselves, and the audit__* files carry the drift judgments.

Running it

uv sync
uv run pytest        # unit tests; integration tests are deselected by default

uv run python scripts/01_build_index.py   --dataset aila_statutes --model gemini-embedding-001
uv run python scripts/03_run_retrieval.py --dataset aila_statutes --model gemini-embedding-001 --mode dense

--mode also takes bm25 and hybrid; --condition and --generator select a rewrite. scripts/05_analyze.py and tools/transform_analysis.py take dataset names as positional arguments and default to chatdoctor curev1_en trec_covid.