jev-rerank-bench

The write-up: Jev vs Cohere Rerank 4 Pro vs zerank-2: reranker benchmark walks through the results, the negation test and the Qwen follow-up.

I gave TypeSafe's Jev thirty search results and asked it which ones were useful. Then I gave Cohere and ZeroEntropy the same passages. This repository contains the experiments, saved responses and scoring code.

The ranking average put Jev's rubric at 0.692 and Cohere Pro at 0.691, without establishing a winner. Giving every query equal weight instead puts Cohere ahead. Jev did better on the negation test than the rerankers; the open-weight Open-Jev 9B, run afterwards, reads negation better than Jev asked the same way (77% vs 71% of pairs). An open-source Qwen recipe improved substantially when I gave it one passage at a time, but still showed no clear gain over keyword ranking. Eight more open models, run September 25, all beat keyword ranking; the best of them, Qwen3-Reranker-4B, is within noise of Jev asked one passage at a time (0.660 vs 0.670), trails Jev's rubric, and beats every Jev setup on French.

The main comparison covers eight English datasets. Five more BRIGHT subsets, NevIR negation pairs and MIRACL French are reported separately. Models start with the same thirty BM25 candidates, cut to 2,000 characters each; the duel variant only compares the first ten, and NevIR supplies its own two-passage pairs.

Measurements began September 16, 2026; the Qwen controls were added September 17. Jev calls used jev-latest, reporting version 1.13.0. The evidence viewer includes the original benchmark and the Qwen follow-up, with individual questions, scores and saved outputs.

Two follow-ups: Does the order of the passages change Jev's pick? reverses, shuffles and renames the thirty passages, and prompt-injection-vs-keyword-stuffing-ai-seo adds one sentence to a wrong page and checks whether 13 rankers put it in their top five (100 searches).

Headline (8 English datasets, 1,617 scored questions)

Self-hosted rows ran on graphics cards rented from RunPod (secure cloud); "Runs on" names the cards. Their cost is GPU time, not an API price: the seconds each call kept the card busy, times that card's hourly price, divided by the number of requests the card was working on at once (model copies × requests in flight per copy). Setup and idle time are left out. Each dataset counts once averages the eight dataset scores (the headline figure); each question counts once averages all 1,617 questions, so FiQA's 411 count for more than BRIGHT's 39 and 38. A dash in the negation column means that setup was not run on NevIR. Want another model here? Test your own model, or request one.

• Ranking quality: Jev rubric minus Cohere Pro is +0.001, with a 95% interval of −0.009 to +0.012. This establishes neither a winner nor equivalence. With equal weight per query, Cohere scores 0.756 and Jev 0.738.
• Top-ranked passage: Jev Choice leads Cohere Pro by 3.1 percentage points (95% interval +0.7 to +5.6). This metric ignores the separate none option; it is not the accuracy of the answer Jev actually selected.
• Negation (NevIR, 1,383 pairs): Jev rubric 71% of pairs right, Cohere Pro 67% (gap +4.2, range +1.5 to +6.9), ZeroEntropy 61%; the chat-model baseline 17–22%, below the 25% of guessing.
• Reasoning (7 BRIGHT subsets, 307 questions): Jev rubric averages 0.493 and DeepSeek JSON 0.487; their difference remains unresolved. These subjects are a separate comparison, not added to the eight-dataset headline.
• Qwen's one-passage control reaches 0.471, up from 0.255 with thirty yes/no fields. BM25 scores 0.486; the Qwen-minus-BM25 interval crosses zero. Details and attribution are below.
• Dataset differences: Cohere Pro leads Jev's yes/no batch by 5.0 points on FiQA, its rubric by 3.9 on Natural Questions, and its Choice setup by 6.4 on French. zerank-2 leads the rubric by 2.1 points on TREC-COVID.

The intervals are exploratory paired bootstraps, without adjustment for the multiple comparisons. NevIR kept passage order fixed and reuses source passages across some pairs. Its 25% chance rate assumes independent random choices for the two questions.

Table latency is the mean of dataset medians, not a pooled median. API calls include network and serving time; self-hosted timings are model calls on rented GPUs. Costs average the eight dataset rates over all 2,327 original queries, including queries without relevant candidates. API costs use saved usage and recorded rates; self-hosted costs cover measured GPU time, excluding setup and idle time. These are not invoice-verified or equivalent deployment costs.

Full tables, per-dataset numbers, latency percentiles, usage totals and every check: results/summary.md, results/*.json, and the charts in results/.

The Qwen follow-up

Harsha Gundala's original recipe uses MLX on Apple Silicon. I tested Shreyansh Singh's Transformers/PyTorch port on rented RTX 4090s. The base model is Qwen2.5-1.5B-Instruct, from Alibaba's Qwen team. Despite “1B” in the recipe's name, it is a 1.5-billion-parameter model. The port adds serving code and presets, with no additional fine-tuning; it does not reproduce Jev's training or establish that their architectures match.

The tested run_parallel_generation processes the shared prompt once, then reuses its internal cache to evaluate the fields in a batch. It scores the permitted labels and assembles JSON in code. Prefill and field evaluation are separate model forwards, and some labels need continuation scoring. I did not test the port's separate tree mode. Allowed types and normalized label scores do not guarantee correct judgments or calibrated confidence.

With thirty passages together, Qwen scored 0.255 using yes/no fields and 0.340 using the rubric, both below BM25's 0.486. Jev's rubric led Qwen's rubric by 35.2 points (95% interval +33.2 to +37.4). Giving Qwen one passage per prompt raised it to 0.471: +21.6 points over its thirty-passage yes/no setup, but no demonstrated improvement over BM25 (difference −0.015; 95% interval −0.033 to +0.004). NevIR paired accuracy was 7% for the batch yes/no configuration, 19% for the rubric and 34% for the one-passage configuration.

Reversing the thirty passages changed Qwen's top-ranked passage on 1,498 of 1,617 queries (92.6%), compared with 400 (24.7%) for Jev Choice. As above, these are passage rankings, excluding Jev's none option. No repeated identical Qwen requests were run. This measures order sensitivity, not repeatability or proof that position alone determines the answer. The one-passage control also changes context length and prompt structure; it does not isolate parallel decoding as the cause of the quality difference.

rlcd_check.py provides the score diagnostics: the relevant-minus-irrelevant yes/no score gap was 0.046 for Qwen and 0.430 for Jev with the same wording; Qwen's mean score was 0.195 at slot 1 and 0.384 at slot 30. Under reversal, mean absolute score changes were 0.160 for Qwen and 0.013 for Jev Choice. Those last two values use different score scales—independent yes/no scores versus a distribution across choices—so they are not a relative stability measure.

This tests the port's ranking quality. It does not reproduce the original quantized Mac demo or test its advertised JSON-generation speedup.

What is measured

• Ranking: nDCG@10, Top-1, Recall@5 and MRR@10 over questions whose BM25 top-30 holds at least one labelled relevant passage. nDCG@10 rewards useful passages near the top; 0.69 does not mean 69% of questions answered correctly. It uses linear relevance gains and all supplied relevance labels for the ideal ranking. Ties keep BM25 order; the report also breaks ties against each model as a sensitivity check.
• Latency: client-observed API timings at the stated concurrency, plus Qwen model-call timings on rented GPUs. network.py records connection and server-header diagnostics; it does not isolate equivalent model-only times across providers. Cohere used OpenRouter, and ZeroEntropy served many calls in its slower fallback mode.
• Cost: saved usage multiplied by the recorded rates, or OpenRouter's returned usage.cost. These fields were not checked against invoices. The Qwen estimate covers measured GPU time only.
• The "nothing relevant" test: for every question, a twin list with every relevant passage removed and refilled from further down BM25. AUROC of the top score, and the false-accept rate at 90% recall. For Jev also its built-in none option and its "does any passage answer it?" question.
• Calibration: ECE and reliability curves for probability-like scores. Normalizing scores does not establish that their confidence values match observed error rates.
• Uncertainty: significance.py uses 10,000 paired bootstrap resamples of queries within each dataset, then averages across the fixed dataset set. An interval crossing zero is inconclusive, not evidence of equivalence.
• Jev under the microscope: TypeSafe's confidence bands against real answers, order sensitivity (same 30 passages reversed), repeated requests and cold start after 1–15 minutes idle. These repeat tests were on Jev, not Qwen.
• Shared text: batching.py records one request per batch size with the same thirty SciFact passages. Each query adds two typed questions and roughly 443 input tokens. Forty queries cost 2.7 times one query in those observations. Only the first query's choice was tracked; complete batched answers were not retained. This does not demonstrate quality across all forty answers or equivalent-quality savings over separate requests.
• Negation: nevir_eval.py scores NevIR by paired accuracy: both questions must rank their correct passage strictly higher; ties fail. Fixed passage order, repeated source passages and unadjusted comparisons limit inference.

Datasets

* questions whose BM25 top-30 holds at least one relevant passage.

Candidates. BM25 (bm25s, Snowball stemmer, language stopwords) top 30 per question over the whole corpus (for MIRACL, over the 100-passage pool MTEB ships per query). Models share the candidate lists and 2,000-character truncation; the duel's top-ten restriction and NevIR's two-passage pairs are listed separately. These are results for this candidate-generation pipeline, not official full-benchmark scores. Truncation can remove relevant text, and the source relevance labels may be incomplete.

Models and how each is asked

The yes/no variants of Jev, DeepSeek and Qwen get the same wording: "Does the passage contain the information needed to answer or verify the query?" (rerankers/__init__.py). Cohere and ZeroEntropy take the query and the documents.

Open weights, self-hosted: Laya, GLiNER2.5 and Open-Jev (runs 2026-09-19 and 2026-09-22, their own block)

Laya (Convai Innovations, 421M, Apache 2.0) is the open-weight model with Jev's question shape: Choice, Score and yes/no answered with probabilities. Its window is 512 tokens, so it ran one passage per prompt (the only shape it allows), with Jev's 4-level rubric and with Jev's yes/no wording, on rented GPUs (mostly RTX 4090s; also RTX 3090, A40 and L40S); the like-for-like Jev row is "Jev yes/no per pair" (0.670). The three GLiNER2.5 models are span and label matchers, not built for this, listed for completeness. Rows copied from the render; never averaged into the headline.

• Laya rubric vs BM25: gap -0.3 points, range -2.3 to +1.6, within noise. NevIR pairs right: 36% (Jev rubric 71%).
• Laya's 512-token window cut 0.3% of passages on Natural Questions and 88.9% on BRIGHT StackOverflow (the cut counts are in results/small_models_headline.txt).
• GLiNER2.5 multi could not process 1 of 7,896 lists on a 24 GB card (a 19,341-character question); it is stored as failed, not scored. GLiNER ran out of GPU memory on 139 code-heavy lists at 30 passages per batch; they were redone in batches of 6 or 1 (same scores, slower).
• torch 2.4's fused attention gave NaN scores for Laya on mixed-length batches; fixed by zeroing NaNs at padded positions after each layer; batched scores match the library's own predict() within 0.004 (English) and 0.012 (multilingual). Libraries: laya 0.3.3, gliner2 2.0.0.
• Runner: small_models_runner.py (same cache rows as run.py, GPU seconds × the pod's hourly price).

Open-Jev (Zefan Cai, github.com/Zefan-Cai/Open-Jev; MIT code, Apache 2.0 adapters) is a LoRA adapter plus a scalar head on Qwen3.5-2B and Qwen3.5-9B trained to answer the same three question types through the same request format as Jev. It ran through this repo's own Jev request builders pointed at a local copy of its server (JEV_URL), one passage per prompt, Jev's exact yes/no wording and criteria, on rented GPUs (2B: four server copies per card, on RTX 4090s for about 85% of calls and at the L40S price for the rest, going by the hourly rate saved with each call; 9B: L40S, A100, H100, RTX PRO 6000, two to four copies per card). Run 2026-09-22.

• Open-Jev 2B vs BM25: gap +5.8 points, range +3.6 to +7.9, real.
• Open-Jev 9B vs BM25: gap +11.4 points, range +9.3 to +13.5, real.
• Jev yes/no per pair vs Open-Jev 9B, same shape: gap +7.0 points, 95% range +5.6 to +8.5, real. NevIR pairs right: 2B 73%, 9B 77% (Jev yes/no per pair 71%).
• Time and cost: GPU wall time of each call with several server copies sharing one card, so the per-list time carries contention and the dollar figure divides the pod price by the requests in flight on the card (copies × 2); both are rough. Corrected 2026-09-25: the first version divided by the copies only, which charged each second of card time twice ($3.16 and $0.63 per 1,000 queries); openjev/fix_cost.py halved the saved costs. RunPod's bill for the nine 9B pods that day, setup and idle time included, was $17.05, less than the $17.84 the first charge put on the calls alone. Passages longer than the servers' 4,096-token window (10 lists per size, all in BRIGHT robotics and StackOverflow) were redone on a 16,384-token server; every response is in cache/open-jev-*.
• Scripts: openjev/pod.sh, openjev/fix.sh, openjev/merge.py; the run's own tables: results/eval_run_sept22.txt, results/significance_run_sept22.txt.
• Setup: pip install -e '.[train]' of Open-Jev, adapters ZefanCai/Open-Jev-2B (rev 0c7aa49) and Open-Jev-9B (rev 47e9668), python -m jev.server --checkpoint models/Open-Jev-<N>/package/checkpoint --max-length 4096, then JEV_URL=http://127.0.0.1:8791/v1/systemone JEV_MODEL=open-jev JEV_GPU_RATE=<pod $/h ÷ (copies × 2)> uv run run.py --model jev-noul-pair --cache-as open-jev-<n>-noul-pair --dataset all.

Open rerankers, two 4B models and three System One rebuilds, self-hosted (run 2026-09-25, their own block)

Three open rerankers, two small open models asked the yes/no question directly, and three community rebuilds of Jev that answer TypeSafe's System One protocol, on five rented RTX A5000 cards (RunPod secure cloud, $0.27 an hour), one model at a time per card, over all 14 datasets and NevIR, one passage per prompt. Their rows are in the main table.

• Every one of the eight beats BM25: from +10.1 points (decider-2b, range +8.3 to +12.0) to +17.4 (Qwen3-Reranker-4B, range +15.6 to +19.2), all real.
• Against Jev asked the same way (yes/no per pair, 0.670), Qwen3-Reranker-4B is within noise: Jev +1.0 points, range −0.2 to +2.3. Jev leads the other seven by 2.8 points (mxbai-rerank-base-v2, range +1.3 to +4.4) to 8.3 (decider-2b, range +6.9 to +9.8), all real. Jev's rubric (0.692) leads all eight; Qwen3-Reranker-4B by 3.2 points, range +2.0 to +4.5.
• MIRACL French: Qwen3-Reranker-4B has the top score there, level with Cohere Rerank 4 Pro (0.766 vs 0.764), and leads Jev's best French setup (Choice) by 6.6 points, range +3.3 to +10.1, real.
• NevIR pairs right: Winnow-12B 73%, Tev1-4B 71% (Jev yes/no per pair 71%), Qwen3.5-4B 62%, reflex 61%, decider-2b 56%, the rerankers 48% (Qwen3), 43% (bge) and 38% (mxbai). Open-Jev 9B stays highest at 77%.
• reflex 4B is Qwen3.5-4B read through reflex's prompt, each question in two option orders; it scores 0.622, against 0.628 for the same weights asked the plain yes/no question.
• The rerankers follow their model cards, pinned to exact revisions (inject/hf_runner.py). Before any benchmark question each scored its card's own printed example, and the run stopped unless the passages came out in the printed order with every score within 0.25 + 3% of the printed one: Qwen3-Reranker-4B gave 6.3125 and −14.3555 where its card prints 6.4375 and −14.375 (bf16), bge −8.1838 and 5.265 where its card prints −8.1875 and 5.26171875, mxbai 9.7507, 1.3697, 8.0808, 2.6612, 0.8730 and 0.3988 where its card prints 9.750735, 1.3697281, 8.080784, 2.6611633, 0.8729458 and 0.39884853.
• mixedbread's own mxbai-rerank library (0.1.0 to 0.1.6 alike) builds a shorter prompt than the card's Sentence Transformers recipe and scores the card's weaker passages up to 1.04 away from the printed numbers, so the card's recipe is the one run here. bge reads the model's whole 8,192-token window where the card's example stops at 512, so it sees the same text as every other model. The rerankers' scores are the logits their cards return by default: they order passages exactly as the probabilities do but never round to 1.0, so two confident passages do not tie.
• Qwen3.5-4B and Tev1-4B had only run in the prompt-injection test before. Both ran one prompt per forward pass: Qwen3.5 mixes linear-attention layers with full attention, and one prompt per pass leaves no padding for those layers to read.
• The rebuilds answer the same noul-pair request Jev got, through rerankers/systemone.py and their models.yaml entries; their "30 passages in one request" setups were not run. reflex and Winnow answer one request at a time (two in flight, the second waits its turn); decider packs up to 32 into one forward pass, which is why its time per list is long and its cost low.
• Not run: SemIf (github.com/TheoLeeCJ/SemIf-OpenJev) has no server, only a command-line scorer with its own input format. djev (github.com/Davipar/djev-dev) was not run for lack of time; anyone who wants it in the table is welcome to run it and open a pull request (Test your own model). Two notes for whoever does: its server takes the same request format at /v1/request, not /v1/systemone, so rerankers/systemone.py needs that path; and its docs recommend one NVIDIA B200 (the model's weights alone are about 52 GB).
• Time and cost: GPU wall time of each call; the dollar figure divides the card's $0.27 an hour by the requests in flight on it (1, 2 or 32, see Prices used). RunPod's bill for the whole step, setup, idle and a failed first attempt included: $10.07 (the five A5000 pods $9.71; four community-cloud pods and a test pod whose hosts' GPU drivers failed, $0.36).
• Scripts: pods/hf.sh, pods/systemone.sh, inject/hf_runner.py; the run's own tables: results/eval_run_sept25.txt, results/significance_run_sept25.txt.

Fairness rules

• Same candidate lists and truncation, with the duel and NevIR exceptions documented above; shared wording for the yes/no variants.
• Ties keep BM25 order for every model; eval.py also reports ndcg10_ties_against, tied_top_share, zero_score_share.
• The 8-dataset headline was fixed before the BRIGHT block and NevIR ran; they are reported separately, not averaged in.
• Public datasets may be in any model's training data. Stated, not fixable.
• API clients ran from Algeria; providers and routing paths differed. Qwen ran on rented GPUs.
• Ranking responses are in cache/, one JSONL line per query, gzipped. Additional diagnostics save the fields listed in their scripts.

Reproduce

uv sync
cp .env.example .env            # JEV_API_KEY, OPENROUTER_API_KEY, ZEROENTROPY_API_KEY
uv run candidates/build.py      # downloads the datasets, builds BM25 top-30 and the "absent" twins
uv run candidates/build_nevir.py
uv run run.py --model jev-score-batch --dataset all --workers 16
uv run run.py --model cohere-pro --dataset all --workers 12
uv run run.py --model zerank-2 --dataset all --workers 6
uv run eval.py                  # results/summary.md, results/summary.json, charts
uv run significance.py          # bootstrap ranges for every gap
uv run nevir_eval.py            # negation, paired accuracy
uv run rlcd_check.py            # signal and position drift of the Qwen RLCD scores
uv run network.py; uv run batching.py; uv run determinism.py; uv run coldstart.py 60 120 300 600 900

Runs resume: a query already in cache/ (plain or .gz) is skipped, and a failed row is redone on the next run. --limit N runs N uncached queries as a smoke test. To re-score saved results without model calls, run only eval.py, significance.py, nevir_eval.py and rlcd_check.py. The run.py, network, batching, determinism and cold-start commands above make new paid requests. Recorded API usage was about $61; that excludes GPU rental costs and has not been reconciled against invoices.

Test your own model

Any model served over TypeSafe's System One protocol can get a row in the main table, asked the way Jev was: the same thirty passages, the same words, the same scoring. No server? Request a model.

1. What your server must speak. POST <base_url>/systemone, JSON in and out, with an optional Authorization: Bearer <key> header. This is the whole request for one passage, the noul-pair setup:

{
  "state": {
    "query": "what causes ocean tides",
    "passage": "Tides are caused mainly by the Moon's gravity pulling on the oceans."
  },
  "model": "your-model-id",
  "questions": {
    "relevant": {
      "type": "noul",
      "instructions": "Does the passage contain the information needed to answer or verify the query?",
      "criteria": {
        "true": "The passage states or directly implies the answer to the query, or the evidence that verifies it.",
        "false": "The passage is only on a related topic; it does not supply what the query asks for."
      }
    }
  }
}

A reply the benchmark reads: Jev's to a SciFact question, from cache/jev-noul-pair/ (its model field left out). noul is the probability of true:

{"answers": {"relevant": {"type": "noul", "noul": 0.02}}, "usage": {"input_tokens": 594, "output_tokens": 22}}

The other setups send the same kind of request with more in it:

A server that holds one passage per request can still run noul-pair and score-pair. usage is optional: the cost comes from GPU time on a self-hosted card, or from its token counts times your price. A reply the adapter cannot read counts as a failed question, with the reason saved; it does not stop the run.

2. Add an entry to models.yaml. Its header explains every field: base_url, model, modes (the setups to run), max_context, passages_per_request, concurrency, and optionally key_env (the name of the .env entry with your key, never the key), price, runs_on and label. uv run run.py --model <name> --plan shows what will run: a setup that needs more passages or tokens in one request than your server takes runs its one-passage form.

3. Run it. Once: uv sync, then uv run candidates/build.py and uv run candidates/build_nevir.py to rebuild the passage texts, which are not in the repo; the committed lists in candidates/*.jsonl must come out unchanged (git status candidates/). Then:

uv run bench.py <name> --smoke     # 20 questions per setup: does it answer, and how long and how much the full run takes
uv run bench.py <name>             # the full run: the 8 headline datasets (both lists) and NevIR, then your rows

A rerun skips every answered question and retries the failed ones. At the end bench.py packs cache/<name>-<mode>/ and prints your rows in the main table's format, scored with the same code (scripts/test_bench.py checks that it reproduces every row already in the table).

4. Open a pull request with your row. On a branch of your fork, commit your entry in models.yaml, the saved responses in cache/<name>-<mode>/, and your rows in the main table (API rows with the API group, self-hosted rows with the self-hosted group, by score). In the description: a link to the weights or API and their license, the server code and its commit, the card and its hourly price if self-hosted, and the date of the run. Before merging I recompute every row from your saved responses with uv run bench.py <name> --score; a row that does not come out the same waits.

Layout

data/loaders.py        dataset download + one common shape
candidates/build.py    BM25 top-30 per query plus the "absent" twin  ->  candidates/<dataset>.jsonl (ids, queries, BM25 scores)
candidates/build_nevir.py
rerankers/             jev.py  cohere.py  zerank.py  llm_logprob.py  bm25.py
                       systemone.py (any System One server, asked Jev's eight ways), registry.py (reads models.yaml)
models.yaml            the System One models under test, one entry each (see "Test your own model")
run.py                 one model on one dataset, both variants, caching every raw response
bench.py               a models.yaml entry through the 8 headline datasets and NevIR, then its rows for the main table
cache/<model>/         <dataset>.<variant>.jsonl.gz  saved ranking responses, scores, latency, usage, cost
eval.py                metrics, nothing-relevant test, calibration, charts  ->  results/
significance.py  nevir_eval.py  rlcd_check.py  network.py  batching.py  determinism.py  coldstart.py
rlcd_runner.py         the self-hosted Qwen RLCD recipe, run on a GPU pod, writing the same cache rows
small_models_runner.py the self-hosted Laya / GLiNER2.5 runs (one passage per prompt), same cache rows
openjev/               pod.sh (one GPU pod: install Open-Jev, adapter, server copies, sharded run.py), fix.sh (full-window rerun
                       of the long lists), merge.py (merge the pod shards, verify coverage, write cache/), fix_cost.py (halve
                       the Sept 22 saved costs: two requests were in flight per server copy)
inject/hf_runner.py    open rerankers and small open models on a GPU pod (Qwen3-Reranker, bge, mxbai, Qwen3.5-4B, Tev1-4B),
                       one passage per prompt, writing the same cache rows
pods/                  hf.sh (one GPU pod for inject/hf_runner.py), systemone.sh (one GPU pod: a System One rebuild's own
                       server at its pinned version, then bench.py against it)
blog.py                renders an earlier experiment write-up from results/*.json
scripts/readme_header.py  draws the README header from saved scores and the website fonts
scripts/validate_systemone.py  replays the saved Jev responses behind the headline through systemone.py; --live asks real Jev
scripts/request_sizes.py  the largest request each setup sends  ->  results/request_sizes.json
scripts/test_registry.py  scripts/test_bench.py  tests for models.yaml and bench.py (no network)
.github/ISSUE_TEMPLATE/request-a-model.yml  the "Request a model" form

The full candidates/*.docs.jsonl passage files are not committed; build.py regenerates them from the datasets. Saved model responses may quote source text, and the evidence exporter includes candidate snippets with attribution.

Public evidence viewer

The interactive evidence viewer lives in the personal website repository and is available at anessbelbati.com/lab/jev-reranking/. This benchmark repository contains the experiments, saved results, and the data exporter; it does not contain a separate website app. The viewer includes the Qwen batched yes/no, four-level rubric, one-passage and reversed-order runs. The reversed-order run is an order-sensitivity diagnostic over the original eight datasets. Missing runs remain marked as missing, including Qwen on MIRACL French. Qwen costs are GPU rental estimates and its timings are measured on the GPU host; neither includes client network time or pod setup. The source responses are also available in this repository's cache/qwen-rlcd-* directories.

With the local candidate passage files available, export the curated snapshot:

uv run scripts/export_evidence.py
# Or write directly to a website checkout:
uv run scripts/export_evidence.py --output <website>/public/lab/jev-reranking/data
uv run -m unittest scripts/test_export_evidence.py

The default output is exports/evidence/ (Git-ignored). The export contains allowlisted saved outputs, the candidate snippets used in the run, prompt templates, attribution, source hashes, and downloadable dataset archives. Regeneration reads local records and makes no paid model calls.

Prices used

Usage totals per run (tokens, search units) are in results/summary.md so the cost can be checked against the vendor dashboards.

License

Code: MIT. Dataset content keeps its source licences (BEIR, MTEB/BRIGHT, CodeSearchNet, NevIR and MIRACL), including text quoted in saved responses or included in evidence exports. The exporter records source attribution; the code licence does not replace dataset licences.