FiQA semantic search — take-home

Semantic search over 57,638 FiQA finance answers on one CPU core: a naive baseline (Part 1), a 5x-latency optimization with a measured quality/latency frontier (Part 2), and an instant-search demo UI over the frozen configs (Part 3).

Start here:

• james-han-search-optimization-writeup.pdf — the writeup (a copy of paper/main.pdf; LaTeX sources under paper/, every number generated from the frozen result JSONs by paper/gen_numbers.py).
• predictions.md — pre-registered predictions committed before each experimental round, with recorded outcomes (misses included).
• results/part2/final_frontier.md — the frontier table with the operating point marked.
• results/part2/round4_rust.md — Day-2 Rust-kernel round: parity gate and results.

Requirements

• macOS on Apple silicon is what everything was measured on (Apple M5 Pro); the code is plain Python + a portable Rust extension and should run elsewhere, but no number in the writeup was produced anywhere else.
• Python 3.12 via uv.
• Rust toolchain (rustup, stable) — only for the *_rs Rust-kernel configurations; every numpy-path configuration runs without it.
• Network on first run (model downloads from Hugging Face).

1. Environment

uv sync                      # creates .venv with pinned dependencies

2. Build artifacts (offline; free to use all cores)

Artifacts are deliberately not committed; each build records SHA-256s into the manifest (artifacts.sha256, artifacts_part2.sha256).

# Part 1: embed the corpus with the fixed MiniLM model (~5 min)
.venv/bin/python embed_corpus.py

# Part 2, minimal set for the operating point + hybrid:
.venv/bin/python part2_build.py bm25            # BM25 indexes (~1 min)
.venv/bin/python part2_build.py splade-query    # int8 ONNX query assets (~2 min)
.venv/bin/python part2_build.py splade-docs     # SPLADE doc expansion (~25-30 min, all cores)
for k in 32 48 64 96; do .venv/bin/python part2_build.py splade-topk --src splade_docs_csc.npz --k $k; done

# Rust kernel (the *_rs configs)
uv pip install maturin
VIRTUAL_ENV=$PWD/.venv RUSTFLAGS="-C target-cpu=native" \
  .venv/bin/maturin develop --release -m rs_kernel/Cargo.toml

# Optional, only for the full round-1 frontier rows (statics, wall point):
.venv/bin/python part2_build.py potion
.venv/bin/python part2_build.py mrl
.venv/bin/python part2_build.py student
.venv/bin/python part2_build.py onnx
.venv/bin/python part2_build.py hnsw --emb artifacts/part2/emb_potion.npy --out artifacts/part2/hnsw_potion.bin
.venv/bin/python part2_build.py hnsw --emb artifacts/part2/emb_mrl.npy   --out artifacts/part2/hnsw_mrl.bin
.venv/bin/python part2_build.py hnsw --emb artifacts/corpus_emb.npy      --out artifacts/part2/hnsw_minilm.bin

# Round-6 synthetic 10x corpus (latency-scaling rows only):
.venv/bin/python part2_build.py splade-x10 --src splade_docs_top96_csc.npz

3. Reproduce Part 1 (baseline)

./run_baseline.sh

Verifies the artifact manifest, boots serve.py on :8000 with pinned threading, gates on a served-pid check and a parity sanity check, runs the unmodified harness three times (median-of-3 p95 is the reported number), captures chip/power/thread environment, and writes results/part1_summary.json.

4. Reproduce Part 2 rows

./run_part2.sh <config> raw [suffix]     # 3 harness runs on the tune split
VAL=1 ./run_part2.sh <config> raw <suffix>   # the sealed validation split

Use a suffix (e.g. repro) so fresh runs never overwrite the frozen result files. Key configs: splade_bow48 (operating point, numpy), splade_bow48_rs / splade_bow32_rs (Rust kernel), fusion_rescore48_rs (rescoring hybrid on the Rust kernel), splade_bmw96_rs (Block-Max WAND, K=96), bm25, splade (full pair), wall / potion / mrl (round-1 rows). Thread pinning is inside the script; run on a quiet machine.

The harness itself, exactly as graded:

.venv/bin/python candidate_bundle/benchmark.py \
  --queries data/part2/tune_queries.jsonl --qrels data/part2/tune_qrels.tsv \
  --url http://localhost:8001/search --k 10

5. Gates and tests

.venv/bin/python -m pytest tests/ -q          # unit suites incl. rs_kernel parity semantics
.venv/bin/python part2_parity_rs.py           # Rust-vs-numpy rankings on all 4,400 tune queries
.venv/bin/python part2_parity_rs.py --pairs "splade_bow96_rs:splade_bmw96_rs" --out results/part2/parity_bmw_repro.json

6. Demo (Part 3)

.venv/bin/python serve_demo.py    # http://localhost:8010/demo/

Instant search over the frozen configs with switchable modes (baseline / wall point / hybrid / operating point / runner-up), per-query stage bars against the 0.83 ms budget, matched-term highlighting, and honest empty / garbage / no-match / error states. Needs the Part 1 + Part 2 minimal artifacts above (all five modes need the optional wall-point assets too).

7. Load exploration (round 7, not graded)

.venv/bin/python part2_load.py --url http://localhost:8001/search \
  --rates 100,200,400,800,1200,1600,2000,2400 --out results/part2/load_repro.json

Open-loop Poisson arrivals; latency measured from scheduled arrival time; client send-lag reported so client saturation is visible. Multi-core rows use the splade_bow48_shard6_rs config and are labeled multi-core wherever they appear.

Notes

• The graded number is always sequential (--concurrency 1) p95 through the unmodified harness; candidate_bundle/benchmark.py has no candidate commits touching it (verify: git log -- candidate_bundle/benchmark.py).
• naver/splade-v3-doc (round 5's primary target) is license-gated on Hugging Face; the executed fallback is the ungated naver/splade-v3-distilbert (part2_build.py splade-v3-docs). Both are CC BY-NC-SA 4.0 — research/evaluation use.
• Dev-split tune/validation query files are committed under data/part2/ (5,500 dev queries, 80/20 at seed 0 via make_split.py).