static-site-search-eval

sss-eval chunks a static site's markdown, embeds the chunks with a static embedding model (model2vec/potion), writes browser-readable index artifacts, and evaluates retrieval quality against a query set. The point of the package is the claim in its name: the whole thing was built to find out whether a static site can ship real semantic search without shipping a neural network to the browser. It can.

Why the model fits in a few megabytes

A model2vec / potion model is not a neural network. It is a lookup table: one row of floats per vocabulary token. There is no attention, no matrix multiply chain, no ONNX runtime, no WASM binary. Inference is four steps:

ids = tokenizer.encode(text)          # WordPiece token ids, no [CLS]/[SEP]
rows = embedding[ids]                 # gather: one row per token id
pooled = rows.mean(axis=0)            # mean-pool the *unnormalized* rows
vector = pooled / np.linalg.norm(pooled)  # L2-normalize the pooled result

That's the forward pass. Training (distilling a real sentence-transformer down to a static table via PCA) is expensive; using the result is a gather and a mean. Because inference has no learned computation graph, the entire "model" a browser needs is the token table itself — an int8 matrix plus a small vocabulary list — which is why the winning configuration below downloads 4.21 MB instead of the 23+ MB an ONNX sentence-transformer needs for the same job.

Install

pip install static-site-search-eval
# or
uv add static-site-search-eval

Requires Python 3.12+.

Building an index

The winning configuration found by the eval in this repository (see Results below):

sss-eval build \
  --corpus content/post \
  --outdir static/search \
  --model minishlab/potion-base-8M \
  --dims 128 \
  --chunk-size 600 \
  --chunk-overlap 120

Title-prefixing (each chunk's embedded text is prefixed with its post title) is the default; pass --no-title-prefix to disable it. --cache-root (default .embed-cache) controls where content-hash-keyed embeddings are cached between runs, so re-running build after editing one post only re-embeds that post's chunks.

This writes six files to --outdir:

• manifest.json
• chunks.<hash>.json
• docs.<hash>.bin
• tokens.<hash>.bin
• scales.<hash>.bin
• vocab.<hash>.json

and prints a one-line summary:

313 chunks, 128d, vocab 29528 -> static/search

To confirm the index only points at anchors that actually exist in the rendered site (heading anchors are indexed today so that section-level deep links can be turned on later without re-embedding, but nothing renders them yet — data that nothing reads rots quietly unless something checks it):

hugo
sss-eval verify-anchors --search-dir static/search --public public

The artifact format

A browser only ever fetches manifest.json by a fixed URL; everything else it names is content-hashed (<stem>.<sha256-prefix-12>.<ext>) and can be cached forever. Bump the corpus or the model and the hash changes, so there is no versioning scheme to get wrong and no stale-cache class of bug — manifest.json is the only response that must be served with no-cache.

Every binary artifact is a raw ArrayBuffer, never base64. Base64 costs +33% on the largest file shipped (tokens.bin); for a payload whose entire point is its download size, that is not a rounding error.

The three tokenizer traps

Reimplementing potion's tokenizer in JavaScript (rather than shipping a WASM tokenizer, which would defeat the purpose) means reproducing three non-obvious behaviors of HuggingFace tokenizers, each verified against model2vec's own StaticModel.tokenize():

1. No [CLS] / [SEP]. tokenize() calls tokenizer.encode_batch_fast(sentences, add_special_tokens=False). The TemplateProcessing post-processor recorded in tokenizer.json describes how special tokens would be inserted — it is a decoy left over from the tokenizer's BERT ancestry. add_special_tokens=False means it never runs. A browser implementation that adds [CLS]/[SEP] ids will embed a vector for a token sequence the Python pipeline never produces.

2. [UNK] ids are deleted, not embedded. After tokenizing, model2vec filters unk_token_id out of the id list entirely: [t for t in token_ids if t != unk_token_id]. It does not gather the [UNK] row and mean-pool it in. A query that is entirely out-of-vocabulary (e.g. a string of emoji, or a language the tokenizer's vocab doesn't cover) tokenizes to an empty id list, and mean-pooling zero rows must yield a zero vector rather than throwing or returning the [UNK] embedding.

3. Accents are stripped even though the config says they aren't. tokenizer.json's normalizer sets "strip_accents": null. That reads like "leave accents alone." It doesn't: HuggingFace tokenizers' BertNormalizer treats a null strip_accents as "inherit from lowercase," and lowercase is true. So accents ARE stripped — café and cafe tokenize identically. A browser reimplementation that takes the JSON at face value and skips accent-stripping will silently diverge from the Python tokenizer on any query containing a diacritic.

Two facts a browser implementer needs

Both measured against this package's own quantization code (src/sss_eval/quantize.py), not assumed:

1. The browser never has to dequantize docs.bin. Cosine similarity is invariant to a positive per-row scale, and every row of docs.bin was quantized with the same global scale (127, applied to unit vectors), so that scale is a constant factor across every document's score — it cannot change the ranking. A Float32Array query vector dotted directly against the raw int8 document rows produces the same ranking as dotting against the dequantized float32 rows, to within 1e-6. Query-side, the query vector is built once per search from the float32 token table math above, dequantizing only what's needed (the tokens actually present in the query), never the 313-row-or-larger document matrix.

2. int8 @ int8 wraps around silently. A 300-dimension dot product of two int8 vectors whose true mathematical value is 3,000,000 does not throw, does not produce NaN, and does not saturate — it wraps modulo 256 per multiply-accumulate step in a naively-typed accumulator and can return something like -64. There is no exception to catch. The fix is structural, not defensive: accumulate the running dot product into a Float32Array (or a plain JS number, which is already a float64), never into an Int8Array or Int32Array sized to the inputs.

Reproducing the eval

The eval this package shipped is pinned against a frozen corpus snapshot; see examples/degoe-de/SNAPSHOT.md for exactly which commit of which site it was taken from and why it is not meant to be refreshed — the numbers in the Results section below and in the accompanying blog post are only meaningful against that exact snapshot.

uv sync
pnpm install

# Dump chunk texts (and query texts) to JSON so the Node arms embed exactly
# what Python did -- same chunker, same text, different runtime.
uv run python -m sss_eval.dump_chunks \
  --corpus examples/degoe-de/corpus \
  --out build/chunks.json \
  --queries examples/degoe-de/queries.yaml

# Embed those chunks and queries with the non-potion arms, each with the
# real runtime a browser would use: transformers.js (ONNX q8) for MiniLM,
# the ternlight WASM engine for its two arms. build_arms (below) looks for
# these three files next to --outdir, i.e. in --outdir's parent.
node node/build_minilm.mjs build/chunks.json build/minilm.json
node node/build_ternlight.mjs build/chunks.json build/ternlight-base.json @ternlight/base
node node/build_ternlight.mjs build/chunks.json build/ternlight-mini.json @ternlight/mini

# Rank the same 30 queries with the site's real Fuse.js config, for the
# keyword baseline and the RRF-fused hybrid arms.
node node/rank_fuse.mjs examples/degoe-de/index.json examples/degoe-de/queries.yaml build/fuse-ranks.json

# Assemble every arm (potion swept over dims/chunking/title-prefix, plus the
# Node arms above) into the format evaluate.py reads.
uv run python -m sss_eval.build_arms \
  --corpus examples/degoe-de/corpus \
  --queries examples/degoe-de/queries.yaml \
  --manifest build/arms.json

# Score every arm, semantic-only and RRF-fused with the keyword baseline,
# against the 30-query eval set, and apply the pre-registered ship rule
# (smallest download within 0.03 recall@3 of the best arm; MRR breaks ties).
uv run python -m sss_eval.evaluate \
  --queries examples/degoe-de/queries.yaml \
  --arms build/arms.json \
  --keyword-ranks build/fuse-ranks.json \
  --json build/results.json

Node is only needed to reproduce the MiniLM/ternlight/keyword arms of the eval; it is not a dependency of sss-eval build or of using the artifacts in a browser.

Results

Winner: potion-base-8M, PCA-truncated to 128 dims, 600-char chunks with 120 overlap, title-prefixed, RRF-fused with a keyword ranking (Fuse.js, already shipped with the site).

First-query download of 4,205,974 bytes (4.21 MB) is the int8 token table plus its float32 per-row scales plus the JSON vocabulary — the three files a browser needs before it can embed a single query. The document index itself, for the 13-post corpus in examples/degoe-de/, is 313 chunks × 128 int8 dims = 40 KB, fetched once and cached alongside the rest of the site's static assets.

License

The code in this repository is MIT licensed (see LICENSE). examples/degoe-de/corpus/ contains Bart de Goede's blog posts, included as evaluation data for the eval above — that content is copyright Bart de Goede and is not covered by the MIT license.