kleinhirn

Encoder models in the browser, on the GPU, with no server.

kleinhirn runs text encoders in the user's browser: classification, zero-shot classification over NLI, sentence embeddings, named entities and reranking. The text stays on the device, there is no inference server to pay for, and it works offline once the model is cached.

It loads Hugging Face checkpoints of six encoder families directly from config.json and safetensors: BERT (with ELECTRA and MiniLM), RoBERTa, XLM-R, DistilBERT, DeBERTa-v2/v3 and ModernBERT. Two models with their own heads run on the same engine: GLiNER2.5, a zero-shot classifier that takes any set of labels, and Julia 1, a decision model that picks one of up to 20 options. The engine ships its own GPU programs (WGSL kernels) for WebGPU; browsers without WebGPU fall back to a WASM-SIMD executor on the CPU that runs the same plan for every family, on up to 8 threads.

The usual tool for running such models in a browser is ONNX Runtime Web (ORT) from Microsoft. It is the baseline for every speed number below.

Results

• Same answers as the original. All 89 checkpoints on our list (the three most downloaded per family and task, up to 200M parameters) make the same decisions as the PyTorch reference: 100 % in full precision (f32), at least 99.5 % in half precision (f16) of the decisions that f16 rounding cannot flip, embeddings with cosine at least 0.9999 and 0.999.

• Faster than ORT at its best, in every family. One model per family (the most downloaded), inputs of exactly 128 and 512 tokens, f16, against ORT's fastest setting we found for each model (graph optimizer variants, three ORT builds, graph capture where it starts):

  Model	128 tokens kleinhirn / ORT	512 tokens kleinhirn / ORT
  all-MiniLM-L6-v2 (BERT)	3.29 / 5.29 ms	8.97 / 12.06 ms
  twitter-roberta-base-sentiment (RoBERTa)	13.40 / 18.96 ms	47.55 / 59.67 ms
  mmarco-mMiniLMv2-L12 (XLM-R)	6.07 / 10.09 ms	17.86 / 24.33 ms
  distilbert-base-uncased-sst-2 (DistilBERT)	6.99 / 9.76 ms	24.54 / 33.34 ms
  deberta-v3-base-prompt-injection-v2 (DeBERTa-v3)	14.54 / 24.35 ms	57.31 / 95.53 ms
  granite-embedding-small-english-r2 (ModernBERT)	6.99 / 12.44 ms	21.35 / 35.81 ms

  1.26x to 1.78x faster; all three kleinhirn repetitions below all three ORT repetitions in each of the twelve cells.

• Faster again on real texts. kleinhirn computes only the rows a text has; ORT runs a fixed-shape graph padded to the bucket or a dynamic-length graph without graph capture. Same six models, each on its 200 test texts at their real length, against ORT's fastest setting per browser and model (including dynamic-length graphs), ms per call:

  Model	Chromium f16	Chromium f32	Safari f16	Safari f32
  all-MiniLM-L6-v2	2.09 / 5.22	2.58 / 6.23	5.25 / 11.44	6.30 / 14.24
  twitter-roberta-base-sentiment	7.27 / 17.23	10.69 / 19.67	19.52 / 40.96	28.85 / 48.05
  mmarco-mMiniLMv2-L12	4.03 / 10.16	5.23 / 11.89	10.68 / 21.94	13.86 / 28.12
  distilbert-base-uncased-sst-2	3.95 / 9.48	5.61 / 10.52	10.38 / 21.14	15.07 / 25.01
  deberta-v3-base-prompt-injection-v2	7.77 / 23.48	11.57 / 30.01	20.85 / 48.73	29.05 / 67.36
  granite-embedding-small-english-r2	3.96 / 10.80	5.67 / 11.98	10.48 / 24.13	13.44 / 27.18

  kleinhirn / ORT. 1.66x to 3.02x faster, on average 2.58x (Chromium f16), 2.17x (Chromium f32), 2.16x (Safari f16) and 1.98x (Safari f32); all three kleinhirn repetitions below all three ORT repetitions in each of the 24 cells. On the 1,000-text corpus (median 45 of 128 tokens) GLiNER2.5-small runs 4.4 ms against 12.2 ms for ORT with graph capture in f16 (2.8x) and 5.4 against 14.8 ms in f32 (2.7x). Julia 1 decides in 10.4 ms against 25.5 ms for its upstream ORT pipeline in f16 (2.4x) and 14.0 against 29.2 ms in f32 (2.1x).

• Faster than ORT without a GPU too. The WASM path against ORT's WASM runtime with all cores in its fastest setting per browser and model, all six models at 128 tokens, 512 tokens and real length (18 cells per browser), ms per call, kleinhirn / ORT:

  Model	Chromium 128	Chromium real	Safari 128	Safari real
  all-MiniLM-L6-v2	8.20 / 55.11	2.34 / 8.91	22.03 / 33.77	3.24 / 10.18
  twitter-roberta-base-sentiment	46.65 / 209.03	15.39 / 60.29	73.40 / 79.69	22.01 / 28.71
  mmarco-mMiniLMv2-L12	13.34 / 63.75	6.73 / 28.54	25.64 / 26.57	10.05 / 25.14
  distilbert-base-uncased-sst-2	23.22 / 103.87	8.27 / 31.45	36.54 / 40.09	11.72 / 18.35
  deberta-v3-base-prompt-injection-v2	51.08 / 231.18	15.84 / 72.72	79.25 / 94.80	22.72 / 43.63
  granite-embedding-small-english-r2	18.57 / 80.26	5.39 / 19.91	27.69 / 37.13	7.62 / 12.05

  Geometric mean of ORT / kleinhirn over the 18 cells: 4.20x in Chromium (kleinhirn ahead in all 18, 3.59x to 6.72x), 1.53x in WebKit, 1.34x in Safari and 4.37x in a regular Firefox 157. At 512 tokens Safari and WebKit are about even (0.98x to 1.29x). Same answers as PyTorch in all 72 cells. At real text length alone (6 cells per browser): 4.00x Chromium, 2.01x WebKit, 1.91x Safari, 4.43x Firefox. One run per cell and engine: single cells vary by 20 to 50 % between two nights, the means over 18 cells hold. Threads need a cross-origin isolated page; without it both engines run one thread, and kleinhirn is 1.74x faster in Chromium and 1.09x in Safari. Measured 8 and 9 October 2026.

• Closer to the reference than ORT. On the measurement inputs above, ORT's f16 output deviates 2.9 to 20.5 times more from ORT's own f32 output than kleinhirn's f16 does. On Julia 1 the largest logit error in f16 is 0.109 for kleinhirn and 1.02 for ORT. Every kernel accumulates in f32.

• Less memory. 52 to 71 % of ORT's peak memory, depending on the path (941 vs 1,442 MiB for GLiNER2.5-small in f16, 1,138 vs 1,599 MiB for Julia 1). The ORT peaks are from September; kleinhirn's grew by 5 to 14 % with the new kernels.

• Small. The engine is 47 KB gzip, plus 16 KB for the WASM path (17 KB with threads). ORT's WebGPU runtime files are 6.3 MB gzip (25.9 MB raw). The model weights come on top in both cases.

• Not done yet. Phones have not been measured since the kernels changed (October 2026); the last iPhone run missed the latency target (below).

• Reproducible. Every number comes from a script in this repo under a fixed protocol: interleaved runs, a quiet machine, the median of three repetitions. Details under "How we measure" and in docs/FINDINGS.md, sections 14 and 15.

Use cases

• Learning platforms that give feedback on student answers on the device. The first user is argmin, a German learning platform that classifies misconceptions in learner answers.
• Local-first and offline apps: notes, mail or task tools that tag, sort or route text where it was written.
• Semantic search and deduplication with sentence embeddings, computed where the text is.
• Browser extensions that classify the current page or the text being typed, with no backend.
• Support forms that suggest the right category or queue while the user types.
• Moderation by your own rules: zero-shot labels checked in the browser before anything is posted.
• Agents and interfaces that need a fast local choice among fixed options, which is what Julia 1 is trained for: which tool, which step, which intent.

Status

• Models: the 89 checkpoints of data/k28/models.json (six families, five tasks: sequence classification, NLI, embeddings, token classification, reranking), GLiNER2.5 small, base and multi (classification) and Julia 1. GLiNER2.5's entity extraction is not implemented yet.
• The speed numbers above are from an M1 Pro in Chromium. The new kernels give the same decisions in Playwright WebKit and Firefox.
• Device matrix from September 2026 (before the kernel rewrite): parity with the PyTorch reference holds in every stage that runs in Chromium, Brave, Playwright WebKit, Safari 27, Playwright Firefox 153 and Chromium without WebGPU, and on an NVIDIA L4 (FINDINGS section 13). Latency for the 256-token bucket missed the targets (p95 of 20 ms on desktop WebGPU, 300 ms for WASM) in every row then; Firefox pays about 104 ms for every asynchronous GPU readback.
• On an iPhone 16 Pro (iOS 27, Safari and Brave, which is WebKit there) f16 ran with full parity, but the 256-token p95 was 166 ms against the 60 ms target; the WASM stage also runs there with exact parity (L256 p95 775 ms). Peak memory and the tab kill threshold are not measured yet.

Measure your device

The repository holds a static page, site/, that runs the engine on the device that opens it. It loads the weights from Hugging Face, checks parity against the goldens, measures latency for two length buckets on every stage the device supports (f16, f32, WASM), and shows one JSON file you can download. Page address: https://smashburger-dev.github.io/kleinhirn/

• Nothing is uploaded. The page requests only itself and the weights from Hugging Face. The JSON stays on your device until you share it.
• The file holds the user agent, core count, memory class, the WebGPU adapter info, features and limits, and what you type into the device field. Adapter info and features can roughly identify a device model. Read the file before you share it.
• The page keeps a small marker (first visit, last visit, visit count) in browser storage to measure how long storage survives on the device. The marker holds no other data.
• To contribute, open an issue with the "Device result" form and attach the file. A maintainer runs tools/check_device_result.mjs, which recomputes parity from the logits in the file, and adds the result to docs/COMMUNITY.md.
• Community numbers are separate from the official matrix. Parity is verified by script; latency is self-reported and unverified. One tab on one device is one sample. The file format is in docs/device-results.md.

Why it is faster

• The encoder matmuls are register-blocked: every GPU thread computes a 4x4 block of the output from tiles in workgroup memory, and each output still sums in a fixed order in f32. Three tile sizes give the same bits; each call picks the one that keeps the GPU busy for its row count.
• Attention runs as three matmul-shaped kernels (scores, softmax, context). DeBERTa's relative terms (content-to-position and position-to-content) are matmuls over the relative positions the input can reach. Tiles that no valid token reaches are skipped.
• Only the rows a text has are computed; buckets only fix buffer sizes.
• Each sequence bucket gets a fixed plan built at load time: pipelines, buffers and bind groups exist before the first call. A call uploads the inputs, encodes one command buffer and reads back once.
• The engine requests exactly the WebGPU minimum device limits (256 invocations per workgroup, 16 KiB workgroup memory, 128 MiB per storage binding) and uses no subgroups, so a desktop GPU cannot use more than a phone GPU guarantees. ORT asks for more (1,024 invocations, 32 KiB) and accumulates its f16 matmuls in f16.

How ORT was set up

For the six families: one ONNX export per model and length with fixed shape [1, L] and the head inside the graph, then onnxruntime.transformers.optimizer with its default fusions and two variants (without EmbedLayerNormalization, with MultiHeadAttention), the optimizer's fp16 conversion, three ORT Web 1.29.0 builds (default, webgpu, jspi), graph capture on and off. The fastest combination per model and length that passes the parity check on the CPU is the baseline (data/k28/k28.8-best.json): the plain export with graph capture in six cells, the optimized graph with capture in four, and for DeBERTa the optimized graph without capture (capture does not start there), always in one of the native WebGPU builds.

For GLiNER2.5 and Julia 1 the plain export goes through the same optimizer (BERT path). For WebGPU, convert/capture_surgery.py replaces a Squeeze/Unsqueeze pair in GLiNER's attention-mask chain with one Reshape, so ORT's graph capture can run. ORT also gets its cheapest timing boundary: inputs in persistent GPU buffers, one readback per call.

Roadmap

• Phones: the iPhone with the new kernels, peak memory, how the weights are loaded.
• int8 weights for the layer matrices: they are 98 to 100 % of what a model downloads once the word rows come on demand.
• Kernel choice per device by the best worst case.
• A public benchmark for encoders in the browser that compares engines with a parity column.

Supported models

• The 89 checkpoints of data/k28/models.json, each at a pinned revision: per family and task the three most downloaded Hugging Face models up to 200M parameters. Convert one with node tools/k28_convert.ts <model-id> after fetching it (convert/k28_fetch.py).
• GLiNER2.5 classification models (DeBERTa-v2/v3 encoders): small-upstream (fastino/gliner2.5-small-v1), base-upstream (fastino/gliner2.5-base-v1), multi-upstream (fastino/gliner2.5-multi-v1), each at a pinned revision (tools/fetch_models.py).
• Julia 1 (SupersonicLabs/Julia-1), a ModernBERT decision model fine-tuned from mmBERT-small: state + question + 2-20 options in, one decision out.

Bundle size

import 'kleinhirn' loads dist/kleinhirn.js, 15.7 KB gzip. Everything a model does not need loads on demand: the WebGPU executor with its kernels, the WASM executor in a worker, the tokenizer, and the GLiNER and Julia engines. Engine bytes a page loads until its first result (node tools/r6_paths.mjs, gzip -9):

Tokenizers ship as tokenizer.bin: the tokenizer.json with vocabulary, scores and merges in binary sections, gzip inside the file, decoded back to the identical object. The XLM-R tokenizer drops from 17.1 MB to 1.99 MB, DeBERTa-v3 from 8.7 MB to 0.89 MB, BERT WordPiece from 0.47 MB to 0.12 MB (node tools/r6_tokbin.ts --survey). Folders that still name a tokenizer.json load unchanged. A tokenizer is ready 4 to 22 times faster than in the first tokenizer.bin release, because the vocabulary lands in one typed hash table instead of a Map of 250,000 strings (Node on an M1 Pro, node --expose-gc --import ./tests/helpers/node-hooks.mjs tools/r4_tokload.ts --survey): XLM-R 232 to 40 ms, Julia 695 to 83 ms, all list tokenizers together 5,050 to 875 ms. Unigram tokenizers also encode twice as fast.

The tokenizers alone are a second entry: import { JsonTokenizer } from 'kleinhirn/tokenizer' (dist/tokenizer.js) with the synchronous JsonTokenizer.fromString and await JsonTokenizer.fromBytes(bytes) for tokenizer.bin or tokenizer.json. A loaded model tokenizes itself, so model code needs no tokenizer import.

For comparison, onnxruntime-web 1.29.0 loads three runtime files on its WebGPU path (ort.webgpu.bundle.min.mjs, ort-wasm-simd-threaded.asyncify.mjs and .wasm): 25,917,382 B raw, 6,338,621 B gzip -9. The model weights are a separate download in both cases.

Word rows on demand

The word-embedding table is 31 to 81 % of a model's download (mMiniLM 192 of 237 MB), but a text needs only the rows of its own tokens. words.bin holds the table with a checksum after each row, and the load option words picks how it arrives:

• 'full' (default): the whole table at load, as before. Nothing about the text leaves the device.
• 'rows': only the rows a text needs, by HTTP Range, checked row by row and kept in memory and Cache Storage. Every text with new tokens waits one round trip (about 250 ms from our test machine to the Hugging Face CDN).
• 'progressive': like 'rows' at first, then the whole table in the background; after that it computes like 'full'.

Ready-made weights for the six bench models, GLiNER2.5 small, base and multi and Julia 1 (f16 and f32, every mode) are on Hugging Face in smashburger-dev/kleinhirn-weights, one folder per source model; pin a revision in the manifest URL.

Row requests show the host which tokens a text contains. Use 'rows' and 'progressive' for public text or a host you control; keep 'full' for private text. A host that ignores Range gets the full download with the same results.

Measured in Chromium on an M1 Pro, WebGPU f16, fresh browser profile (node bench/run-r4-load.mjs, bench/results/r4-r4-load-summary.md). Logits are bit-identical in every mode.

The 'full' mode got lighter too: the table is read without a copy and the WASM worker no longer receives it, so mMiniLM peaks at 907 MiB on WASM instead of 1,538 MiB, and models load 12 to 32 % faster from a local server.

const model = await EncoderModel.load({ manifestUrl, words: 'progressive' });
console.log(model.info().words); // { mode, range, rows, requests, bytes }

Runtime path

Fallback chain (backend: 'auto'):

1. WebGPU f16 (when the adapter supports shader-f16)
2. WebGPU f32
3. WASM-SIMD in a worker (AssemblyScript, no Emscripten), every family; threads: 'auto' uses up to 8 threads on a cross-origin isolated page (COOP and COEP headers), else one, and the relaxed-SIMD build where the browser has it

The engine requests the device with exactly the WebGPU minimum limits (limits: 'minimum', the default): 256 invocations per workgroup, 16 KiB workgroup memory, 128 MiB per storage binding, etc. All kernels stay inside these limits, so a desktop adapter cannot silently use more than a mobile adapter provides.

Sequence buckets: the general engine loads 128 and 512 by default (any multiple of 4 up to the model's positions); GLiNER uses L64-L1024 with up to 16 labels per call and a wide L1280/K80 bucket for label sets up to 80 (see docs/ARCHITECTURE.md).

Quickstart

npm ci
npm run build                       # dist/kleinhirn.js + wasm worker

# Python env for conversion + goldens (uv or venv, Python 3.12)
uv venv .venv && uv pip install --python .venv/bin/python -r convert/requirements.txt

# Julia-1 scripts (fetch_datasets, julia_golden, export_julia) need
# transformers 5.x in a second env
uv venv .venv-julia && uv pip install --python .venv-julia/bin/python -r convert/requirements-julia.txt

# Pinned model checkpoints -> models/ (gitignored)
.venv/bin/python tools/fetch_models.py small-upstream
.venv/bin/python convert/export_weights.py small-upstream   # -> models/small-upstream/{f32,f16}
# Julia-1: fetch_models.py julia-1, then .venv-julia/bin/python convert/export_julia.py

# Pinned benchmark inputs -> models/k9-data, models/jev-benchmarks
.venv-julia/bin/python tools/fetch_datasets.py

node --test tests/*.test.ts         # tokenizer + limits tests

In the browser (the package is not on npm; serve dist/ or import the built file directly):

Any model of the list (Python env with convert/requirements-k28.txt):

.venv-k28/bin/python convert/k28_fetch.py sentence-transformers/all-MiniLM-L6-v2
node tools/k28_convert.ts sentence-transformers/all-MiniLM-L6-v2   # -> models/k28/<slug>/{f32,f16}

import { EncoderModel } from './dist/kleinhirn.js';

const model = await EncoderModel.load({
  manifestUrl: '/models/k28/sentence-transformers__all-MiniLM-L6-v2/f16/manifest.json',
});
const [vector] = await model.embed(['The rocket launched successfully.']);
// also: classify(text), zeroShot(text, labels), rerank(query, passages), tokenClassify(text)

GLiNER2.5:

import { loadEngine } from './dist/kleinhirn.js';

const kh = await loadEngine({
  manifestUrl: '/models/small-upstream/f16/manifest.json',
  buckets: [128, 256, 512, 1024],
});
const out = await kh.classify(text, [
  { task: 'topic', labels: ['science', 'sports', 'politics'] },
]);

How we measure

• Reference is PyTorch fp32 on CPU. Goldens are regenerated from the public corpus tests/corpus/texts1000.json (1,000 texts: 500 from the argmin content tree under CC-BY-4.0, 500 generated by tools/synthetic_texts.py) via convert/golden.py.
• Engine parity runs in a visible Playwright Chromium against those goldens: node bench/run-parity.mjs <model> <f32|f16>.
• A numpy reference forward (convert/manifest_forward.py) reads the kleinhirn manifest + shards without PyTorch and must match the goldens to 1e-4 with 100 % argmax. This validates the weight format independent of any browser.
• Accuracy benchmarks follow docs/BENCHMARKS.md (K9 suite): every dataset is fetched at a pinned revision by tools/fetch_datasets.py; per-example prediction files contain only id, gold, predicted, probabilities.
• Latency and peak memory are only reported from official runs on a 10-core M1 Pro: interleaved ABAB, median of 3 repetitions with the range recorded per run; a run is provisional when the 1-minute load at its start reaches the pass's bound (4 for the pass against the fused ORT graphs, 6 for the earlier passes). The memory metric is calibrated against a known 512-MB GPU buffer.
• The ORT side runs on the best configuration we could produce: the plain ONNX export (convert/export_onnx.py) after onnxruntime.transformers.optimizer (convert/optimize_onnx.py), in f32 and in the optimizer's own fp16 conversion. On WebGPU a two-node rewrite of the attention-mask chain (convert/capture_surgery.py) lets ORT's graph capture (enableGraphCapture) run, with inputs in persistent GPU buffers and one readback per call.

Numbers

The tables below are the September measurements of GLiNER2.5 and Julia 1 (before the kernel rewrite); current numbers are under Results and in docs/FINDINGS.md, sections 14 and 15.

Engine parity on the 1,000-text corpus (WebGPU, minimum limits; bench/run-parity.mjs):

WASM-SIMD parity (f32, --backend=wasm, small-upstream): argmax 100 %, max logit diff 2.25e-5; all gates passed.

Julia 1 (bench/run-julia-parity.mjs): f32 100/100 choices, logit diff 1.0e-4; f16 100/100 choices, max probability diff 2.4e-2, above our 1e-2 probability gate (known limitation, see docs/FINDINGS.md).

Tokenizer parity (node --test tests/): 100 % equal token ids vs the HF tokenizer on all 1,000 corpus texts, for all three GLiNER models and Julia 1.

Latency and memory (official runs)

Measured on a 10-core M1 Pro (16 GB) in Playwright Chromium 151.0.7922.34 (Apple Metal adapter, WebGPU minimum limits), interleaved ABAB, median of 3 repetitions; ranges show the repetition min-max. Runner bench/run-official.mjs, results under bench/results/, full detail in docs/FINDINGS.md. The WebGPU comparison against ONNX Runtime Web with graph capture comes from a pass at commit 2b7dacd (k20gc-official-*); the WASM pair, Julia 1 and the memory table come from a pass at commit ba3a2c7 (k20-official-*, k20-mem-p*), both with a start-load bound of 4. The bucket sweep and the batch throughput come from earlier passes with a bound of 6 (commits ccef9a7 and 8535538, official-*).

What the ORT optimizer does to these graphs (onnxruntime 1.29.0, BERT path, the only one that applies; bench/results/k20-optimize-*.json): it fuses LayerNorm, SkipLayerNorm and Gelu, but not the attention of either model. DeBERTa-v2 keeps its relative-position terms (c2p, p2c) as separate gathers, ModernBERT keeps RoPE and the windowed mask unfused: 0 Attention and 0 MultiHeadAttention nodes in both. The fused graphs are still faster, mostly in f16, because they drop the casts that the plain fp16 conversion inserts. ORT's graph capture does not start on the fused graph: a Memcpy pair around a CPU-placed Squeeze in the DeBERTa mask chain blocks it. Replacing that Squeeze and the following Unsqueeze with one Reshape to the fixed shape (element-identical, same CPU parity) puts every node on WebGPU, and capture runs.

kleinhirn vs ONNX Runtime Web, small-upstream, L128 (median ms per inference call on 968 corpus texts; the ORT page serves L128 only):

Each kleinhirn row pairs with the ORT row below it (ABAB). The WASM pair comes from the ba3a2c7 pass; the wasm EP has no graph capture. In its own ABAB pair the plain export ran 17.4 ms (f16) and 18.2 ms (f32) against 14.2 and 17.5 ms for the fused graph; graph capture then took the fused graph from 14.8 to 12.5 ms (f16) and from 18.4 to 15.7 ms (f32). The same path moves by up to 14 % between passes (kleinhirn f32 6.5 ms in the ba3a2c7 pass, 7.4 ms in the capture pass), so a factor holds only within its pair. Both ORT fp16 graphs miss our fp16 probability gate (max probability diff 5.3e-2 plain, 3.5e-2 fused, gate 1e-2); kleinhirn f16 stays at 4.2e-3.

kleinhirn across sequence buckets (earlier pass, commit ccef9a7; no ORT counterpart beyond L128):

kleinhirn WASM at L256 in the same pass: 360.9 ms (358.8-361.2), p95 450.4.

Julia 1 against the upstream WebGPU loop on the fused upstream graph. 100 upstream parity requests; every path picks the same option as PyTorch in 100/100. bench/upstream-julia/bench-graph.html runs the upstream benchmark loop verbatim with the graph file as a parameter. kleinhirn reports ms per decision, upstream ms per request, the same unit:

Against the unchanged upstream graph (earlier pass, commit ccef9a7) kleinhirn f16 ran 24.9 ms per decision, upstream batch 4 23.9 and batch 1 30.2 ms per request.

Download, GPU buffers and peak process memory. Peak memory is the footprint of the benchmark's own browser process tree, median of 3 browser launches with the launch range in parentheses; footprint varies up to ~15 % per launch, so a "uses less" claim stands only where the ranges do not overlap:

With graph capture the ORT WebGPU peaks were 1476 MiB (f16) and 2368 MiB (f32), provisional: three of the six launches started above the load bound.

A load-sensitivity pass found no latency change under pure CPU load up to a 1-minute load of ~10 on this machine; mixed build and browser load did affect earlier runs, so the start-load bound stays.

Second platform (provisional)

On a Modal NVIDIA L4 under Linux (Chromium 151, Vulkan; gate: adapterInfo reports vendor nvidia, architecture lovelace, no fallback adapter) kleinhirn f32 passed every parity gate: small-upstream 100 % argmax (968 cases) with max logit diff 2.35e-5, julia-1 all parity gates green. small-upstream f32 at L128 measured 4.5 ms median per call in a single pass (provisional). That Chromium build does not expose shader-f16, so the engine runs in its f32 fallback by design there. bench/results/nvidia-*.json.

Read: against the best ORT configuration we could produce (fused graph plus graph capture), kleinhirn is 1.9x faster than ONNX Runtime Web in f16 and 2.2x in f32 on WebGPU, and 2.6x on the WASM path (small-upstream, L128); it peaks at 45-59 % of ORT's memory. For Julia 1 the fused fp16 graph makes the upstream loop faster than kleinhirn: 21.5 vs 24.1 ms at equal boundaries, 15.8 vs 20.7 ms between the fastest paths. That graph deviates up to 1.02 in logits from PyTorch fp32, kleinhirn f16 by 0.091. In f32 the two are level (24.3 vs 24.8 ms). kleinhirn's Julia advantage is peak memory (1001 vs 1599-1621 MiB) and fp16 accuracy, not speed. Graph capture was not tried for Julia: it needs fixed shapes, and the upstream loop calls with varying lengths.

The batch API (classifyBatch / decideBatch, batch sizes 1/4/8/16, stride-packed at 64-token increments) produces outputs bit-identical to single calls on the same inputs (batch parity B1-B16, f32 and f16, both models, mixed lengths). Official throughput (ms per decision, same protocol):

For small-upstream, batching matches the single-call path at B1 and helps slightly up to B8; B16 regresses. For julia-1 the batch path is faster than the single path at B1 (21.7 vs 24.9 ms, though the batch number times pre-tokenized inputs while the single-call number includes tokenization) but slower per decision from B4 on: in this throughput pass kleinhirn B4 ran 35.8 ms against an upstream batch-4 25.6 ms from a different pass. The published Julia comparison is the fused-graph pass above. The API is kept for the parity guarantee and for hosts that batch anyway.

Julia 1 re-measurement (PyTorch CPU fp32, K9 suite)

AG News, DAIR Emotion and Banking77 are read from the BTZSC copies, sampled as in jev-benchmarks; MASSIVE from the AmazonScience/massive dataset.

kleinhirn's Julia engine (WebGPU f16, minimum limits) scores the same 1,451/2,000 on Typed Decisions as PyTorch CPU; argmax agreement 1,997/2,000 (bench/results/k9-typed-kleinhirn-julia-f16.json).

GLiNER2.5 zero-shot on the K9 tasks (PyTorch CPU, Jev adapter)

Jev reference values under the pinned protocol are reported, not measured by us: AG News 91/100, DAIR Emotion 48/100 and Banking77 87/100 come from the pinned btzsc-pilot-v1 report (see docs/BENCHMARKS.md); Typed Decisions 72.70 % is listed on the Julia 1 model card ("Evaluation", column "Jev reference") as a supplied comparison reference.

Repository layout

src/       engine: device, weights, tokenizer, WGSL graph, WASM fallback
convert/   checkpoint -> weight format, goldens, numpy reference forward
bench/     measurement pages + Playwright runners + k9 suite + results
tests/     Node tests, corpus, goldens (per model)
tools/     corpus builder, pinned model/dataset downloaders
site/      device measurement page (source)
pages/     prebuilt page served by GitHub Pages, no weights
docs/      ARCHITECTURE, RESEARCH, BENCHMARKS, FINDINGS
models/    checkpoints and weights, never in git

Third-party projects

Every project, model and dataset this README names, with the place its owners publish it. kleinhirn is not affiliated with any of them.

Models measured or supported:

• fastino/gliner2.5-small-v1, -base-v1, -multi-v1 by Fastino; library and paper code in fastino-ai/GLiNER2
• SupersonicLabs/Julia-1, fine-tuned from jhu-clsp/mmBERT-small (JHU-CLSP/mmBERT)
• sentence-transformers/all-MiniLM-L6-v2
• cardiffnlp/twitter-roberta-base-sentiment-latest
• cross-encoder/mmarco-mMiniLMv2-L12-H384-v1
• distilbert/distilbert-base-uncased-finetuned-sst-2-english
• protectai/deberta-v3-base-prompt-injection-v2
• ibm-granite/granite-embedding-small-english-r2
• the other 83 checkpoints: Hugging Face IDs and revisions in data/k28/models.json

Model families:

• BERT: google-research/bert
• ELECTRA: google-research/electra
• MiniLM: microsoft/unilm
• RoBERTa and XLM-R: facebookresearch/fairseq
• DistilBERT: distilbert/distilbert-base-uncased
• DeBERTa: microsoft/DeBERTa
• ModernBERT: AnswerDotAI/ModernBERT

Runtimes, formats and tools:

• ONNX Runtime and ONNX Runtime Web: microsoft/onnxruntime
• PyTorch: pytorch/pytorch
• Hugging Face Hub: huggingface.co
• safetensors: safetensors/safetensors
• WebGPU and WGSL: W3C WebGPU, W3C WGSL
• AssemblyScript: AssemblyScript/assemblyscript
• Emscripten (not used, named for contrast): emscripten-core/emscripten
• Playwright: microsoft/playwright
• GLiNER2.5 CoreML export, from which convert/gliner2_export.py is adapted: FluidInference/gliner2-5-small-coreml

Browsers and platforms:

• Chromium: chromium.org
• Brave: brave/brave-browser
• WebKit: WebKit/WebKit
• Safari: apple.com/safari
• Firefox: mozilla-firefox/firefox
• NVIDIA L4: nvidia.com, run on Modal

Datasets and benchmarks:

• Typed Decisions: LocalLLaMA/typed-decisions
• BTZSC: btzsc/btzsc
• Jev benchmark protocol: AbdelStark/jev-benchmarks
• AG News: fancyzhx/ag_news
• DAIR Emotion: dair-ai/emotion
• Banking77: PolyAI/banking77
• MASSIVE: alexa/massive, data from AmazonScience/massive

Corpus and first user:

• argmin, a German learning platform by the author of kleinhirn: smashburger-dev/argmin, app at smashburger-dev.github.io/argmin. 500 of the 1,000 corpus texts come from its content (CC-BY-4.0).

Contributing

Device results, bug reports and pull requests are welcome; see CONTRIBUTING.md. Anything beyond a small fix starts with an issue. Security problems go through SECURITY.md, not a public issue.

License

Apache-2.0, copyright Noa Katana (see LICENSE). Third-party models, datasets, adapted code and corpus texts keep their own licenses (see NOTICE).