Study documentation and benchmark for various Jev-like products (commercial and open source)
Python
6
30 commits
updated Oct 15, 2026
Measure how reliably decision models apply supplied rules, how their probabilities behave, and what those decisions cost. Compare TypeSafe Jev, Microsoft Decision-1 and OpenAI Decisions with hosted and local Clef/Clef-flash, Decision 3.0, Laya, Kev, Eikos, Jebadiah, Mapika Decider, Simple-Jev and GLiNER, using frozen inputs, inspectable expected answers, raw response archives, and reproducible scoring.
The goal is to find where each model works, where it fails, and whether its quality, latency and operating cost justify using it for classification, support routing, retrieval and similar tasks. Schema-valid JSON is not evidence that a decision is correct.
This repository combines source-based research with our own synthetic experiments. It is not an official vendor benchmark, a production accuracy guarantee, or a demonstration that one model is universally best. See methodology and limitations.
15 configurations, 14,632 valid responses. The same pinned models are 1.7×–10.3× faster in median request latency on this RTX 5090 deployment. Compact scores change by at most two answers out of 168; 99.1% of v2 predictions match across backends. Probability outputs can still move enough to affect confidence thresholds.
Jebadiah leads compact CUDA accuracy (90.5%, 39 ms); Kev 9B leads all v2 conditions combined (441/504, 69 ms); Clef Flash local leads original v1 policy accuracy among these CUDA entrants (118/130, 86 ms v2 median). The two 27B reference models need roughly 49 GiB of weights and do not fit this 32 GiB card. Hosted services were not rerun. Local dollar costs remain unmeasured.
Analysis · Ranking · All paired results · Reproduce and clean up. These ratios compare whole machines and serving stacks, not an isolated CUDA/ROCm software effect.
2,096 new valid responses, one attempt each, with no input loss. Both containers are stopped. Only the requested extremes were tested: d3 (nominal 27B) and d3-edge (0.6B). The flagship scores 167/168 compact (99.4%) and 501/504 across all v2 conditions, matching Perplexity's prediction on every case. It also matches all 520 v1 labels, with 129/130 originals (127/127 after the common ambiguity exclusions). Both large models miss the same English retrieval family. These results cannot distinguish their classification quality.
Edge scores 102/168 compact (60.7%), 287/504 overall, and 85/130 v1 originals. Its compact English result is 57/84, PT-BR 45/84; worked examples reduce its combined score. It uses roughly 1.24 GiB of allocated GPU tensors versus d3's 48.7 GiB. Observed v2 median/p95 is 45/102 ms for Edge and 1,420/2,135 ms for d3. Small size is useful, but its errors require a narrower application-specific evaluation.
No hosted inference cost was added; local operating cost is unmeasured. Microsoft remains the paid hosted cost-benefit candidate for this input mix, while d3 joins Perplexity at the top local quality tier. This test covers text, not the new models' image/video capabilities. Analysis, full ranking, setup.
The following named updates retain their historical scope.
1,048 new valid responses. Microsoft ties Jev at 97.6% compact bilingual accuracy, behind local Perplexity's 99.4%. It scores 80/84 English, 84/84 PT-BR, 493/504 across all conditions, and 124/130 original policy cases.
| Entrant | Compact bilingual | USD / 1,000 compact calls |
|---|---|---|
| Perplexity Decider 27B, local | 99.4% | Local cost unmeasured |
| Microsoft Decision-1, OpenRouter | 97.6% | $0.00645 |
| Jev | 97.6% | $0.01916 |
| Clef | 95.8% | $0.06309 |
| OpenAI Decisions | 90.5% | $0.02331 |
| Jebadiah 4B v2, local | 90.5% | Local cost unmeasured |
| Clef-flash | 87.5% | $0.00999 |
Microsoft becomes the first paid hosted candidate to evaluate for this workload: about 66% cheaper than Jev by reported compact usage. Jev remains faster (255/304 ms median/p95 versus 422/551) and slightly stronger across all conditions. The compact tie does not imply identical errors. Microsoft still mishandles some threshold rules and known-random probability probes. Perplexity retains the highest local quality point estimate; small model resource constraints remain relevant.
Successful usage including the smoke totals $0.011135376. Ten v2 HTTP 429 attempts recover on retry; any separate retry charge and account fees are unverified. Cloudflare allowances and unmeasured local costs qualify the price comparison. Analysis, costs, setup.
The following sections retain the results and recommendations of earlier stages.
99.4% compact bilingual accuracy (167/168) and 501/504 overall. Perplexity scores every PT-BR case correctly and misses one English retrieval family across all three conditions. It also scores 129/130 original policy cases, rising to 127/127 after the same existing disputed-family exclusions used for every model. Jev scores 97.6% compact; Clef 95.8%; OpenAI and Jebadiah 90.5%. The small observed lead over Jev does not establish general superiority.
This local BF16 deployment needs roughly 49 GiB of weights plus runtime memory, and measures 1,547/2,446 ms median/p95 on AMD ROCm with fallback kernels. It is the strongest measured local quality candidate here; Jev remains the leading hosted quality/list-price choice. Local total cost is unmeasured. Its Boolean probabilities still fail known-random-outcome probes, so high classification accuracy does not establish general probability reliability. Full analysis and limitations, reproduction.
2,096 new scored requests, all valid and untruncated. At this earlier stage the comparison had 19 bilingual configurations and 17 original-policy configurations. Julia's 144.3M checkpoint scores 31.5% compact bilingual accuracy on both GPU and CPU, close to the suite's 31.0% uniform-random baseline. GPU median/p95 is 12/25 ms; four-thread CPU is 71/197 ms. Low resource requirements and speed are useful, but these results do not support autonomous difficult policy decisions.
Before the Perplexity addition, Jev (97.6%) and Clef (95.8%) led, followed by the OpenAI/Jebadiah compact tie (90.5%). No new hosted charges; local compute cost is unmeasured. Analysis, cost-benefit and limitations, reproduction protocol.
1,048 new scored requests, all valid, no retries. At this earlier stage the comparison contained
17 bilingual configurations and 15 original-policy configurations. OpenAI uses
gpt-6-luna through the native Decisions endpoint, with the same frozen rules,
option descriptions and evidence. The new usage cost is $0.03475, including
the successful independent smoke test; the $20 account deposit is prepaid credit.
| Model | Compact EN + PT-BR | All three v2 conditions | Listed USD / 1,000 compact calls |
|---|---|---|---|
| Jev | 97.6% | 98.2% | $0.01916 |
| Clef | 95.8% | 96.4% | $0.06309 |
| OpenAI Decisions | 90.5% | 93.1% | $0.02331 |
| Jebadiah 4B v2, local | 90.5% | 86.5% | Local cost unmeasured |
| Clef-flash, hosted | 87.5% | 87.1% | $0.00999 |
OpenAI has the lowest observed hosted v2 median/p95 latency (222/274 ms), but the timed runs occurred on different dates. Jev retains the stronger measured quality/price balance. OpenAI's predicate probabilities preserve known random-event uncertainty well, while its choice probabilities remain biased on those probes. Exact-threshold rules and the critical data-loss support ticket still expose errors.
Full analysis and limitations, v1 policy tables, native adapter and reproduction.
9,392 new scored responses, all valid, zero retries. Together with the first runs, that stage contained 16 bilingual configurations and 14 policy-suite configurations. The new inference ran sequentially on the AMD homelab in about 55 minutes; all benchmark containers are stopped afterward.
The observations in this table describe that earlier sixteen-configuration stage.
| Configuration | Compact bilingual accuracy | Main observation |
|---|---|---|
| Jev | 97.6% | Highest point estimate; full hosted Clef is close at 95.8%. |
| Jebadiah 4B v2 | 90.5% | Highest tested local compact point estimate. |
| Kev 9B | 88.1% | Larger gains with examples/distractors than compact inputs. |
| Mapika Decider 4B / Kev 4B / Flash deployments | 87.5% each | Equal aggregate counts conceal different task/language errors. |
| Simple-Jev with stock Qwen 4B | 74.4% | Fixed prompt/readout baseline; matches Jebadiah on v1 originals but trails on v2. |
| Laya multilingual | 36.9% | Fastest observed local median, about 19 ms; weak on these supplied-rule cases. |
The full sixteen-row ranking also includes Eikos, Kev 0.8B, the original Laya checkpoints and both GLiNER models. GLiNER is a native classification mapping track, without fabricated probability metrics.
Size correlates strongly with compact score in the tested known-size sample (Spearman 0.920), but the Kev series is more informative: 63.1% → 87.5% → 88.1% at 0.8B/4B/9B. The last step is uncertain on compact inputs and more useful with examples or distractors. See size analysis.
The new wave adds no hosted inference charges; local operating cost remains unmeasured. Analysis and cost-benefit recommendations, concrete support-ticket failure, and reproduction protocol.
Seven tasks, 84 new scenarios, paired languages and three conditions: 3,024 new classifications across six entrants, all valid, with no retries. Compact inputs are the primary comparison; examples and longer distractors are reported separately.
| Entrant | Compact English | Compact PT-BR |
|---|---|---|
| Jev | 97.6% | 97.6% |
| Clef hosted | 97.6% | 94.0% |
| Clef-flash hosted | 91.7% | 83.3% |
| Clef-flash local | 89.3% | 85.7% |
| Laya | 44.0% | 41.7% |
| Laya-typed | 50.0% | 35.7% |
Full Clef matched Jev's English total. Local Flash matched Jev on sentiment in both languages and English incident status, but task cells contain only twelve scenarios. Neither Laya checkpoint beat Jev or full Clef in any compact task/language cell. Examples did not consistently help. Base Laya truncated every long PT-BR input, so that stress result cannot isolate language understanding.
Hosted/local Flash now disagree on 19/504 labels despite equal overall accuracy; v1's exact agreement does not generalize. The new hosted run's list-price usage is $0.07236 before allowances, with local operating cost unmeasured. Read the analysis and scoped cost-benefit ranking for paired uncertainty, concrete errors, language token costs and limitations.
13 tasks × 10 original scenarios × four presentations = 520 classifications/model, plus 24 separate uncertainty probes/model. Hosted runs were recorded 9 October 2026; AMD homelab runs on 10 October. All six entrants completed 544 valid responses each.
| Entrant | Originals / all variants | Median client latency | Deployment |
|---|---|---|---|
| Jev 1.13.0 | 96.2% / 95.2% | 253 ms | TypeSafe API |
| Clef | 94.6% / 95.0% | 606 ms | Cloudflare Workers AI |
| Clef-flash | 90.8% / 89.0% | 491 ms | Cloudflare Workers AI |
| Laya | 43.8% / 38.5% | 38 ms | Pinned snapshot, AMD ROCm |
| Laya-typed | 43.8% / 38.5% | 38 ms | Pinned snapshot, AMD ROCm |
| Clef-flash local | 90.8% / 89.0% | 595 ms | Pinned snapshot, AMD ROCm |
Jev and full Clef are effectively tied in classification quality on this small suite. Local Clef-flash matched all 520 hosted classification labels, though probabilities varied slightly. Laya's fastest latency came with much weaker policy-following here. The local main run took about six minutes with already-loaded models; downloads and cold starts are excluded. LAN and WAN timings are not controlled hardware comparisons.
These results do not establish that small open models are weak on every task. All entrants received an untuned explicit-policy track. The separate bilingual expansion tests ordinary semantics, worked examples and distracting context. Clef is itself open-weight: open/proprietary and local/hosted are separate comparisons.
Rates rechecked 11 October 2026: Jev $0.042/M input tokens, Clef $0.24/M, Clef-flash $0.038/M, Microsoft/OpenRouter $0.042/M, OpenAI Decisions $0.10/M. Flash's earlier observed rate was $0.09/M; the dated original valuation is retained. For v1's classification-only input mix, current list values per 1,000 decisions are approximately $0.02093 Jev, $0.07401 Clef, $0.01172 Flash. Sources, arithmetic and error-cost trade-offs.
Cloudflare's available daily free allowance can make small runs free. Actual billing was not verified. Local electricity, hardware and operations were not measured and are never labeled $0. A cheaper API response is not necessarily a cheaper workflow if a wrong label creates review work or damage.
Requires Git and Docker; no API credentials or GPU needed to inspect/recompute results.
git clone https://github.com/akitaonrails/jev-docs-benchmarks.git
cd jev-docs-benchmarks
docker build -t jev-benchmark-runner -f benchmark/docker/Dockerfile.runner benchmark
benchmark/tools/offline.sh -m unittest discover -s /work/tests
benchmark/tools/offline.sh /work/tools/compare.py \
/work/reports/hosted-v1 /work/reports/homelab-v1 --output /reports/comparison-recomputed
benchmark/tools/offline.sh /work/tools/score_v2.py \
/work/reports/hosted-v2 /work/reports/homelab-v2 --output /reports/comparison-v2-recomputed
offline.sh disables network access. To run new inference, configure credentials or
the model server and follow setup.md. Inference requires an explicit
--execute. All benchmark dependencies stay in containers; the client uses no GPU.
| Experiment | Dataset/protocol | Evidence/status |
|---|---|---|
| v1 hosted | Frozen protocol, cases | Archive, analysis |
| v1 local | Local extension, same v1 cases | Archive, environment, verification |
| v2 EN / PT-BR | Protocol, 504 cases/model | Analysis, tables, hosted archive, local archive |
The v2 expansion contains 84 new semantic scenarios, paired EN/PT-BR versions, and compact/example-guided/distractor conditions. Report by task, language and condition; never interpret translations as independent examples or mix v2 into the v1 headline. The installed Laya checkpoints are not its separate multilingual checkpoint.
docs/ Research, interpretation, pricing and setup
benchmark/datasets/ Frozen v1 cases, source policies and errata
benchmark/expansion-v2/ Paired EN/PT-BR source cases and frozen manifest
benchmark/tools/ Stdlib runners, validators, scoring and valuation
benchmark/tests/ Offline correctness checks
benchmark/docker/ Client and legacy workstation image recipes
benchmark/cuda/ CUDA locks, lifecycle, verification and cleanup
benchmark/homelab/ Original ROCm service and pinned downloader
benchmark/next-wave/ New service snapshots, adapters and lifecycle protocol
benchmark/reports/ Published metrics, raw gzip responses and provenance
benchmark/results/ Ignored working outputs from new runs
For model licenses and upstream implementations, follow the source links in alternatives. Check current provider rates and terms before new runs; snapshots in this repository document what was observed at the stated dates.
Study documentation and benchmark for various Jev-like products (commercial and open source)
Python
6
30 commits
updated Oct 15, 2026
Measure how reliably decision models apply supplied rules, how their probabilities behave, and what those decisions cost. Compare TypeSafe Jev, Microsoft Decision-1 and OpenAI Decisions with hosted and local Clef/Clef-flash, Decision 3.0, Laya, Kev, Eikos, Jebadiah, Mapika Decider, Simple-Jev and GLiNER, using frozen inputs, inspectable expected answers, raw response archives, and reproducible scoring.
The goal is to find where each model works, where it fails, and whether its quality, latency and operating cost justify using it for classification, support routing, retrieval and similar tasks. Schema-valid JSON is not evidence that a decision is correct.
This repository combines source-based research with our own synthetic experiments. It is not an official vendor benchmark, a production accuracy guarantee, or a demonstration that one model is universally best. See methodology and limitations.
15 configurations, 14,632 valid responses. The same pinned models are 1.7×–10.3× faster in median request latency on this RTX 5090 deployment. Compact scores change by at most two answers out of 168; 99.1% of v2 predictions match across backends. Probability outputs can still move enough to affect confidence thresholds.
Jebadiah leads compact CUDA accuracy (90.5%, 39 ms); Kev 9B leads all v2 conditions combined (441/504, 69 ms); Clef Flash local leads original v1 policy accuracy among these CUDA entrants (118/130, 86 ms v2 median). The two 27B reference models need roughly 49 GiB of weights and do not fit this 32 GiB card. Hosted services were not rerun. Local dollar costs remain unmeasured.
Analysis · Ranking · All paired results · Reproduce and clean up. These ratios compare whole machines and serving stacks, not an isolated CUDA/ROCm software effect.
2,096 new valid responses, one attempt each, with no input loss. Both containers are stopped. Only the requested extremes were tested: d3 (nominal 27B) and d3-edge (0.6B). The flagship scores 167/168 compact (99.4%) and 501/504 across all v2 conditions, matching Perplexity's prediction on every case. It also matches all 520 v1 labels, with 129/130 originals (127/127 after the common ambiguity exclusions). Both large models miss the same English retrieval family. These results cannot distinguish their classification quality.
Edge scores 102/168 compact (60.7%), 287/504 overall, and 85/130 v1 originals. Its compact English result is 57/84, PT-BR 45/84; worked examples reduce its combined score. It uses roughly 1.24 GiB of allocated GPU tensors versus d3's 48.7 GiB. Observed v2 median/p95 is 45/102 ms for Edge and 1,420/2,135 ms for d3. Small size is useful, but its errors require a narrower application-specific evaluation.
No hosted inference cost was added; local operating cost is unmeasured. Microsoft remains the paid hosted cost-benefit candidate for this input mix, while d3 joins Perplexity at the top local quality tier. This test covers text, not the new models' image/video capabilities. Analysis, full ranking, setup.
The following named updates retain their historical scope.
1,048 new valid responses. Microsoft ties Jev at 97.6% compact bilingual accuracy, behind local Perplexity's 99.4%. It scores 80/84 English, 84/84 PT-BR, 493/504 across all conditions, and 124/130 original policy cases.
| Entrant | Compact bilingual | USD / 1,000 compact calls |
|---|---|---|
| Perplexity Decider 27B, local | 99.4% | Local cost unmeasured |
| Microsoft Decision-1, OpenRouter | 97.6% | $0.00645 |
| Jev | 97.6% | $0.01916 |
| Clef | 95.8% | $0.06309 |
| OpenAI Decisions | 90.5% | $0.02331 |
| Jebadiah 4B v2, local | 90.5% | Local cost unmeasured |
| Clef-flash | 87.5% | $0.00999 |
Microsoft becomes the first paid hosted candidate to evaluate for this workload: about 66% cheaper than Jev by reported compact usage. Jev remains faster (255/304 ms median/p95 versus 422/551) and slightly stronger across all conditions. The compact tie does not imply identical errors. Microsoft still mishandles some threshold rules and known-random probability probes. Perplexity retains the highest local quality point estimate; small model resource constraints remain relevant.
Successful usage including the smoke totals $0.011135376. Ten v2 HTTP 429 attempts recover on retry; any separate retry charge and account fees are unverified. Cloudflare allowances and unmeasured local costs qualify the price comparison. Analysis, costs, setup.
The following sections retain the results and recommendations of earlier stages.
99.4% compact bilingual accuracy (167/168) and 501/504 overall. Perplexity scores every PT-BR case correctly and misses one English retrieval family across all three conditions. It also scores 129/130 original policy cases, rising to 127/127 after the same existing disputed-family exclusions used for every model. Jev scores 97.6% compact; Clef 95.8%; OpenAI and Jebadiah 90.5%. The small observed lead over Jev does not establish general superiority.
This local BF16 deployment needs roughly 49 GiB of weights plus runtime memory, and measures 1,547/2,446 ms median/p95 on AMD ROCm with fallback kernels. It is the strongest measured local quality candidate here; Jev remains the leading hosted quality/list-price choice. Local total cost is unmeasured. Its Boolean probabilities still fail known-random-outcome probes, so high classification accuracy does not establish general probability reliability. Full analysis and limitations, reproduction.
2,096 new scored requests, all valid and untruncated. At this earlier stage the comparison had 19 bilingual configurations and 17 original-policy configurations. Julia's 144.3M checkpoint scores 31.5% compact bilingual accuracy on both GPU and CPU, close to the suite's 31.0% uniform-random baseline. GPU median/p95 is 12/25 ms; four-thread CPU is 71/197 ms. Low resource requirements and speed are useful, but these results do not support autonomous difficult policy decisions.
Before the Perplexity addition, Jev (97.6%) and Clef (95.8%) led, followed by the OpenAI/Jebadiah compact tie (90.5%). No new hosted charges; local compute cost is unmeasured. Analysis, cost-benefit and limitations, reproduction protocol.
1,048 new scored requests, all valid, no retries. At this earlier stage the comparison contained
17 bilingual configurations and 15 original-policy configurations. OpenAI uses
gpt-6-luna through the native Decisions endpoint, with the same frozen rules,
option descriptions and evidence. The new usage cost is $0.03475, including
the successful independent smoke test; the $20 account deposit is prepaid credit.
| Model | Compact EN + PT-BR | All three v2 conditions | Listed USD / 1,000 compact calls |
|---|---|---|---|
| Jev | 97.6% | 98.2% | $0.01916 |
| Clef | 95.8% | 96.4% | $0.06309 |
| OpenAI Decisions | 90.5% | 93.1% | $0.02331 |
| Jebadiah 4B v2, local | 90.5% | 86.5% | Local cost unmeasured |
| Clef-flash, hosted | 87.5% | 87.1% | $0.00999 |
OpenAI has the lowest observed hosted v2 median/p95 latency (222/274 ms), but the timed runs occurred on different dates. Jev retains the stronger measured quality/price balance. OpenAI's predicate probabilities preserve known random-event uncertainty well, while its choice probabilities remain biased on those probes. Exact-threshold rules and the critical data-loss support ticket still expose errors.
Full analysis and limitations, v1 policy tables, native adapter and reproduction.
9,392 new scored responses, all valid, zero retries. Together with the first runs, that stage contained 16 bilingual configurations and 14 policy-suite configurations. The new inference ran sequentially on the AMD homelab in about 55 minutes; all benchmark containers are stopped afterward.
The observations in this table describe that earlier sixteen-configuration stage.
| Configuration | Compact bilingual accuracy | Main observation |
|---|---|---|
| Jev | 97.6% | Highest point estimate; full hosted Clef is close at 95.8%. |
| Jebadiah 4B v2 | 90.5% | Highest tested local compact point estimate. |
| Kev 9B | 88.1% | Larger gains with examples/distractors than compact inputs. |
| Mapika Decider 4B / Kev 4B / Flash deployments | 87.5% each | Equal aggregate counts conceal different task/language errors. |
| Simple-Jev with stock Qwen 4B | 74.4% | Fixed prompt/readout baseline; matches Jebadiah on v1 originals but trails on v2. |
| Laya multilingual | 36.9% | Fastest observed local median, about 19 ms; weak on these supplied-rule cases. |
The full sixteen-row ranking also includes Eikos, Kev 0.8B, the original Laya checkpoints and both GLiNER models. GLiNER is a native classification mapping track, without fabricated probability metrics.
Size correlates strongly with compact score in the tested known-size sample (Spearman 0.920), but the Kev series is more informative: 63.1% → 87.5% → 88.1% at 0.8B/4B/9B. The last step is uncertain on compact inputs and more useful with examples or distractors. See size analysis.
The new wave adds no hosted inference charges; local operating cost remains unmeasured. Analysis and cost-benefit recommendations, concrete support-ticket failure, and reproduction protocol.
Seven tasks, 84 new scenarios, paired languages and three conditions: 3,024 new classifications across six entrants, all valid, with no retries. Compact inputs are the primary comparison; examples and longer distractors are reported separately.
| Entrant | Compact English | Compact PT-BR |
|---|---|---|
| Jev | 97.6% | 97.6% |
| Clef hosted | 97.6% | 94.0% |
| Clef-flash hosted | 91.7% | 83.3% |
| Clef-flash local | 89.3% | 85.7% |
| Laya | 44.0% | 41.7% |
| Laya-typed | 50.0% | 35.7% |
Full Clef matched Jev's English total. Local Flash matched Jev on sentiment in both languages and English incident status, but task cells contain only twelve scenarios. Neither Laya checkpoint beat Jev or full Clef in any compact task/language cell. Examples did not consistently help. Base Laya truncated every long PT-BR input, so that stress result cannot isolate language understanding.
Hosted/local Flash now disagree on 19/504 labels despite equal overall accuracy; v1's exact agreement does not generalize. The new hosted run's list-price usage is $0.07236 before allowances, with local operating cost unmeasured. Read the analysis and scoped cost-benefit ranking for paired uncertainty, concrete errors, language token costs and limitations.
13 tasks × 10 original scenarios × four presentations = 520 classifications/model, plus 24 separate uncertainty probes/model. Hosted runs were recorded 9 October 2026; AMD homelab runs on 10 October. All six entrants completed 544 valid responses each.
| Entrant | Originals / all variants | Median client latency | Deployment |
|---|---|---|---|
| Jev 1.13.0 | 96.2% / 95.2% | 253 ms | TypeSafe API |
| Clef | 94.6% / 95.0% | 606 ms | Cloudflare Workers AI |
| Clef-flash | 90.8% / 89.0% | 491 ms | Cloudflare Workers AI |
| Laya | 43.8% / 38.5% | 38 ms | Pinned snapshot, AMD ROCm |
| Laya-typed | 43.8% / 38.5% | 38 ms | Pinned snapshot, AMD ROCm |
| Clef-flash local | 90.8% / 89.0% | 595 ms | Pinned snapshot, AMD ROCm |
Jev and full Clef are effectively tied in classification quality on this small suite. Local Clef-flash matched all 520 hosted classification labels, though probabilities varied slightly. Laya's fastest latency came with much weaker policy-following here. The local main run took about six minutes with already-loaded models; downloads and cold starts are excluded. LAN and WAN timings are not controlled hardware comparisons.
These results do not establish that small open models are weak on every task. All entrants received an untuned explicit-policy track. The separate bilingual expansion tests ordinary semantics, worked examples and distracting context. Clef is itself open-weight: open/proprietary and local/hosted are separate comparisons.
Rates rechecked 11 October 2026: Jev $0.042/M input tokens, Clef $0.24/M, Clef-flash $0.038/M, Microsoft/OpenRouter $0.042/M, OpenAI Decisions $0.10/M. Flash's earlier observed rate was $0.09/M; the dated original valuation is retained. For v1's classification-only input mix, current list values per 1,000 decisions are approximately $0.02093 Jev, $0.07401 Clef, $0.01172 Flash. Sources, arithmetic and error-cost trade-offs.
Cloudflare's available daily free allowance can make small runs free. Actual billing was not verified. Local electricity, hardware and operations were not measured and are never labeled $0. A cheaper API response is not necessarily a cheaper workflow if a wrong label creates review work or damage.
Requires Git and Docker; no API credentials or GPU needed to inspect/recompute results.
git clone https://github.com/akitaonrails/jev-docs-benchmarks.git
cd jev-docs-benchmarks
docker build -t jev-benchmark-runner -f benchmark/docker/Dockerfile.runner benchmark
benchmark/tools/offline.sh -m unittest discover -s /work/tests
benchmark/tools/offline.sh /work/tools/compare.py \
/work/reports/hosted-v1 /work/reports/homelab-v1 --output /reports/comparison-recomputed
benchmark/tools/offline.sh /work/tools/score_v2.py \
/work/reports/hosted-v2 /work/reports/homelab-v2 --output /reports/comparison-v2-recomputed
offline.sh disables network access. To run new inference, configure credentials or
the model server and follow setup.md. Inference requires an explicit
--execute. All benchmark dependencies stay in containers; the client uses no GPU.
| Experiment | Dataset/protocol | Evidence/status |
|---|---|---|
| v1 hosted | Frozen protocol, cases | Archive, analysis |
| v1 local | Local extension, same v1 cases | Archive, environment, verification |
| v2 EN / PT-BR | Protocol, 504 cases/model | Analysis, tables, hosted archive, local archive |
The v2 expansion contains 84 new semantic scenarios, paired EN/PT-BR versions, and compact/example-guided/distractor conditions. Report by task, language and condition; never interpret translations as independent examples or mix v2 into the v1 headline. The installed Laya checkpoints are not its separate multilingual checkpoint.
docs/ Research, interpretation, pricing and setup
benchmark/datasets/ Frozen v1 cases, source policies and errata
benchmark/expansion-v2/ Paired EN/PT-BR source cases and frozen manifest
benchmark/tools/ Stdlib runners, validators, scoring and valuation
benchmark/tests/ Offline correctness checks
benchmark/docker/ Client and legacy workstation image recipes
benchmark/cuda/ CUDA locks, lifecycle, verification and cleanup
benchmark/homelab/ Original ROCm service and pinned downloader
benchmark/next-wave/ New service snapshots, adapters and lifecycle protocol
benchmark/reports/ Published metrics, raw gzip responses and provenance
benchmark/results/ Ignored working outputs from new runs
For model licenses and upstream implementations, follow the source links in alternatives. Check current provider rates and terms before new runs; snapshots in this repository document what was observed at the stated dates.