README
Experiments
Random vibe-experiments — one-off explorations and data products. Each
subdirectory is self-contained: scripts, data, and a results file
(RESULTS.md for pipeline runs, README.md for build/recipe artifacts,
or the artifact itself when it speaks for itself, like an HTML page).
Trust conventions. ANCHORS.md registers every published
constant in use with its covered range, because a range gap is invisible from
inside a green run. Each experiment should carry an ERRORS.md (what was wrong,
how it was caught, which direction it pushed the conclusion — the base rate is
the most useful calibration number about a body of work) and a recheck.py
(a sub-5-minute fixture that checks the prose against the artifacts, so the
writeup and the data cannot drift apart between full rebuilds).
remex-vs-higgs-ablation/ carries all three and is the reference shape.
Results are reported as they came out. Several of these are negative results, one is a correction of earlier rejected work, and one turned out to reproduce a 2004 paper rather than extend it — all labelled as such rather than quietly dropped.
Migrated from oaustegard/claude-workspace/experiments/, which is a
session-boot repo and was the wrong home for 37 research projects.
Index
| Experiment | Started | Status | Results | Origin |
|---|---|---|---|---|
jev-tag-encoder/ | 2026-09-23 | done — one Jev call per document answering 256 "The document is about ." Nouls works as a zero-shot topic labeller and fails as an RRF leg beside dense; without dense, Jev carries an interpretable pipeline. 24 arXiv categories mapped to tags: micro-F1 0.636 [0.616, 0.659] with no labels, between gemini-embedding-2 probes on 50 (0.517) and 200 labels (0.714). A probe on the 256 raw probabilities beats the dense probe at 50 labels (0.566) and trails at 500 (0.697 vs 0.778); standardizing the probabilities first costs up to 0.22 F1. On SciFact the general tags alone score nDCG@10 0.18 to 0.34 and as a third RRF leg cost 0.087; a 256-tag taxonomy fitted to SciFact (KMeans clusters named by Gemini) lifts that to 0.386 alone and a 0.055 cost, still below BM25 + dense, and a perfect labeller of it (query-to-centroid cosine) scores 0.183. Round 4, dense ruled out: BM25 + λ·fitted-tag score (z-scored sum, not RRF) goes 0.662 → 0.699, and reranking its top 20 with one pairwise Noul ("The document provides evidence that supports or refutes the claim.") reaches 0.763, against RRF(BM25, dense) 0.774. A 3-bit code (sign bit plus logit-spaced confidence buckets) matches the floats on retrieval (−0.011); 1 bit keeps zero-shot F1 and loses 0.18 micro-AP. 7,449 calls, p50 0.59 s, ~$1.30 total, paced at 48/min under the AI Gateway's 50/min limit; Round 3's 5,483 direct TypeSafe calls ran at 600/min, p50 0.46 s; reruns flip 5 of 7,680 values across 0.5 | RESULTS.md + tags.txt + jev.py + data.py + encode_all.py + embed.py + fit.py + classify.py + retrieve.py + ablate.py + quant.py + quant_eval.py + probe_codes.py + build_domain_tags.py + tags_scifact.txt + encode_domain.py + retrieve_domain.py + rerank.py + recheck.py + vectors/*.parquet + results/*.json | #98: the claude.ai probe (c98b700) showed 256 Nouls fit one call; run the six-step spec at corpus scale from code. Oskar mid-run: "test a 2 bit and 3 bit version (using confidence buckets for the extra bits)"; then: does a per-domain tag set change the SciFact result? And with dense out, what is the best interpretable pipeline? Prior art: QA-Emb (arXiv 2405.16714), binary answers to task-generated questions. |
subagent-context-filter/ | 2026-09-22 | done — writing only the task and letting Jev pick the parent transcript's chunks works on fact-carrying delegations: 99/105 facts with ~20k tokens of selected chunks vs 102/105 with the full ~81k-token transcript (paired −2.6 to −2.9 points, CI reaching zero) and 82/105 with a brief the same model wrote. The brief's whole gap is 5 of 32 tasks where it pointed the subagent at a lookup instead of stating the fact; elsewhere every arm is within two facts. Filter median 1.1 s and $0.002 per delegation; the brief took 528 output tokens and 23 s. Paging at 4x view ties the condensed window on these ≤97-chunk sessions and fixed a 181-chunk case. Found: TypeSafe's edge WAF blocks some transcript content, surfaced by the AI Gateway as HTTP 402 "Payment error from model using BYOK"; the filter bisects around it. Shipped as the delegating-with-context skill + PreToolUse hook | RESULTS.md + ERRORS.md (5) + chunking.py + jevfilter.py + context_hook.py + preview.py + run_eval.py + adjudicate.py + analyze.py + recheck.py + results/summary.json (counts only; transcripts are private) | Oskar: "wrap subagents in a jev-aided transcript-parser which per transcript chunk determined whether that chunk should be included for a given subagent task", then "page in 30k token increments" and "updatedInput is too late though? This is why a skill is needed". 8 archived sessions, 32 tasks written blind by Sonnet labellers, five arms on a tool-less claude -p executor, regex scoring plus a blind adjudicator validated in-batch (TPR 100%, TNR 95%). Pilot and design in memories 90ac1ab8, 9557f376, cb532ad0. |
sparseup-tag-probe/ | 2026-09-21 | done — negative for the learned expansion. A linear map from SPARSEUP's 50k-dimension sparse vector to Muninn's 325 human tags exists (micro-AP 0.612 vs 0.043 prior), and word + char TF-IDF over the same text carries more of it: 0.664, paired difference −0.051 [−0.059, −0.044]; on the encoder's own 512-token window TF-IDF still leads by 0.044. P@5 ties. gte-small is below both (0.576). Binarizing the SPARSEUP vector costs 0.021 micro-AP and 0.008 P@5. Untrained, 16.4% of tag instances appear as an exact expansion token at k = 32 (18.9% with every BPE piece); 227 of 325 tags are two or more pieces. The pre-registered micro-F1@0.5 ordered the arms the other way and was measuring calibration. Round 2, the 2024 map step as the first layer: a binary vector over K spaCy noun phrases selected per fold by frequency reaches micro-AP 0.369 at K = 512, 0.453 at 2,048 and 0.481 with all 7,655; word unigrams beat phrases at every K (0.573 with all 16,316) and the embedding-dissimilarity filter costs up to 0.035. Round 3, retrieval: a binary vector over all 5,827 tags (5.7 active per memory) finds cited memories as well as gte-small (R@10 0.667 vs 0.648, tie), holds on the recency-controlled half, and fusing the two beats either (+0.059); PMI co-occurrence expansion never helps and hurts at weight >= 0.5; TF-IDF leads every single arm (0.808) on refs written after lexical recall. Round 4: gemini-3.5-flash-lite re-tagging 300 of those memories retrieves at R@10 0.331 with a frequent-tag hint prompt and 0.461 when asked for specific names, against 0.697 for the original tags; the original tags' retrieval lives in their rare, specific tags (df < 20 alone: 0.742; df >= 20 alone: 0.309). Round 5: shown the tags of its five nearest older memories, flash-lite reaches 0.565 and 0.34 Jaccard with the originals; copying those neighbours' tags with no model reaches 0.581 at k = 10, so the context, not the model, carries the gain. Round 6: with the cited memories' tags in the prompt, flash-lite reaches 0.629 forward (−0.067, CI touching zero) and 0.617 on the leak-free reverse direction against 0.674 for the originals and 0.626 for gte-small; a no-model union of context tags matches it at R@10 and loses on MRR | RESULTS.md + PLAN.md + ERRORS.md (5) + fixture.json (ids, cleaned tags, hashes; no memory text) + build_fixture.py + fetch_text.py + encode.py + arm1_recall.py + probe.py + select.py + make_tables.py + recheck.py + phrase_vocab.py + retrieval.py + tag_model.py + flashlite_tags.py + results/*.json | muninn-utilities#137, from Oskar's 2024 binary tag-vector idea meeting Tom Aarsen's SPARSEUP thread (memory 5312cc75): if a SPLADE-style encoder's dimensions are vocabulary tokens, do they line up with a human tag vocabulary? 3,457 memories (37 private excluded), 3,315 labelled, one shared stratified 5-fold, one-vs-rest LR, C swept {1…10000} and selected per arm by micro-AP because TF-IDF and gte-small rows are unit-norm and SPARSEUP rows are not. Eight predictions, six wrong on F1@0.5, four of those right once the threshold is removed. Two Sonnet workers wrote the arm and check scripts; a scheduled adversarial read found the threshold artefact and the truncation asymmetry. |
modernbert-bidirectionality/ | 2026-09-20 | done — two results that disagree, and the trained one wins. Ablation on the trained encoder: right-context dependence is redundant per layer (any one of ModernBERT-base's layers 1–21 causal costs ≤ 0.14 nats) but not confinable (the 8 global layers alone retain 33%), and local rather than anchored (an 8-token lookahead in every layer retains 79%; [CLS]/[SEP] alone is worse than nothing; punctuation ties the content-free masked positions). Adaptation from ettin-decoder-32m, 6M tokens per arm: bidirectional attention in the 4 global layers with the rest causal ends at 2.945 nats, full bidirectionality 2.996, the 8-token lookahead 3.164, causal control 4.181. The dQwen3.5 cut is at least as good as flipping every layer, and the untrained ablation ranked the two cuts backwards | RESULTS.md + PLAN.md + PLAN-adapt.md + PRIOR-ART.md + probe.py + masks.py + adapt.py + results/ | CCotw handoff from the claude.ai session (Muninn memory 879c3575, title only survived): does non-causal dependence concentrate in few layers and few positions?, motivated by dQwen3.5 making only the attention quarter of a Gated-DeltaNet hybrid bidirectional. Per-layer boolean masks injected by forward pre-hook; 120 arms on ModernBERT-base and 69 on ettin-encoder-32m over 9,728 masked wikitext tokens. The pre-registered all-causal reference collapsed to 23.5 nats (above uniform), so an in-distribution left-only reference was added after the smoke. Oskar then asked for the training test: five attention patterns adapted from the encoder's causal twin at equal budget, six predictions pre-registered, two wrong, both the ones the ablation had motivated. Training repairs a layer cut completely (0.43 untrained → 1.04 of the reachable benefit) and a position cut only partly (0.65 → 0.86). |
msd-context-classifier/ | 2026-09-20 | done, three rounds plus checks: the paper filter's 0.98 AUC was the negative sampling — against PubMed's own similar-articles it is 0.82, and the uncurated modern population (9,754 PMC full-text MSD papers) against its neighbours is 0.75 on SPECTER2 with every nonlinear classifier lower citation-graph features adding three points at most, and TF-IDF + LR within two points of the embedding (BM25 nearest-known-paper retrieval is below chance against the neighbours, which are the lexically nearest papers by construction); SPECTER2 is a recall filter, not a classifier: full-text search first, embeddings only over the non-open remainder. Round 1 (classifier): a seven-way classifier over mesoscale.com's site sections scores 0.91–0.96 on held-out pages and collapses to 0.21–0.35 on question-shaped inputs, routing 65–76% of questions to support; 140 model-written questions train a frozen gte-small probe to 0.78 on questions, 617 pages to 0.25. Round 2 (vocabulary, RESULTS-vocab.md): the site's 4,567 terms fragment to 3.4 BPE pieces; two epochs of continued pretraining on the 4,760-page crawl take ettin-32m's whole-term recovery from 0.32 to 0.69, whole-term masking to 0.83, 300 added tokens to 0.75 and −5% tokens/page; mean-pooled retrieval stays at chance and classifiers within two points of stock. Round 3 (paper filter, RESULTS-paper.md): "does this paper relate to MSD?" from 1,004 bibliography abstracts vs 2,466 PubMed negatives — every fine-tuned encoder AUC 0.98, 91–95% recall on the 89% of positives whose abstracts never name the platform at 8–14% false positives on topic-matched papers; stock and adapted ettin-32m within noise, ettin-150m no better. The vocabulary lives in methods sections PubMed does not serve; the abstract-level task is topic recognition | RESULTS.md + RESULTS-vocab.md + RESULTS-paper.md + three PLAN*.md + ERRORS.md (16 entries) + vocab/ (4,567-term list, 300 added tokens; adapted checkpoints on branch msd-vocab-weights) + train.py + dapt.py + mlm_ppl.py + retrieval_eval.py + paper_*.py + results/*.json | Oskar: "I'm interested in finetuning to allow for a niche bioscience context classification. Consider for example the public content for mesoscale.com"; "I was looking more for finetuning the vocabulary"; "One of our challenges is: does this paper relate to MSD?" Labels from URL paths and from MSD's own bibliography; corpora fetched by tracked background jobs; questions written by workers from descriptions and titles alone. Twenty-four pre-registered predictions across three rounds, nineteen wrong or half-wrong, all expecting adaptation and vocabulary to show up downstream in proportion to what was learned. Token-level masked-LM loss cannot see a vocabulary gap (one masked piece of IL-6 is predicted from the other two), so the diagnostic masks whole terms. BioClinical ModernBERT (53B biomedical tokens) ties plain ModernBERT-base on the catalog's terms. Ship: regex on the 11% of papers that name the platform, stock fine-tuned ettin-32m on the rest; next, real triage decisions and a full-text arm. |
encoder-platform-survey/ | 2026-09-20 | done — survey plus one measured arm. No open Jev-class model is a base to build on: every clone is under a week old, single-author, and compares its own fine-tune against Jev zero-shot; the one same-benchmark external test puts a 150M encoder at 48% against Jev's 91%. Jev itself is a prefill-only causal decoder (~10B active by black-box probe), Sonnet-accurate on TypeSafe's own evals (67.8% vs Sonnet 5 67.8%, Opus 5 73.1%). Recommendation: fixed-schema classifiers fine-tuned on frontier labels, on Ettin encoders (MIT, beat ModernBERT at every size), int8 ONNX on CPU | RESULTS.md + ERRORS.md + ladder.py + cut_heads.py + ladder_t{1,4}.json | Oskar: "Survey the field and determine a) if there's a solid version to base further work on or b) if we should endeavor to create one of our own and if so c) what platform." Use cases: model routing, compliance gates, first-round paper filters. Surveyed 19 clones (Latent.Space, awesome-jev, 17 Hub checkpoints), the four prior reviews in memory, TypeSafe's evals site and Archer Hume's 10k-call probe. Measured a CPU latency ladder of int8 encoder exports with the masked-LM head cut off (the first run timed the vocab head and inflated mmBERT-small 3x): ettin-17m 6 ms and ettin-32m 12 ms per 128 tokens on 4 vCPU, ModernBERT-base 46 ms, ModernBERT-large 135 ms; at 512 tokens 27 / 63 / 312 / 662 ms. Zero-shot dynamic-schema decisions need the pretraining a 10B decoder has and a 150M encoder does not — the same mechanism hypothetical-classification measured. All three named use cases are fixed-schema and frontier-labellable, which is where 150M encoders score 85–95% in-domain. First experiment proposed: the paper filter, 1,500 Fable-labelled abstracts with a 150-row human check, zero-shot GLiClass and a frozen-embedding probe as floors, recall at 95% precision and coverage at a 5% error budget as the metrics. |
noncommutative-composition/ | 2026-09-17 | done — negative. The commutator K = h(AB) − h(BA) of a meaning-changing composition (milk chocolate / chocolate milk) is 1.2–1.5× the commutator of an order-inert one (milk and chocolate) at mid/late layers of SmolLM2-135M, Qwen2.5-0.5B and Qwen2.5-1.5B, under the registered 1.5× on every model and layer; random unrelated nouns in the same carriers give 1.0–1.3×, and reversing two adjacent nouns in a bag-of-words frame moves the residual as much as reversing a compound. The direct analogue of the SL(n) paper's Flickr ablation (P5) comes out sign-reversed on all three models: the additive model fits the compound better (R² 0.76–0.91) than the conjunction (0.67–0.80). 6 registered predictions per model: P1 WRONG, P4 WRONG, P5 WRONG everywhere; P2 RIGHT; P3 RIGHT but passed by any vector against a 3-d subspace; P6 PARTIAL/WRONG | RESULTS.md + run.py + results/*.json + 6 plots | experiments#97, the Between-the-Spokes follow-up to arXiv:2609.15083 (SL(n) representation learning): does an LLM's residual stream compose knowledge noncommutatively in the paper's sense, generating directions that commutative addition cannot reach, more so where English order carries meaning? Three carriers × three paraphrases × 20 compound pairs + 20 random-noun pairs, residual read at the final : token at every layer, 12 three-word compounds over all 6 orders, 1000-draw paired permutation tests, a real-vs-random interaction test. Every reversal is noncommutative by about the same amount whatever the nouns mean; the compounding-specific excess is a 1.1–1.3× modulation on the Qwen models (p 0.01–0.08) and nothing on SmolLM2, carried by one of the three carriers (I saw a {A} {B}: alone clears 1.5×). Commutators are pair-specific (PC1 13–21%) in every carrier, so P2 says nothing about semantics. Three design flaws surfaced and are reported both ways: the CONJUNCTION basis has no carrier for and (a post-hoc frame-mean term narrows P5 without flipping it), P3's 70% threshold is passed by construction at d ≥ 576, and a one-token read is dominated by the token before it. The 1.5B run was OOM-killed once for holding 3780 × 151,936 log-probs beside the fp32 model. |
rht-operator-native/ | 2026-09-17 | done — positive. A C kernel applying remex's randomized Hadamard plan directly is 2–140x faster than the materialized matrix under sgemm in every measured cell from d=1024 up, on x86, ARM and Apple Silicon, over four CI runs; d=768 wins except batches of 128–256 on x64, where the serial cutoff is too high; d=384 is a split. Its output hashes identically on all four machines where dense output differs on every one. Reproducible codes also need a one-line codebook change: boundaries taken from float64 centroids drift with NumPy's SIMD level | RESULTS.md + ERRORS.md + rht_kernel.c + rhtop.py + ci_run.py + ci/*.json + make_tables.py + recheck.py | From a conversation about arXiv:2609.15083 (SL(n) representation learning) and what remex's rotation could still yield. remex#72 closed a NumPy operator form as won't-do and named a compiled FWHT as the untested lever. Measured on the 1-vCPU container plus three GitHub runners via .github/workflows/rht-operator-native.yml, which commits each machine's JSON back. Nine errors logged, four of which biased the comparison against the operator. Proposed remex changes are filed as remex#86 and not applied. |
deepseek-batch/ | 2026-09-13 | recipe, unrun. Overnight batch coding harness for a local model, shaped by the one number that decides viability: an agent turn is prefill-dominated, so at 30k context 800 t/s prompt processing is ~75 s/turn and ~20 tasks a night while 40 t/s is ~13 min/turn and two. Oskar's M5 Max 128 GB at 2-bit does 17 t/s decode, sitting on antirez's published ds4 ceiling of 16, and ~40 t/s prefill against his ~800 — a 20× gap worth more than any harness design. Both published 2-bit V4.1-Flash numbers are self-reports, and evanwtf/local-llm#321 is open asking for exactly this machine's reproduction | README.md + deepseek-batch.sh + example.task | Oskar, offloading paid-token Claude work to a local model: "I want to be able to schedule batch like work for this model so that it can run during all hours when I'm not working on the computer myself." mini-swe-agent, one process per task, over ds4's OpenAI-compatible server, chosen because its message list only appends — no compaction anywhere in DefaultAgent.run(), so the prompt prefix stays byte-stable and ds4's prefix cache can work. Aider was the first choice and was dropped: last push 2026-05-22, 114 days stale, 1,865 open issues, and its built-in --auto-test loop is the weaker shape anyway per harness-bench (8/12 feeding tests back vs 11/12 letting the agent run them). OpenCode rejected on its own issue #16589 (stalls unattended); OpenHands rejected despite being very much alive, because its context condenser rewrites the middle of the prompt and forces a full re-prefill. The design point is the gate, not the agent: at 2-bit (one write-up puts top-token agreement at 78%) an unsupervised model produces confident garbage at scale, so every task carries a machine-checkable acceptance command, the baseline is checked before the agent runs, and the full suite runs after for collateral damage. Also corrects a 2026-09-10 piece asserting V4.1-Flash cannot run on 128 GB at any quantization — false on the day it published. Nothing measured locally; bash -n is the only check that has passed. |
query-rewrite-lexical-arm/ | 2026-09-10 | done. BM25 alone scores 73.62 HIT@10 on EnterpriseRAG-Bench where arXiv:2609.05637 reports 39.22 for the dense baseline it calls deliberately strong and 55.74 for its all-method oracle. That ordering is not new — the benchmark's own paper (Sun et al., arXiv:2605.05253), which arXiv:2609.05637 cites, already publishes BM25 at 68.4% document recall against 46.0% for text-embedding-3-large; this run replicates it to within 0.34 with a matched analyzer. The finding is the omission: bm25, lexical, sparse, hybrid, keyword and tf-idf appear zero times in the paper. BM25 rescues 63.2% of the dense arm's misses against 9.0% the other way; fusing dense in changes nothing (−0.43, p=0.639) and an S4 rewrite leg on top costs 2.13 (p=0.048), the sign PLAN.md predicted from hybrid-code-index. The rewrite still rescues 12.1% of the hybrid's misses, inside the paper's own 10.1–14.8% band | RESULTS.md + PLAN.md + ERRORS.md + bm25_sparse.py + run_arm_sparse.py + run_arm_dense.py + analyze.py + overlap.py + make_tables.py + recheck.py + results/ | Muninn, from reviewing arXiv:2609.05637 (ServiceNow, EMNLP 2026) at Oskar's request. The paper varies six query-rewriting strategies under one fixed dense pipeline and reads a +12.5 HIT@10 enterprise gain as complementarity between rewrites; no configuration anywhere retrieves lexically. Pre-registered before measurement, including a weakened thesis after the paper's own Appendix Table 7 was found to account for its result without any missing arm. Thirteen arms, 470 questions, 511,962 documents, no LLM judge — expected_doc_ids are benchmark-supplied. Six errors logged, one of them an OOM I caused by running an analysis beside a live arm. |
threejs-lion/ | 2026-09-10 | done | index.html (interactive, one file) + render.png + live page | spike — Oskar: "Using Three.js reproduce this image", a generated stock cartoon lion on black. Procedural: spheres for body and skull, ~3,000 swept-tube hair strands in one buffer per layer, a lathed body under the mane so gaps read as depth, a canvas-gradient environment for the falloff. Verified by headless Chromium renders against the reference at each step. The source image's second tail tuft is a generation artifact and was dropped. Moved here from claude-workspace PR #268. |
toc-path-remax_kb/ | 2026-09-09 | done — negative. Prepending a chunk's markdown heading path to its indexed text does not improve remax_kb: fused Recall@1 0.592 -> 0.586, a difference of −0.0063 [−0.0325, +0.0188] over 799 queries, with every fused contrast at every k straddling zero. The harness is not blind — with the heading path itself as the query the same arm gains +0.2450 [+0.1950, +0.2950] Recall@1. The two modalities move in opposite directions and cancel: dense MRR −0.0200 [−0.0378, −0.0019], BM25 Recall@10 +0.0113 [+0.0013, +0.0225]. Heading-bounded chunking (43% more chunks) is actively worse, fused Recall@3 −0.0300 [−0.0551, −0.0063] | RESULTS.md + PLAN.md + ERRORS.md + build_corpus.py + make_prompts.py + consolidate_queries.py + run_arms.py + rescore_modalities.py + analyze.py + subgroup.py + recheck.py + results.json | Muninn, from assessing arXiv:2609.03874 (STAIR/SearchTome, IBM Research India). The paper reports Recall@1 82.6 against BM25's 59.5 and reads the gap as structure helping retrieval, but every baseline indexes section content while only the proposed system is shown section titles, and no table varies that axis. This runs the cheap version — heading path spliced into the text you already index, no finetuning — on a retriever not built for it. Four arms: production default_chunker, a source-path leakage control, the heading path at byte-identical boundaries (an instrumented chunker reproduces default_chunker exactly on all 95 files, asserted at build), and heading-bounded chunks. 799 queries written by ten subagents shown only chunk bodies, never the headings and never the purpose. Pre-registered before measurement, including that a negative delta was a real expected outcome; PLAN.md called the mechanism and got the signs backwards, predicting BM25 would suffer when dense did. Also closes the title-boosting question lexical-kb/RESULTS.md deferred, and argues for BM25F field weighting over concatenation if it is revisited. Three errors logged, one caught by recheck.py overstating the result before commit. |
spd-hungarian-decoder/ | 2026-09-08 | done — on 1,785 MSLR-WEB10K slates of 50 items, sorting the score matrix's first column ranks as well as solving the assignment problem over the whole matrix (NDCG@10 0.5186 vs 0.5194, +0.0008 [−0.0028, +0.0044]), holding across 3 seeds and 4 Sinkhorn temperatures; the Hungarian solve does beat the row-argmax-plus-repair strategy the paper names as its alternative, +0.0076 [+0.0039, +0.0113]; the trained matrix carries 80.9% of its Frobenius energy in one singular value (96% for a linear-probe head), and on a rank-1 matrix the optimal assignment is provably argsort | RESULTS.md + PLAN.md + prep_data.py + train.py + decoders.py + run_eval.py + controls.py + adversarial.py + adversarial2.py + solver_bench.py + out/ | Oskar, on arXiv:2609.01807 (Laftchiev et al., Meta; v1 hLLM, v2 SPD): "This seems testable and test-worthy". Every ablation in the paper varies the head, the training signal or the backbone; none varies the decoder, so this registers and runs the missing one. Pre-registered readings for both outcomes, plus a positive control on a synthetic matrix with real two-dimensional structure (D1 ahead by 0.04–0.09 NDCG@10 there, so the harness can detect a Hungarian advantage) and a negative control confirming D1 is exactly argsort on rank-1 input, 500/500. The N×K formulation does beat a scalar score plus a sort (+0.0070 [+0.0030, +0.0110] NDCG@10 against the same encoder, same attention, ListMLE) — what it buys is not delivered by the solver. Two side findings: near-rank-1 matrices are LAPJV's bad case (3,072 ms at N=2,000 against 225 ms for random), and the empirical exponent is 2.16, not the O(N³) the paper quotes. Backbone replaced by an MLP over 136 LTR features, so nothing here reaches the paper's prefill-hidden-state or latency claims. |
neomme-remex-quant/ | 2026-09-07 | done — SciFact text + ViDoRe DocVQA and ShiftProject page images; NeoMME ships no quantized vectors; remex 1-bit token codes for its late-interaction head (5.1 KB/doc, 32x smaller) score nDCG@10 0.7070 on SciFact against 0.7198 fp32 and beat the fp32 dense head (4.1 KB/doc, 0.5527) by +0.154 [+0.116, +0.192]; on the dense head remex 4-bit on a truncated Matryoshka dim wins every byte budget from 64 B to 512 B | RESULTS.md + ERRORS.md + neomme_quant.py + encode.py + bench.py + report.py + recheck.py + results.json | Third corpus, ShiftProject (1,000 pages, 100 queries, an easy set at R@10 0.99): the 1-bit token index is +0.016 [−0.006, +0.040] over fp32 at 48 KB/page, pool2 + 1-bit +0.003 at 24 KB, and the dense SciFact ordering returns (4-bit d=256 over 1-bit d=1024 by +0.040 [+0.003, +0.077]); across all three corpora, 1-bit tokens are within noise of fp32 at 32× smaller, 4-bit dense at full width matches fp32, and 1-bit dense at full width loses everywhere. Second corpus, DocVQA (500 OCR-free scanned pages, 2,921 tokens each): the 1-bit token index is −0.004 [−0.015, +0.007] from fp32 at 46.7 KB/page against 1.5 MB, pooling is free there too (pool2 −0.004), so pool2 + 1-bit lands at 23 KB/page, 64× under fp32, for −0.004; the token head beats the dense head by +0.130 on pages. The dense 128–512 B frontier is flat within noise on DocVQA; 1-bit at full width and fp32 at 128 dims both lose about 0.025 on both corpora. Oskar, from Tom Aarsen's Bluesky thread on H Company's NeoMME release: "check if they already ship a quantized vector version, and if not implement a remex/remax style vector quantization for it". Prior-art check first: ten Hcompany/neomme repos, safetensors only, no ONNX/int8/binary siblings, no derivative repos; the native size levers are Matryoshka 128/256/512/1024 on the dense head and HierarchicalTokenPooling on the token head, and both are arms in the bench. neomme_quant.py gives both heads a remex (multi-bit Lloyd-Max, RHT) and remax (1-bit, asymmetric float query) index and scores MeanMaxSim in rotated space, self-tested against sentence_transformers.util.mean_maxsim. Quantizing tokens beats pooling them per byte (remex 2-bit at 16x vs pool2 at 2x: +0.006 [−0.005, +0.016]). The dense→late top-100 pipeline the thread suggests caps at 0.6854 because dense R@100 is 0.864 vs late's 0.943; a full scan over 1-bit tokens beats the all-fp32 pipeline at 3% of its bytes. remex 1-bit and remax k=1 asym are the same code up to the rotation draw, and their ±0.01 gap is the bench's seed floor. Encode 39 min on 4 vCPU (transformers main needed; 5.16.1 has no NeoMME). Text only, one corpus, one seed. |
latent-calculator/ | 2026-09-06 | done — a frozen SmolLM2-135M answers arithmetic at 0.90 exact match with a calculator wired in between layers 16 and 17 and no tokens in either direction (add 0.99, sub 0.98, cmp 1.00, 12-digit mul 0.66); frozen alone 0.001, the result pasted as text 0.000 exact; asking needed a head that attends over the prompt tokens (0.993 query recovery vs 0.10 from one vector) and reading needed the result re-injected at every answer step with digits most-significant first (0.60 → 0.75 → 0.91) | RESULTS.md + PREDICTIONS.md + demo.html + demo.py + make_demo_page.py + data.py + model_utils.py + probe.py + query_head.py + train_port.py + eval.py + run_all.sh + run_phase2.sh + test_latent_calculator.py + results/ | Oskar, the depth-3 port from the in-model-tools design space (memory 88477132): can a tool go between layers, latent in and latent out, for an exact result rather than a retrieval. Frozen Pleias Monad 56.7M and SmolLM2-135M; trained parts 1.3M params (attention query head, step-conditioned result encoder). Three designs: one-vector query + single injection (phase 1, D1/D3 confirmed, D2/D5 refuted, D4 refuted on the held-out length because consumption failed first); attention query head (asking solved in distribution, fails on unseen operand lengths by position-counting); streamed injection, then most-significant-first digit slots (reading 0.91, generalizes to the held-out length at 0.85 with true operands). Residual add beats a KV slot on both models; delayed-by-one injection is free behind a space token and blind behind a word. Monad follows the same curve at 0.50. 29 tests, resumable drivers, ~9 h CPU. |
harness-bench/ | 2026-09-06 | done — on 12 Aider Polyglot exercises at a fixed model (Haiku 4.5), a single-shot edit harness scores 3/12, one retry carrying the test output 8/12, and an agent that runs the hidden suite itself 11/12; 8 recovered, 0 regressed, exact McNemar p=0.0078 | RESULTS.md + PLAN.md + ERRORS.md + recheck.py + harness/ + results/ | Oskar, from dollspace.gay's post that a self-built harness beat the leader on a harness benchmark: "Can you test our own harness à la...". Pre-registered the null and a ceiling stop, because orchestrated-coding-pareto and the PR #76 pilot both went vacuous at 14/14 — polyglot cleared it at 3/12, the first arm comparison here with room to measure. Every task certified both ways (reference PASS, stub FAIL) before any agent ran, which caught a shared CARGO_TARGET_DIR scoring a todo!() stub as 23 passed. Four of the five retry recoveries were a non-compiling import, capitalisation, operand sign order and a reset() that did not reseed. go/zebra-puzzle survives all three arms. Not comparable to the leaderboard's aider+gpt-5.2 0.880: different model, 12 tasks against 225. |
harness-bench/ | 2026-09-06 | done — Opus 5 single-shot scores 182/203 = 0.897 (Wilson 95% CI [0.847, 0.931]) on Aider Polyglot, statistically indistinguishable from the leaderboard's aider+gpt-5.2 0.880 over 225; separately, at a fixed Haiku 4.5 the harness loop moves 3/12 → 8/12 → 11/12, 8 recovered 0 regressed, exact McNemar p=0.0078 | RESULTS.md + PLAN.md + ERRORS.md + recheck.py + harness/ + results/ | Oskar, from dollspace.gay's post that a self-built harness beat the leader on a harness benchmark: "Can you test our own harness à la...", then "the whole point was testing your harness to see how you compare". Pre-registered the null and a ceiling stop, because orchestrated-coding-pareto and the PR #76 pilot both went vacuous at 14/14 — polyglot cleared it at 3/12. Every task certified both ways (reference PASS, stub FAIL) before any agent ran: 203 of 225 admitted, 6 rejected as refactoring exercises whose stub already passes and 16 whose reference will not build here, so the denominator is 203 and the exclusions are not random. Three harness defects found and fixed mid-run, each of which would have measured the prompt rather than the model: a shared CARGO_TARGET_DIR scoring a todo!() stub as 23 passed; prompts rendering only the first of a cpp exercise's two solution files, so the header the test includes was invisible; and regenerating prompts before resetting the work tree, which fed a run its own prior solutions. Per-language: javascript 0.958, python 0.941, cpp 0.909, java 0.897, rust 0.833, go 0.806. |
brief-compression/ | 2026-09-06 | done — negative; telegraphic briefs to Sonnet/Haiku subagents save ~110 brief tokens and cost 1.5k–6.3k receiver tokens, 2–3x tool calls, and 4 of 10 Sonnet runs misread lc "svc_" as case-insensitive; plain prose was the only style with zero failures | RESULTS.md + ERRORS.md + PREDICTIONS.md + recheck.py + gen_b.py / score_b.py + data/ | Oskar, from two screenshots of GPT Astra briefing its Luna sub-agents in telegraphese: would Muninn gain by briefing Sonnet and Haiku the same way? Two file tasks, three brief styles carrying identical information (prose 287 tok, structured 218, telegraphic 178 on Task B; o200k proxy), Haiku 4.5 and Sonnet 5 as Agent-tool subagents, 96 runs. Stripping inter-word spaces, which the screenshots appear to do, saves zero tokens on o200k (53 vs 53). Task A (9 checks) hit ceiling at 33/36. Task B (13 checks, n=10 per cell): prose 10/10 on both receivers; telegraphic 8/10 Haiku, 6/10 Sonnet (pooled 14/20 vs 20/20, Fisher p=0.02); structured 9/10 Haiku, and 2/10 Sonnet on a wording defect in one field of the structured brief (10/10 with it excluded, ERRORS.md #1). Every Sonnet-telegraphic miss is the same miss: !startswith lc "svc_" read as case-insensitive, with rows_kept matching the case-insensitive count to the row. Telegraphic receivers made 4.1 vs 2.0 (Haiku) and 2.9 vs 1.0 (Sonnet) tool calls per run and took 2–2.6x the wall time, re-reading data to resolve abbreviations. The brief's token count is the wrong quantity to optimise; the receiver's total is, and it moved the other way. |
alta-superposition/ | 2026-09-06 | done — on ALTA, every compiled program (looped SUBLEQ, looped parity, feed-forward parity) compresses below its live count with shared directions under a trajectory-SVD continuation, and cliffs at its live count with orthogonal survivors under the code-SVD continuation LAC used; the iteration-vs-feed-forward split does not appear | RESULTS.md + PREDICTIONS.md + alta_common.py + train_code.py + run_all.sh + plot_results.py + test_alta_superposition.py + curves.png | Oskar, testing whether LAC's no-sharing result (llm-as-computer experiments/superposition/RESULTS-A.md) is about iteration, on a second compiler: ALTA (Shaw et al. 2024) with its weights frozen and only a residual code trained by the LAC hinge objective and continuation. Trajectory-SVD arm: subleq 95 of 121 live (interference 0.47), parity_seq 17 of 31 (0.46), parity_ff 5 of 7 (0.38). Code-SVD arm: 121, 31, 7 with zero off-diagonal, the LAC pattern on all three. B1 refuted, B2 confirmed, B3 refuted under trajectory SVD; the LAC learned-code run should be repeated with the trajectory rule. jax + tensorflow-cpu, 18 tests, ruff clean; per-width checkpoints, resumable driver. |
rasp-numeric-select/ | 2026-09-05 | done — a numeric select_at for RASP: one attention head of head_dim 2 attends to a computed integer address through the parabolic key (2j, -j^2), winner/runner-up gap exactly 1, and the compiled weights carry no compile-time length bound | RESULTS.md + rasp_ns.py + compile_ns.py + programs.py + margin.py + test_rasp_ns.py + gap_vs_beta.png | Oskar, extending the LAC line (oaustegard/llm-as-computer) after torchlean-lac/: RASP/Tracr, ALTA and B-RASP[pos] all reach a position through a categorical predicate over one-hot keys, so a computed address costs one residual dimension per reachable position. select_at(addr) reaches it by key geometry instead. Three programs a categorical compiler cannot express without a bounded position table — y[i] = x[2i+1], y[i] = x[x[i]], y[i] = x[i - x[0]] — compile to 13, 11 and 19 residual dimensions and match the interpreter exactly at n = 3, 8, 32, 128, 257 and 1000, having never been shown a length; the same gather through a categorical position table costs 55 dimensions at n_max = 8 and 391 at n_max = 64 and is wrong above its n_max. Softmax attention matches average-hard attention within 0.5 from beta 3.5 (gather), 3.0–6.25 (chase, rising with n because the read values do) and ~9 (shift, whose range gate costs about 6 extra units of beta). numpy only, 34 tests, ruff clean. |
torchlean-lac/ | 2026-09-05 | done — LAC's parabolic addressing proved in TorchLean's spec layer, and its float32 capacity ceiling (exact through address 4096, fails from 4097) proved over TorchLean's FP32 arithmetic; the hard-argmax read cannot reach the graph IR | RESULTS.md + LAC/Core.lean + LAC/Check.lean + LAC/Capacity.lean + capacity_numerics.py + recheck.py + ERRORS.md | Oskar, after the TorchLean survey: "Can we describe/craft LAC in TorchLean?" Five lemmas about score j i = 2*j*i - j*j elaborate against Mathlib, including score_gap (adjacent integer addresses differ by at least 1 in score); paraKey/paraQuery/Mem C/rowScores elaborate against a fully built NN (4352 jobs, clean). #print axioms gives the three standard Lean axioms. Blocked: NN/IR/Graph.lean's OpKind has no argmax, gather, or one-hot, so no autograd and no IBP/CROWN on hard-argmax LAC; and Spec.Tensor.argmax has no characterization lemma anywhere in NN/ (the three argmax theorems in the tree are all about argmaxClassifier), so read-exactness needs the loop invariant proved first. The binary32 capacity ceiling (j^2 > 2^24) is proved in LAC/Capacity.lean against FP32 = NF binaryRadix fexp32 rnd32: read_exact_below_ceiling (every intermediate of the float32 score is exact for j ≤ i ≤ 4096) and read_ties_above_ceiling (addresses i-1 and i score identically for 4097 ≤ i ≤ 5792, the first binade), plus not_both_representable_above_ceiling for every binade and four bit-level IEEE32Exec instances closed by kernel decide. numpy float32 confirms the pipeline fails on exactly 4097–5793 and then only sporadically. |
eml-prove2me/ | 2026-09-05 | done — 11 EML witness theorems verified on Prove2Me; lower bounds posted open; mission proposal drafted | README.md + Definitions/ + Theorems/ + Solutions/ + mission_description.md + scripts/ + published.json | Oskar, after the Anthropic FLT formalization post: "Try the EML table as a Prove2Me mission. Sign yourself (or me if needed) up for an account." Account muninn registered under his email; EmlComplexity definition (Tree, size, real eval, valid, Attains, Complexity) published; the eleven real-branch witnesses from eml-sr benchmarks/eml_complexity.md (e:1, e-1:2, 0:3, e-2:7, -1:8, 2:9, ln2:12, 3:14, 1/2:17, -3:20, 4:21) proved by exhibiting the enumeration's tree and rewriting with log_exp/exp_log under pre-proved positivity facts, all ACCEPTED by the server's Lean (v4.33.1, Mathlib 0df444a); complexity_two, not_attains_four_below_twenty_one, complexity_four posted as open theorems; proposal 9a4eba3f with 13 milestones awaits the human confirm-and-submit step. Lean tactic lessons in the README. |
static-code-embed/ | 2026-09-03 | done — no cheap adaptation of a static table reaches rg, let alone bekko; fusing one with rg lowers rg | RESULTS.md + scripts/ + results.json + data/vocab_*.json | Oskar, on the zvec-grep evaluation: "isn't part of the attraction ease of fine tuning?" then "Run it!". Same n=59 sklearn harness as bekko-embedding-bench; rg reproduces 0.596/0.682 exactly and bekko-a25m re-encodes to 0.651/0.706. Sibling of potion-code-quant/ (same day, separate session; that one covers quantization and index compatibility, this one adaptation) — its vanilla potion r@5 0.436 is reproduced here independently. Five adaptations of potion-code-16M-v2, all paired against vanilla: MNRL on 4,223 docstring→code pairs (val loss 1.64→1.14, r@5 0.435, Δ −0.001); +12,276 corpus identifiers as whole-word rows, sum-initialised so epoch 0 equals vanilla (0.439), then MNRL (0.418 / r@10 0.563, n.s.); the same with mean-initialised rows 0.334, p=0.021 worse — one row replaces k pieces and long identifiers lose k-fold pooling weight; a hand-rolled Model2Vec distillation of bekko-a25m (23.6k tokens, PCA 92%, SIF) 0.249, +MNRL 0.350; model2vec.distill.distill() of the same teacher (268,850 rows) 0.052, +MNRL 0.135. bekko-a25m vs potion +0.214 r@5, 26/2, p<0.001; every static arm vs rg p≤0.004; potion RRF with rg 0.537 vs rg 0.596. Static encode 6 s vs bekko 20.6 min for 11,439 chunks. Caveats: identifier-poor stratum n=1; one learning rate for all arms; ast chunking only. |
potion-code-quant/ | 2026-09-03 | done — potion trails bekko-a8m by 0.16 r@5; no static table is a drop-in query encoder for a transformer-built index | RESULTS.md + run.py + distill.py + fit_table.py + specter2_fit.py + check_centered.py + results_*.json | Oskar: test the Model2Vec static code model (minishlab/potion-code-16M-v2, a 63k x 256 fp16 token table, mean-pool) against our embedders and see if its output quantizes; then, can a static table produce embeddings compatible with an index a transformer built, including a remax/remex-quantized SPECTER2 set. On bekko-embedding-bench's n=59 sklearn file-discovery task: potion fp16 r@5 0.436 vs bekko-a8m 0.595 (Δ −0.159, p<0.001); bekko's 96 B remex 2-bit sidecar (0.590) beats potion's uncompressed 512 B vector. potion encodes 80x faster (1.3 vs 104 ms/query) and its own output quantizes about like bekko's: remex 2-bit −0.015 (n.s.), remax 1-bit −0.075 (p=0.10); Model2Vec's native int8 table is free and its dimensionality= truncation equals a post-hoc slice. Token-level Model2Vec distillation of bekko (pca_dims=None) is not space-compatible: cosine 0.26 to the teacher, a 0.957 common component across all student vectors, r@5 0.017 querying bekko's index. Fitting the table by ridge regression to the teacher's sentence vectors gets cosine 0.76 on held-out code (0.60 on NL queries) and r@5 0.36 against bekko's index (teacher 0.60; remex 2-bit index 0.39, remax 1-bit 0.24), 9 MB table. SPECTER2 (remax bench 10k set, seed 99): the fitted 26 MB table reads cosine 0.96 raw but 0.71 after centering (SPECTER2's shared mean gives raw pairwise cosine 0.85), sign agreement 0.75 after the index's rotation, and r@10 0.288 against the teacher's remax 1-bit index where the teacher gets 0.645; float index 0.264, remex 2-bit 0.117. The quantizer on the index side is not the limiting factor; the student/teacher residual is. |
embedding-inversion/ | 2026-09-01 | done — a vec2text-shaped inverter on bekko-a8m behaves as the paper describes (verifier selection, then one large correction round, then convergence) and at 40k pairs / t5-small / 4 vCPU recovers the exact string 2.4% of the time from the float vector and 0.9% from the 384-bit sign code; not retrieval, not an inverter either, a paraphraser that lands on topic | RESULTS.md + ERRORS.md + recheck.py + results_float.json / results_bin1.json | Oskar: "How would you craft a reverse embedding model?", then "implement that PoC". Pre-registered four predictions before the first stage ran; three held, and the one that decides whether this is an inverter (beat nearest-training-string by 20 points of exact match) failed in both arms. Zero-step base scores cosine 0.55 on its own training set and 0.547 on dev, so the shortfall is underfitting: vec2text's base starts near 0.9 after 5M pairs, this one at 0.55 after 40k. The arm trained on the sign code loses to the float-trained arm even at matching the code's own bits (69.2% vs 71.8%). Exact match is zero past 10 words in both arms. Also recorded: a container restart kills a nohup driver about a minute after the turn ends, and a harness-tracked Monitor keeps the container up between turns; a Monitor with persistent: true still dies at 30 min. |
ms13-k4/ | 2026-09-01 | done — theorem: R_max(4) = 4/5 exactly (Conjecture 12.2 and Q7′ at k=4) | RESULTS.md + bbk.py + splits2.py | Reopens ms13-campaign's open question at the k it called a 2,070-hour no-go. The tree census is unnecessary: maximal row-set types are the split systems of binary trees on 2k leaves with leaves paired into chords (Buneman), 4 shapes × 105 pairings at k=4, 14 maximal types, reproduces the campaign's k=3 census exactly. Fail-first branching in the campaign's own exact rational B&B (next rounding = fewest live (row, side) options at the parent LP optimum) proves R ≤ 4/5 per type in 3–64 s where lexicographic order had not finished type 0 after 900k LPs. All 14 maximal types have R = 4/5 exactly, each attained at unit demands; unequal demands never beat Doerr at k=4. Lower bounds re-verified by a code-disjoint evaluator; two-sided calibration at v=3/4 finds margin-1/20 witnesses on all 14. Hand lemma: the spider type is ≤ k/(k+1) for all demands. Open: k ≥ 5, a proof (the gap is named: weighted rows lose the two-valued error structure that Hoffman–Kruskal + Carathéodory needs). |
bts-coordinates/ | 2026-09-01 | negative (mechanism), headline withdrawn on its own null | RESULTS.md + PLAN.md | Transplant of the growing-feature-set mechanism in Large Discovery Models (arXiv:2608.15669 §4.2) onto the between-the-spokes cross-field prior-art problem, against the SLT objection in memory a8b97f70 that novelty on a new representational direction is unreachable by flat embedding search. Retro-eval targets from the ms13 campaign (Doerr 2004, cross-field; MSW25, same-field) over arXiv-built pools of 1101 and 1241 titles, one encoder across all arms so only the basis changes. The growing-coordinate arm does not beat its own frozen ablation (one win each over four informative comparisons, sign test p = 1.0; both arms also beaten by a static ranking that costs zero reads). The first pass's headline — a blind-named axis ranking the target 5th of 1101 — was destroyed by the null control the pre-registered adversarial pass demanded: 12 random pool titles used as axes give a median best-of-12 rank of 10, so best-of-k is an order statistic. Surviving effect is narrow: mean-over-named-probes beats both the raw query and random probes in 5/5 configurations, i.e. multi-probe query expansion, which METHODS.md records losing three times before. Also found: paraphrasing only the target's title moves its rank by up to 383 places, larger than every measured effect; a fabricated negative control outscores the real target (0.7719 vs 0.7295), so absolute similarity says nothing about whether an answer exists; and issue #179's own P1 test case leaks the answer's vocabulary, so a stripped variant was written. Upstream of all of it, a subagent with zero tool calls, given the problem with all cross-field vocabulary removed, returned the target's verbatim title as its top query in under two minutes. Infra: arXiv keyword search works again (it killed PR #180 in July) and HF weights are reachable from CCotw. |
hypothetical-classification/ | 2026-08-31 | done — the pattern beats every model-free baseline (0.564 vs 0.417 acc@1 on WANDS) and loses to shipping the vocabulary by 14 points (0.701); the prompt must anchor on the vocabulary's REGISTER, never on novelty, or a Haiku subagent that obeys scores a fifth of the no-model control. Shipped: muninn-utilities#127, claude-skills#782 | RESULTS.md + ERRORS.md + recheck.py | Oskar: "Don't classify. Hallucinate!" (softwaredoug.com, 2026-08-10) — "it just takes a class corpus, a cheap model and a cheap embedder". Measured on WANDS (860 labels, 468 queries, one gold label) and on Muninn's own tag vocabulary (1,273 tags, 250 memories). The source post's novelty-anchored prompt is the single largest variable and is wrong: gemini-3.5-flash-lite half-ignores "novel, never-seen-before" and writes Salon & Styling Chairs, while a Haiku 4.5 subagent obeys it, writes Hydraulic Styling Thrones, and scores 0.100 acc@1 against a 0.500 no-model control. Register-anchored, the same subagent takes the best acc@3 of any arm (0.750). A boundary claim — that the pattern halves against direct embedding on long documents — was published into both downstream PRs from the novelty prompt and then withdrawn: under the register prompt that arm goes 0.200 → 0.500, past the control, and the union of both rankings reaches 0.676/0.848/0.876. Also measured: batching 40 items per call is free at 1/17 the input tokens, char-ngram TF-IDF snaps nearly as well as MiniLM (0.528 vs 0.564) and beats it outright where documents contain their own label words, and a general-purpose Haiku subagent spends 32,539 tokens to output the word ok. On the in-browser question: Pleias Monad (57M) and Baguettotron (321M) package fine (35 MB and 236 MB at q4f16, onnx-community builds) and earn nothing — 0.425/0.400 acc@1 as label writers and 0.325/0.350 as likelihood rerankers, against a 0.500 no-model control, with the gold label in the encoder's top-10 for 82.5% of queries. What the cheap model contributes is a prior over how taxonomies name things, which is the first thing a shrinking model loses. A fully client-side classifier is real and it is the encoder alone: gte-small int8 is 33 MB and scores 0.455 acc@1 / 0.594 acc@3 with no API call. |
hyde-recall/ | 2026-08-31 | negative — HyDE query expansion over Muninn's FTS5 corpus is a wash at matched depth (0.263 vs 0.250 R@10, n=80); plain recall(n=40) scores 0.525 for free and dominates every arm | RESULTS.md + ERRORS.md | Built on a misreading of the request above (HyDE, arXiv 2212.10496, rather than hypothetical classification) and kept because it closes a question METHODS.md records as open. HyDE's filter is a dense encoder's lossy bottleneck; BM25 has none, so a hallucinated term either matches nothing or drags in an off-topic document. Generating without corpus exemplars scores below the baseline (0.175 vs 0.250) — the same register failure the sibling experiment measures at scale. The confidence gate nl2sh-dense flags as unmeasured is now measured: term-coverage gating reaches 0.300 against a 0.338 oracle ceiling and still loses 5 queries. Third consecutive loss for query expansion on this account's corpora, after nl2sh-dense and muninn-rm3. The real finding is that Muninn's default recall depth of 10 leaves half the reachable targets unretrieved at zero cost. |
halo-ccotw/ | 2026-08-24 | done — HALO's engine, WASM sandbox and trace tooling all run in a CCotw container; the RLM needs an OPENAI_API_KEY the environment does not carry, and the multi-session corpus it wants does not exist because release writes are 403 for this session type | RESULTS.md + cc_to_halo.py + validate_dataset.py | Oskar: "Assess the viability of a solution like this in the Claude Code on the web environment" — context-labs/halo. Measured rather than reasoned about: pip install halo-engine is clean, the bundled deno 2.7.14 and ripgrep 15.1.0 wheels run, and the Deno+Pyodide sandbox boots numpy/pandas/pydantic in 7.5s cold (24 MB of deno cache, ~7s per invocation since each run is a fresh subprocess). POST to OpenAI and OpenRouter /v1/chat/completions returns the provider's own 401, not a proxy 403, so a key supplied through the environment is all the engine needs; there is no way to route its calls through the harness, since the Agent tool is a model tool call rather than an HTTP endpoint a subprocess can reach. cc_to_halo.py converts Claude Code transcripts to OpenInference spans — one 11-minute session became 161 spans with all three of the index's health counters at zero, and view_trace correctly reports it oversized at 391 KB against a 150 KB budget. Two conversion traps: durations are derived (each record has one timestamp, so LLM-span duration is an upper bound while tool spans are true wall-clock), and prompt tokens are three Anthropic fields summed — counting only input_tokens reports 106 for a session that processed 13.7M. The corpus is the real blocker: persist-transcript.sh has archived nothing since 2026-04-05, because /tmp/.workstation-booted is gone and, underneath that, the proxy returns 403 on every release write and Contents-API PUT for this session type. claude-workspace#246 fixes the discovery and makes the 403 audible. |
coherence-remex/ | 2026-08-24 | done — one mechanism in daniloc/coherence is worth having (the meta-oracle, which reads a test's own AST and refuses a via test claim whose test loops a hand-written list); the surrounding apparatus is not. Adopted into remex as a live trial: remex#80 | RESULTS.md + ERRORS.md + recheck.py + artifacts/ | Oskar: "Read this full thread" (daniloc.xyz, 2026-08-22 — models write fine code at file level; the failure is complexity across files, swamped by volume no human reads; cites Conant & Ashby on regulators), then "attempt implementation against one of our repos to assess it with a live use case. Maybe remex?". Tool ground-truthed rather than read: builds clean on Node 22, 936/936 tests pass, verify on its own repo is 83 claims all green over a 643-decision dogfood ledger — but 66 of 66 of its own boundary claims use via guard, the escape hatch that skips the meta-oracle, which its own Known Limits calls "a laundering channel for hand-lists dressed as guards". Zero via test, zero parity, zero conforms to in its own specs, so the flagship check is unit-tested and never exercised through its own enforcement path. That made a trial on foreign code the only assessment available. On remex it ran on Python out of the box (shipped tree-sitter grammar, no adapter) and immediately flagged test_pq_and_npz_round_trip_rotation — @pytest.mark.parametrize("rotation", ["haar", "rht"]) against a three-member ROTATION_CODES — as "iterates a LITERAL domain… a sampling oracle, not totality". Four perturbations, each reverted: "hadamard2": 3 into ROTATION_CODES with no construction, and "hadamard2" into Quantizer.ROTATIONS alone, both leave the entire 267-test suite green while only the live-domain oracle goes red by name; the other two are co-detected and labelled as the weaker refutations they are. Adoption cost is a refactor, not a spec edit: the second literal domain had no registry to loop — the packable widths were spelled five times as bits in (5, 6, 7) across three modules plus twice as [1, 2, 3, 4, 8] in tests — so closing the claim meant extracting SUPPORTED_BITS in remex itself. Also found wrong: the parity arm false-fails a correct oracle that aliases its domain (for name in sorted(persistable) refused; for name in ROTATION_CODES accepted), against a README that claims the analyzer never false-fails; and redundancy found two disagreeing README benchmark tables while missing the five-site source duplication the claim machinery had just caught. Ends at 7 claims/7 green/3 anchored invariants each with a recorded refutation, 288 Python tests passing. Verdict: transplant the LIVE/LITERAL/NO-ITERATION classification into verifying-claims; leave the 40+ subcommands, the 217 KB README (~54k tokens it tells agents to read) and the doctrine/gyroscope/premise-lease vocabulary. |
hyparam-survey/ | 2026-08-24 | done — hypvector is remax_kb's architecture in a Parquet container, and the one thing it has that we do not is a build-time IVF whose cells are physically contiguous on disk, so a reader range-fetches 28 of 112 clusters instead of downloading the index | README.md + NOTES.md + ERRORS.md + results.json + evidence/ (nine cited source passages, re-derivable from the sha256-pinned npm tarball) + recheck.py | Oskar: "This org's work is seriously impressive! Spelunk and pick up tooling/references that may aid us in our work. The vector search on parquet part is maybe something to consider for remax-kb?". Surveyed 27 public repos; the vector library is npm-only (hypvector@0.2.2 ships unminified src/, the GitHub repo is private). Ran the sweep independently on 50k synthetic 384-dim vectors: their constants.js claim that residual misses are a rerankFactor limit and not a probe limit reproduces — scanning all 112 clusters instead of 28 moved recall@10 from 50% to 49% — and their default rerankFactor: 10 reaches only 50% on this corpus, 93% at 100. remex.IVFCoarseIndex already partitions a corpus but is deliberately data-oblivious and keeps an 8-byte-per-vector permutation; hypvector reorders rows physically and stores per-cluster counts instead. |
model-register-drift/ | 2026-08-23 | done — Opus 5 is third-cleanest of six models on declaude_lint.py and roughly twice the next model's rate once the register entries regex cannot reach are counted by hand | RESULTS.md | Oskar: "people are not all that happy with your Opus 5 substrate's attitude and personality... I find its idiolect insufferable". Six bare CCR sessions (create_session, explicit model id, no source_url, so no repo, no boot, no identity) wrote the same 700–900 word post with no voice instruction from anyone. The Agent tool could not run this — its model parameter takes four aliases and cannot address 4.8 or 4.6. Scored twice: declaude_lint.py normalised per 1000 words, then every one of the 42 entries by hand. The two rankings invert. Opus 5 sits at 5.04 tics/1k mechanically (3rd of 6) and 25.1 violations/1k adjudicated (1st, next is 16.4), with 85% of them in the aphorism/verdict family — entries 3, 7, 12, 13, 37, 38, 39, which SKILL.md already names as the third its regexes miss. Four of six headers are verdicts; six paragraphs end on a quotable line. Within the Opus line the adjudicated rate rises 11.4 → 15.3 → 25.1; the Sonnet line does not (16.4 → 13.4). n=1 per model, one prompt, one judge — only the Opus 5 gap is wider than adjudicator noise. |
nl2sh-cli/ | 2026-08-23 | done — the shippable half of the nl2sh line as a CLI: hybrid search over shell documentation with no model at all, plus an optional generator through ten backends | README.md + nl2sh/ (search.py, backends.py, cli.py, config.py, vendor/) | Oskar: "let's create a product that optionally invokes a model, either running locally via ollama/llm.cpp/mlx etc, or remotely. If no model is chosen the user just gets the hybrid search". backend='none' is the default and the product — ranked documentation, no weights, no network, no key — rather than a degraded mode. Local: ollama, llamacpp, llama-server, LM Studio, mlx, transformers. Remote: any OpenAI-compatible endpoint, Anthropic, Gemini. Remote costs no dependency because all of it is HTTP and urllib speaks it, so the extras cover only in-process runtimes. Every probe returns Availability(ok, detail) with an actionable reason instead of a bare False, the same rule compose_layers._tag_exists was fixed for. Defaults come from the measurements: instantiate_anchored because it wins every column at 1B, with a warning rather than a silent switch when a model's name looks small, since that prompt collapses to 0.146 at 270M. prompts.py and extract_params.py are vendored with pinned hashes because every quoted number came from those exact bytes. Search reaches 0.506 and not the research's 0.555: the query adapter that separates them gained +0.184 on the 207 utilities it trained on and lost 0.039 on unseen ones, so it stays out. Nothing here executes a command; it prints one. |
caps-emphasis/ | 2026-08-23 | done — capitalising a directive does nothing on Baguettotron: mid-sentence CAPS is +0.003 log-odds [-0.147,+0.168], and violation rate is 42/43 in every surface form including no directive at all; where CAPS does move the number it tracks the extra tokens capitalising costs, not the case (+0 token bin CI spans zero). Markdown bold looked 20x stronger, but most of that was entropy and the rest reverses sign when the bold is moved into the reasoning register, where SYNTH actually puts it. Corpus reason: SYNTH holds one genuine capitalised directive in 22,100 documents; bold outnumbers emphatic caps 23:1. Largest effect measured was not typographic — restating the constraint in the reasoning register cuts the ironic rebound by ~0.8 log-odds | RESULTS.md + Q5_CORPUS.md + METHOD.md + PRIOR_ART.md + ERRORS.md | #45 — does writing part of a prompt in capitals change what a model does, and by what mechanism |
tc-interference-weights/ | 2026-08-22 | done — review: three checkable problems in Anthropic's interference-weights note, all re-derived from its own figure data | RESULTS.md + check_claims.py + mirror.sh | Oskar: "get this in its entirety, including the charts, then review" — Turner, Wu & Batson, Characterizing interference weights in a tiny language model, Transformer Circuits Thread, 21 Aug 2026. A 1L transformer plus a transcoder, expanded into 331M virtual weights, scored by Fisher effectiveness (2nd-order KL, all 331M weights over 537M tokens) and helpfulness (mean loss change under ablation, 7,765 sampled weights over 1B tokens). Its genuinely new contribution stands: " IN "→" utions " is the first interference weight demonstrated inside a trained transformer with a loss measurement attached — largest raw virtual weight from that token, target never once follows source in training, ~3 OOM below top effectiveness, negative helpfulness. Three problems, each reproducible from shared/data/figures/*.js via check_claims.py. (1) The stated pruning exception names the wrong family. Text: Fisher beats raw |w| "for every density and individual weight family (except for negative Features→Logits weights)". threshold_data.js says Fisher wins that family at all 19 densities by 3–10×; the family where it actually loses is Tokens→OV→Logits negative-only, worse at 8/19 densities and up to 2.22× across density 0.05–0.40. (2) The main conclusion does not survive the paper's own effect-size appendix. The Discussion rests "the model is still dense in this basis" and "no saliency scheme will perform much better" on 47.6% of weights having positive mean helpfulness. Its ROPE appendix, population-weighted over all 331M: at ε→0, 15.1% positive / 64.3% uncertain; at the paper's own yardstick ε = budget/N_total = 1.5e-8 nats/token, 90.1% practically zero and 2.9% positive — ~9.6M weights, roughly 3× the transformer's parameter count, landing next to its own helpfulness-mass estimate (2.43% density holds 90% of positive mass). Three routes converge on 2–3%; only the sign-of-the-mean route reaches the Discussion, which never cites ROPE. (3) Helpfulness and Fisher effectiveness are the same quantity on the tail. Expanding the paper's own formula to second order gives helpfulness(w) = −w·∂L/∂w + fisher(w) + O((sw)³) — the Optimal Brain Damage saliency it cites in related work but never connects to its headline. fisher ≥ 0 always and grows as w² while the gradient term grows as w, so weights above a crossover in |sw| are forced to measure as helpful. Measured on the helpful branch: log-log slope 0.89–1.06, r 0.956–0.990, median h/fisher 0.58–1.22 in all six families over 6+ OOM. So "the model puts its most effective weights in helpful directions" is partly an identity between the two metrics, not solely evidence about training pressure — and it inverts the claim that "a sharper metric has little room to improve", since the first-order term the two metrics don't share is exactly what separates helpful from harmful. Checked and clean: the 7,765 sample is family-stratified (~1,210–1,379 each) while families range 16.8M–104.9M weights; population-weighting moves h>0 only 47.6% → 48.4%, so the headline is not biased by that. Pruning numbers verified (70% pruned → +0.0107 nats, 85% → +0.0702). Minor: the dead fraction of one sample is reported three ways — 12.7% (table), 13.1% (histogram), 13.07% (implied by the mass figure's counts). Artifact: mirror.sh reproduces a 107 MB offline mirror — main page, 21 interactive figures with data and vendor bundles, the 34-page figure gallery, the 6.2 MB feature_vis page. Two assets 403 on the live site (shared/data/token_splits.js, shared/data/token_vocab_split.js, Feature 157 figure), broken upstream. |
avo-supervisor-specter2/ | 2026-08-22 | done — best bits=4/haar/seed=2026/two-stage(300) R@10=0.7630 vs bits=4/rht/seed=42 baseline 0.7510 (+0.012), 30 candidates, 3 strategy classes, 2/3 plateau switches; the Stop-hook loop itself did not run — this session is a child session with no supervisor hook wired, so candidates ran via a manual driver applying the same plateau rule instead | RESULTS.md + run_candidates.py + ledger.json | claude-workspace#233 — first real search run off the supervisor infrastructure assessed there |
avo-supervisor-assessment/ | 2026-08-22 | done — go: exit-2 Stop hook blocks and injects verbatim (tested live), hook wall clock 38 ms, single-candidate remex fitness 1.4–1.7 s, in-repo JSON ledger beats Turso 0.2 ms vs 0.5–0.7 s; Fable 5 session with create_session(model=…) makes the three-arm ablation spawnable | RESULTS.md + artifacts/ | claude-workspace#233 — assess AVO-style supervisor loop before building |
nl2sh-instantiate/ | 2026-08-21 | done — the bake-off in MODELS.md revises this headline: a zero-shot Gemma 3 1B beats the fine-tuned 270M on every column (routing 0.799 vs 0.610, usable 0.793 vs 0.470), and the instantiation framing WINS at 1B (0.848 vs 0.799) once the model stops imitating the source-line format. The 270M finding below stands, scoped to 270M: framing the task as substitution does not move routing on a 270M model (p=0.76) but collapses token-repeat loops 0.183 → 0.049, and execution scoring puts the real functional accuracy at 0.055 against a 0.427 routing headline | RESULTS.md + prompts.py + run_gen.py + train.py + score.py + funceq_ext.py + funceq_alfa.py + alfa_prep.py + bench.py | Issue #52, stage 2: retrieval finished at 0.555 gold-in-sources against an 0.640 oracle ceiling, so the loss is the generator's. The issue argues the job is instantiation — the user's literals substituted into a documented example — because §6 of nl2sh-dense measured an exemplar worth +0.189 routing while the choice of exemplar was worth zero. Oskar: "let's start with the 270 million Gemma three". Model, sources, distractors, decode and seed held fixed; only the instruction varies. Zero-shot the substitution prompt loses 0.146 to 0.500, and the loss is a format artifact: on 0.774 of rows the model answers in the shape of the source lines it was shown, bullet included. One epoch erases that entirely (0.774 → 0.000), the same shape as stage 1's 0.026 → 0.706. Past training the prompts route the same — 23 wins to 20, p = 0.76 — and separate on garbage: loops fall 0.183 → 0.049 and usable gains +0.092 (31–16, p = 0.040). nl2sh-selfhist/MODELS.md had named degeneracy the real ceiling after repetition_penalty=1.3 bought a similar reduction at a cost of 0.118 routing; this buys it free. The published benchmark reproduces the direction on rows built to execute (westenfelder/NL2SH-ALFA, MIT, arXiv:2502.06858): 0.854 → 0.866 routing, 0.144 → 0.085 degeneracy on the non-find slice — after subtracting its 0.393 always-find prior, which eats most of the 0.911 headline. And the first functional number for this line of work is 0.055: with the funceq fixture built from every path the gold commands name, 36 of 164 cyber rows are decidable and 0.250 of those are equivalent — utility_ok overstates by ~8x, corroborating the issue's hand-read estimate of "nearer 0.05". The cyber corpus caps at 0.22 coverage no matter how wide the fixture; ALFA is where execution decides. Recovered from a lost container: the session that ran the grid wedged on a run_in_background poll loop and was reclaimed with everything uncommitted — scripts and tables came back from its transcript and re-ran to identical numbers. |
gemma-proxy-tuning/ | 2026-08-20 | open — vocabulary precondition holds, logit scale does not; experiment not run | RESULTS.md + tokenizer_diff.py + tokenizer_diff.json | Oskar, after establishing that speculative decoding cannot merge two models: "how similar are Gemma 3 and Gemma 4? Could we fine-tune the smallest Gemma 3 and apply it to the Gemma 4 ~30B?" Proxy-tuning (Liu et al., COLM 2024) and emulated fine-tuning (Mitchell et al.) are the same equation, softmax[s_base + alpha*(s_expert - s_antiexpert)], and both need the three logit vectors to share an index and a scale. Index: yes. Both tokenizers are 262,144-piece BPE with a byte-identical 514,906-rule merge list; 255,938 ids (97.63%) carry the same token and zero ordinary text tokens moved — all 6,187 that did are special or reserved. The disagreeing ids form exactly two contiguous ranges, [46,106] (61 <unusedNN> shifted by 7) and [255999,262143] (6,145 Gemma 3 image/turn tokens replaced by Gemma 4 `< |
needle-depth-growth/ | 2026-08-20 | done — the surgery is 30 lines and lossless; the training is pretraining, not a fine-tune | README.md + grow.py | Oskar, reading the Cactus paper next to needle-bsky: "can a model like needle be retro trained to add MORE layers, à la Monad?". Mechanically yes, and verified. Needle 2's stack is an nn.scan, so every per-layer tensor carries a leading axis of num_layers — block weights inside the scanned collection, MHC lane parameters as explicit (L, ...) arrays — and nothing downstream hardcodes 27 (decode.py sizes the KV cache from cfg.num_layers, export.py loops over it). Growing is a concat along axis 0 plus a config bump. grow.py does it with identity-initialised blocks and the grown model's logits are byte-identical (`max |
needle-tool-naming/ | 2026-08-20 | done — negative: tool names and tool descriptions are two near-equal channels, and better names buy nothing | RESULTS.md + PREREG.md + ERRORS.md + recheck.py + results_*.json | Oskar, after reading Cactus's attention-only paper (which is Needle 2's architecture — the shipped class is literally SimpleAttentionNetwork): "is it worth further experimentation?". needle-bsky left profile at 0.250 and identity at 0.333 in every arm, unmoved by a schema rewrite and a fine-tune — even though the discriminating fact was already in the description verbatim (get_profile says "follower count"; the query "how many followers does pfrazee.com have" still returns get_followers at 0.80). That is the opposite of the failure the paper predicts, which localizes the SAN deficit to low-context tokens. Hypothesis: names outrank descriptions. Six variants over the same 18 tools, same 62 queries, same scoring code, predictions committed before the first run. The hypothesis failed: names-only 0.444 vs desc-only 0.407, predicted gap ≥0.15, measured 0.037 at p=0.82. What replaced it is cleaner — the two channels are near-equal and partly redundant: deleting descriptions costs 16.7pp (p=0.035), deleting names costs 20.4pp (p=0.019), deleting both leaves 0.074 against a 0.056 chance floor, and against that floor names are worth 0.370, descriptions 0.333, both together 0.537 — a quarter of each carried by the other. Rule-written names buy exactly nothing (separated = canon = 0.611 flat and 0.778 oracle, p=1.00 both) and cost the gate: confidence separation collapses 0.191 → 0.101 because mean confidence on wrong calls rises 0.392 → 0.480. Name capture is nonetheless real and visible where it acts — mechanically rotating names onto neighbours moves profile 0.250 → 0.750, with "how many followers does pfrazee.com have" → get_profile @ 0.81 and "look up the account jay.bsky.team" → get_profile @ 0.90, descriptions unchanged by one byte — it is just too small (4 queries of 54) to surface as a main effect. Mirror case in the same arm: resolve_identity wearing the name check_network_outage_status is still chosen correctly on its description alone, at confidence 0.584 → 0.167 — conflict keeps the answer and destroys the belief. The two heads read different things: retrieval cost (oracle − flat) is 0.056 with opaque names and real descriptions but 0.185 with real names and no descriptions and 0.278 with rotated names, so the contrastive head reads descriptions and is damaged more by a wrong name than by a missing one — though needle-bsky's +26pp auto→tuned rewrite survives the oracle at +20.4pp, so most of that win was decode, not retrieval. Answers the deployment question directly: no, there is no naming route to declaring all 18 — retrieval costs 16.7pp under the best naming available and separated reproduces that gap to three decimals, leaving the already-measured ladder (regex→≤5 at 0.722/316 ms, regex-only at 0.833/0.022 ms, two-model agreement at 0.880) unchallenged. Side benefit: stripping all 18 descriptions cuts the median turn 808 → 532 ms, the only lever here that trades the other way. Harness and version check: canon reproduced needle-bsky's tuned-min exactly — 0.611, identical calls and confidences — on cactus-needle 2.0.7 against that experiment's 2.0.6. Also censused the shipped checkpoint while scoping this: 45,211,383 params, of which 18.6% are Engram hash-indexed n-gram tables (a parametric store the paper never mentions, added back after the paper concluded a SAN lacks one), and LoRA reaches 28.31M of 45.21M — out_proj, the paper's write path, is adapted, so the fine-tune negative was never a frozen-write-path story. Caveats: n=54 routable (one query = 1.85pp), the profile flip is 4 queries, separated is one author's rule applied once, and descriptions were stripped rather than degraded. |
nl2sh-dense/ | 2026-08-20 | done — a 25.6 MB encoder plus a page-level index lifts gold-in-sources 0.262 → 0.390 (p=0.0003); query reformulation is a clean negative; and the old 34-row eval would have called the whole thing a regression | RESULTS.md + sample_cyber.py + cyber_nl_ext.json + results_*.json | Issue #48: the on-device shell helper is retrieval-bound — Gemma 3 270M routes 0.706 with the gold page in context and 0.206 with real BM25, which surfaces it 26% of the time. The eval was extended first. sample_cyber.py drew 149 more commands from the same Zenodo/UCI corpus under the same tiered protocol and gen_nl.py wrote their NL, taking the independent eval from 34 to 164 leak-free requests over 132 distinct gold utilities (constant prior 0.012). That turned out to be the load-bearing step: on the original 34 rows the winning retriever lowers routing 0.206 → 0.147 while raising sources 0.235 → 0.382, and on the 130 new rows both rise — the old eval's headline would have been that better retrieval makes the system worse. A 23.5 MB encoder matches a 164.5 MB one. all-MiniLM-L6 int8 scores 0.341 in sources against bekko-a8m's 0.354 (one query on 164) at a seventh the disk, so the issue's 157-MB footprint worry does not bind. Page-level indexing is the free half of the win: grouping 31,169 chunks into 6,397 pages lifts BM25 alone 0.262 → 0.323 with no encoder at all, and composed with the dense arm reaches 0.390, p = 0.0003, 27 wins to 6. But feeding whole pages to the model does nothing (0.159 vs 0.165) — the index wants pages, the prompt does not. Reformulation lost, confirming muninn-rm3's prediction: RM3 costs 0.036 and dense-PRF costs 0.030, even though dense-PRF surfaces fcrackzip on the exact query the issue quotes. The abstention gate's margin >= 5 fails because 5 is in BM25 score units, not because it is a difference — a quantile-set absolute margin transfers as well as top2/top1 does; the ratio's advantage is needing no calibration sample. RRF is the wrong substrate for the gate (margin AUC 0.47–0.53, a coin flip) because it discards score magnitudes. End to end the retrieval gain is decisive and the routing gain is not: 0.128 → 0.165/0.183, p = 0.26/0.11 at n=164. A follow-up answers 'should we fine-tune the embedder' with no: one identity-initialized d x d matrix on frozen query vectors — 4,588 NL2Bash pairs, 40 seconds on 4 CPU cores, 4.2 MB, document vectors untouched — takes gold-in-sources 0.384 → 0.463 (p=0.024) and routing 0.128 → 0.201 (p=0.058), but the whole gain is on the 207 utilities the training data covered (+0.184 seen, −0.039 unseen), and a rank-64 adapter with 16x fewer parameters reproduces the same split — so the limit is utility coverage, not capacity, and a fine-tune would hit the same wall with more room to memorize. The largest lever turned out to be the corpus, not the retriever: a flash-lite pass over all 6,397 pages adding goal-level phrasings (Pleias-Redline-style) takes BM25 alone 0.311 → 0.427 and end-to-end routing 0.128 → 0.226 (p=0.0052), composing with the adapter for 0.555 / 0.250 — validated against a human-authored control (+0.098, p=0.0001) because both the eval's NL and the corpus's came from Gemini, and localised by a fidelity split (+0.224 on pages whose generated intents added vocabulary the page lacked, +0.086 where they only echoed it). Ships as a greppable handbook with the tldr-pages CC-BY-4.0 attribution this repo had been missing. |
nl2sh-retrieval/ | 2026-08-19 | done — the 350M model works after 25 minutes of fine-tuning (0.923 where a constant scores 0.000), retrieval is now the bottleneck, and an adversarial pass returned OVERSTATED on the first draft of the retrieval numbers | RESULTS.md + EXTRACTION.md + results_*.json | Oskar's six-component architecture — LLM-composed regex, a small model for intent, regex parameter extraction, man pages with remax-kb-style hybrid indexing, ICL grounding, and error logging — built and gated overnight. The gate is the result. Pleias-RAG-350M is trained to quote sources literally, which is exactly the operation monad-bsky measured its 56M sibling failing (51% identifier copying), so the bet was that converting generation into extraction rescues a tiny model. Handed the gold utility's tldr example every time: 0 usable commands in 40, verbatim rate 0.000, 6.5 s on 4 CPU cores. It quotes descriptions into cited prose, never a command — monad-bsky's zero-shot result one generation later. It degrades with source count (6/8 at 3 sources, 4/8 at 5, worse at 15), so k≈3 is an architectural constraint. Two harness findings cost a false negative: a prompt assembled from the special-token list alone gives a 0.000 parse rate that mimics incapacity (it must end with `< |
nl2sh-scoping/ | 2026-08-19 | done — the terminal-helper problem is utility selection over a long tail, not flag composition; and man pages are the coverage backbone, not the fallback | README.md + results.json + doc_corpus.json | Oskar, after the gh-mcp-regex-fit cascade: "how about a model + regex combo for something much bigger: bash + zsh commands". Scoping measurement run before building, to choose between a cascade (needs a thick head) and retrieval (needs a long tail the model does not know). Shell history was unavailable — a week-old laptop — and the substitute is arguably the better corpus: a helper is asked about what you would look up, not what you type most, and NL2Bash was scraped from forums and tutorials. One caveat governs everything: 60.3% of NL2Bash leads with find, so it is a correctness corpus and not a usage distribution; quote the non-find column. The head is thin and the tail is long: top 10 utilities cover only 29.1% of non-find requests, you need ~50 for 70%, and 176 utilities appear exactly once — a weak case for rules that answer and a strong one for rules that narrow. The difficulty is utility selection, not flag composition: 72.6% of non-find commands carry at most one flag. A self-correction worth the entry: the first draft argued for tldr over man pages, conflating RAG over man pages with putting a man page in the context window. Measured properly, whole pages run to 47k tokens but .TP option entries are median 56, 93.4% under 350, and tldr covers 96% of the top-50 utilities against only 50% of the used-once tail — where the requests and the model's ignorance both are. Both corpora, tiered by whether a chunk is quotable or needs composition. Unused lead: SYNOPSIS is a grammar, the same object needle-bsky compiled constrained decoding from. |
nl2sh-selfhist/ | 2026-08-19 | done — an eval where neither side is Claude's drops the fine-tuned model from 0.92 to 0.62; a real 16k-command corpus with a 0.189 constant prior replaces NL2Bash's 0.603 find-skew | RESULTS.md + corpus_probe.py + gen_nl.py + run_independent_eval.py + results_*.json | Oskar: "you've issued hundreds of commands tonight, you could use your own history" then "SURELY there are bash history logs to be found online". This session's 289 Bash calls proved unusable as a benchmark (26 general-shell, ~15 of one 'print lines X-Y' shape, and the description field is not persisted), but the search turned up the Zenodo/UCI hands-on cybersecurity training corpus (record 8136017, CC-BY-4.0): 16,065 real bash commands, 696 utilities, constant prior 0.189 (ls) against NL2Bash's 0.603 (find). Documentation coverage on that real distribution: 87.7% of invocations, 24.4% of utilities, 9.8% of the used-once tail, with the uncovered tail mostly undocumentable (ll a shell alias at 114 uses, ./ssh2john.py a local script, mfsconsole a typo) — arguing for reading shell config and $PATH over a bigger corpus. Found a builder bug: 379 tldr alias pages dropped as stubs rather than resolved to their target (whoami->id), a nearly-free +1.2-point fix. The capstone is the eval that finally has neither side authored by the model under test: the cyber corpus supplies real commands, and gen_nl.py has Gemini write the request for each (instructed not to name the utility; 4 of 38 leaked). The fine-tuned RAG model routes 0.618 leak-free (n=34) against its 0.923 on the NL2Bash gate — a 0.30 drop that is the cost of templated phrasing, the find-heavy distribution, and NL2Bash naming the answer 34.7% of the time. Failures are distractor-utility hallucination (pgmbentley, calligrastage) and abstention. This closes the eval-authorship problem that ran through the whole thread: every prior number measured against a self-authored or utility-naming eval is an upper bound, not a capability. Still utility-routing, not functional equivalence, which would be lower again. |
gh-mcp-regex-fit/ | 2026-08-18 | done — fitted routing rules lose to hand-written ones; the answer is neither, it is a cascade: precise rules first, a scored fallback that can also abstain second (+0.136 wild accuracy at zero abstention cost) | RESULTS.md + results*.json + wild.jsonl | Oskar, after monad-bsky: "what other tool use and routing use cases ought to sit behind a trained regex heuristic like that of regex_only.py" — then "build the fitting harness against the GitHub MCP catalog". monad-bsky left its 20 regex rules hand-written after reading the eval's failures, so 0.833 was fitted to an unknown degree. This searches for the rules instead, on the real 58-tool GitHub MCP catalogue (50 with upstream schemas, 79 routing targets once the seven method-enum dispatchers are counted), built from github/github-mcp-server's own committed schema snapshots. fit.py induces an ordered decision list by greedy precision-constrained covering (CN2/RIPPER shape) over structural cues, catalogue-vocabulary tokens and IDF-weighted schema-overlap features, abstaining rather than falling back. Result is negative and clean: every fitted arm loses. Best fitted reaches 0.239 on a held-out phrasing family and 0.351 on hand-authored queries against 0.546 / 0.486 for rules written by hand (McNemar p=1.7e-44 on family B, p=0.058 on the 74-row wild set); no regularisation knob — Laplace scoring, min-coverage 8, dropping bigrams — moves it. The diagnosis is vocabulary, not entity memorisation: the learned rules are sensible (tok:diff -> get_diff) but never fire on "what code does this PR actually change", whereas a human writes `\b(diff |
monad-specdec/ | 2026-08-18 | done — negative: Monad drafting for Baguettotron runs at 0.90x baseline at best; depth sets decode latency, so a 5.7x smaller model is only 2.1x faster | RESULTS.md + specdec.py + results.json + analysis.json + depth_scaling_both.json | Oskar: "could we use monad as a speculative decoder for baguettotron?". Both PleIAs, both LlamaForCausalLM on SYNTH, but different tokenizers — 7,397 of Monad's 8,192 tokens exist as strings in Baguettotron's 65,536-piece vocabulary and only 157 share an id — so drafts are verified by string-level exact match (Timor et al. 2025) rather than against target logits. The loop is lossless: every run is token-identical to plain greedy. It is also slower at every draft length: 0.90x at γ=1, 0.42x at γ=8. Three causes compound. (1) Latency tracks depth, not parameters: Monad is 64 layers to Baguettotron's 80 and decodes at 57.0 ms vs 119.9 ms, so c = 0.476 rather than the 0.18 its parameter count suggests. Truncating each layer stack gives 0.804 ms/layer at width 256 against 1.264 ms/layer at width 576 — a ratio of 1.57 where a compute-bound decode would show 2.25² = 5.06, so fixed per-layer overhead dominates and a narrower model of equal depth is barely cheaper. (2) A smaller vocabulary needs more draft steps: 3.25 chars/token vs 4.12 means 1.27 draft steps per target token, lifting the effective c to 0.602. (3) Acceptance lands just under break-even: 0.546 measured against a 0.602 break-even at γ=1. Monad is a good predictor — Baguettotron's own first-N layers agree with its full stack on only 2.1%/7.3%/25.0% at 20/40/60 layers — it is just not a cheap one. A viable draft would be ~24 layers at width 256 sharing the target's tokenizer (c ≈ 0.21, ~1.3x at γ=2), assuming acceptance holds, which it would not fully. CPU, 4 threads, fp32, batch 1, 5 prompts. |
monad-bsky/ | 2026-08-18 | done — a fine-tuned 56M generalist reaches ~2/3 of a purpose-built 45M tool-caller; the gap is transcription, not choice | RESULTS.md + ERRORS.md + params.json + recheck.py + results_*.json | Oskar: "do the same but fine-tune Pleias' Monad on the same task". Direct continuation of needle-bsky — same 18 Bluesky tools, same 62-query eval, same scoring code (imported, not copied), same 800 training rows from the same generator and seed. Monad is 56M params, 64 layers x 256 hidden, trained on SYNTH, with no decode grammar and no confidence head. Zero-shot it routes nothing — 0/54 routable, 0/62 parseable, it analyses the instruction instead of answering it. A full fine-tune (3 epochs, 108 min on 4 CPU cores) takes it to 0.481 routable at epoch 2 against Needle's 0.611 base and 0.722 best config; paired McNemar vs Needle-LoRA p=0.043, vs Needle two-stage p=0.0037. The gap is transcription, not routing: over the 41 eval arguments that appear verbatim in the query, Monad reproduces 0.512 against Needle's 0.780 base / 0.902 LoRA — austegard.com comes back as afethew.com, jetstream as jetforek — and more training makes copying worse (0.561 at one epoch). The obvious explanation is wrong and was retracted before publishing: both models carry 8,192-piece vocabularies and segment these strings identically (111 vs 109 pieces over ten identifiers), so the cause is the training objective — Needle's base weights, never exposed to this data, already copy at 0.780. It also invents undeclared tool names on 6.5-14.5% of queries, which a decode grammar makes impossible by construction. Constructive fix measured: keeping Monad's tool choice and refilling arguments by regex lifts args 0.296 -> 0.370 against a 0.444 ceiling, and what it cannot fix is free-text search terms, which have no structure to extract. Two invented numbers caught in one draft (Needle's vocab size, its layer count) — see ERRORS.md. Synergy (synergy.py, eight combinations, pure post-processing over both experiments' committed rows): one works. Where the two models independently name the same tool, that answer is right 0.880 across 0.455 coverage, against 0.741 for Needle's own confidence head at matched coverage — and the two signals compose (0.929 at 0.255). Agreement needs no confidence head, which is exactly what fine-tuning Needle destroys; the price is running both, 11x latency. Clean negative: calibration does not transfer — Needle's confidence separates its own correctness (0.584 right / 0.392 wrong) but is flat-to-inverted for Monad (0.486 / 0.532), and worsens with threshold. The other five combinations do not pay for a second model: union ceiling 0.806 vs 0.710, name-snapping buys ~3.7pp not 14, split roles (Monad chooses / Needle transcribes) reaches 0.407 args against 0.685 for Needle doing both, fallback rescues 1-2 queries, per-category dispatch tops out at 0.758 and is fitted. Four follow-ups (cascade.py, classifier.py, regex_only.py). A retry cascade (Needle confidence, then agreement, then escalate) reaches 0.613 coverage at 0.842 precision where the confidence gate alone gives 0.323/0.800 — cascading beats any single gate. Rewriting the ask adds 1-3 of 24 escalated queries and lowers precision; the english→Monad→Needle pipeline is unavailable because Monad corrupts handles inside its own think trace. Scoring Monad as an 18-way classifier over the declared names removes hallucinations and yields a softmax confidence but drops routable to 0.241 from 0.481 (verified not a harness bug via independent forward passes). And the null model nobody had measured: 20 regex rules with no model route at 0.833 routable, 0.022 ms — beating Needle's two-stage 0.722 and its oracle ceiling 0.778, holding at 0.824 on unseen template queries. The models' remaining edge is refusal (0.625 vs 0.183) and a confidence score. |
needle-bsky/ | 2026-08-18 | done — schema wording is worth +26pp; the confidence gate only works if you declare no optional arguments | RESULTS.md + ERRORS.md + params.json + recheck.py + evalset.jsonl + results_*.json | Oskar, overnight: "implement Cactus Needle in your compute environment and have it set up as an interface in front of some tools — maybe the Bsky tools in muninn-utilities and/or the ATProtoing skill". Needle 2 is a 45M-parameter, 14 MB tool-calling model for phones and microcontrollers; this puts it in front of an 18-tool Bluesky read surface drawn from the browsing-bluesky and atprotoing skills, with a CLI (route / ask / repl) and a 62-query eval set. Base-model top-1 routing is 61–70% across a 2×2 of schema wording × argument arity; only the wording contrast is significant (auto 0.444 → tuned 0.704 on 54 routable queries, paired McNemar p=0.0072). Schema arity moves the confidence gate: declaring an optional argument the query does not license makes the model fill it anyway, and since the head scores the whole call, a correct routing decision lands at 0.0004. Drop optional arguments and the same gate becomes monotone and usable — 38% coverage at 0.762 precision, 20% at 0.909, 13% at 1.000, against tuned's 2% coverage for the same precision. Confirmed on a single-tool catalogue where misrouting is impossible: 30 queries, confidence falls 0.199 → 0.111 → 0.068 as unlicensed arguments are declared, sign test p=0.043 / 0.043 / 0.00032. Two more numbers for anyone deploying it: an oracle five-tool catalogue is worth +11 to +17pp (retrieval, not selection, is where most remaining errors are — best measured arm 0.815), and declaring a sixth tool costs 3.6× the per-turn latency (284 ms → 1034 ms) and then nothing more out to 18, because retrieval is a fixed per-turn cost above five. tool_index_path does not help — it caches tool embeddings, not the per-turn query embed. Acting on both: splitting the 18 into five groups of ≤5 and routing in two steps works, but only if stage 1 is not a model — a Needle turn over group descriptions scores 0.370 routable (24pp worse than the flat 18) while ~20 lines of regex over structural cues scores 0.722 (+11pp over flat, against a 0.778 five-tool ceiling). Needle's contrastive retrieval head is much better at picking 5-of-18 than Needle-the-model is at picking 1-of-5 categories. Also found: the engine holds one global session per process, so alternating agents re-runs needle_init every turn and a loaded .cact can never be unloaded. The LoRA arm is a negative: 800 templated rows and ~2h of CPU moved routing 0.611 → 0.667 (paired McNemar p=1.0), left profile (0.25) and identity (0.33) — the two categories the templates specifically covered — completely unmoved, regressed off-topic refusal 0.625 → 0.375, and replaced every confidence score with None, since fine-tuning does not update the confidence head. Extraction is a second negative: across 22 attempts over live and constructed posts, every single one scored below 0.05 confidence (max 0.0434) — at any threshold that makes the routing gate useful, all of it escalates. Route with this model; do not extract with it. |
orchestrated-coding-pareto/ | 2026-08-16 | done — orchestration arms never activated; token economics, not accuracy, set the frontier | RESULTS.md + ERRORS.md + params.json + recheck.py + tasks/ + data/ | Continuation of luna-onprem-tco (PR #37): is a big orchestrator driving a fleet of Luna-class workers Pareto-optimal for coding? Built a 14-task bank (precise specs, hidden pytest suites validated against references before any model saw them) in three escalating tiers, run one-shot at haiku/sonnet/opus tiers in the CCotw Workflow harness with per-arm output-token metering, plus two orchestration arms (haiku+raw-test-feedback, opus-diagnoses→haiku-fixes) seeded from haiku failures. Quality saturated everywhere: haiku 14/14 = opus 14/14, sonnet 13/14 (accepted 1.0.0-01 as semver) — the ceiling survived two difficulty escalations ending in a 20-opcode stack-VM and a character-exact table formatter, so both orchestration arms went vacuous: zero haiku failures to orchestrate over. The measured story is verbosity: haiku emitted 6.7× opus's output tokens (280,717 vs 42,019; 11× on tier-3), which more than cancels its 5× per-token discount — haiku-solo $0.101/task vs opus-solo $0.079/task at equal quality; at Anthropic prices the cheap tier is Pareto-dominated by frontier-solo. Repricing haiku's measured token profile at Luna direct ($0.20/$1.20) gives $0.024/task (4.2× cheaper than opus-solo), DeepSeek-Flash $0.0057 (13.8×) — the thesis lives or dies on the fleet tier's sticker price and verbosity discipline, not on orchestration structure. Tie-back to PR #37: an orchestrated fleet is batchable (Luna batch halves it again) but at ~20k output tokens/task, saturating the 7.64 B-token/night self-host break-even needs ~380k tasks/night — self-hosting still doesn't pencil. Caveats carried in the writeup: all tasks are single-module fully-specified stdlib work (the saturation claim does NOT extend to ambiguous/multi-file/long-context coding); the harness's effort: medium has no documented mapping on Haiku 4.5, so some verbosity may be tunable; Luna prices inherit PR #37's secondary provenance. Tier-1's difficulty was misjudged and caught mid-run by early grading after Oskar questioned it — disclosed as an adaptive extension; 3 of 14 hidden suites had authoring bugs caught by reference validation before any model ran. Follow-up (same day, Oskar's effort question): re-run at effort: low cut haiku's tokens only 26% (verbosity is intrinsic, not the knob) but broke the ceiling — 12/14 — which finally activated the orchestration arms on the 2-failure seed: raw pytest feedback and opus-diagnosis both went 2/2 in one round, so the measured orchestrator premium over mechanical test feedback is zero quality at +$0.017/task; the best cheap pipeline (haiku-low + test-retry, 14/14, $0.080/task) ties opus-solo at Anthropic prices and wins 4.1x at Luna prices. |
mdbr-leaf-mt-bench/ | 2026-08-16 | done — no swap of the remax_kb default; leaf-mt-int8 ties bekko-a8m at 5.2x smaller | RESULTS.md + ERRORS.md + recheck.py + results_*.json | "Another embedding model to evaluate": MongoDB/mdbr-leaf-mt, 23M params / 1024-d, distilled from mxbai-embed-large-v1, #1 on MTEB v2 (Eng) ≤30M — run through bekko-embedding-bench's Part B harness (179-chunk blog + 179-chunk sklearn-AST self-retrieval, incumbents re-encoded on the same splits so every verdict is paired). The billing does not transfer: jina v5 nano q4 wins both distributions, decisively on code R@1 — 0.888 vs 0.581, Δ −0.307, 2 wins/57 losses, p<1e-5 — the same cell, with nearly the same margin, that settled the bekko verdict; blog R@10 −0.067 (p=0.036), and iso-byte truncation widens the gap (leaf@64 vs jina@64 code: −0.251). Where it does land: the compute-bound rung. The int8 export (23.7 MB) is a paired statistical tie with bekko-a8m in all four cells (all p>0.07) at 5.2x smaller and 8.0 vs 10.8 ms same-session 1-vCPU query — the smallest credible remax_kb embedder measured in this family, 19.2x faster per query than jina. Export quirk worth knowing: leaf's int8 is faster than its own fp32 (7.3 vs 15.4 ms) with no significant retrieval cost — unlike bekko, whose transformer-int8 ships _not_recommended — while q4 is dominated on every axis (slower than fp32 at 1 thread, only export with a directional code dip). Cross-run anchors all reproduced exactly (jina 0.631/0.978, bekko 0.575/0.888, corpus 11,380 chunks/674 files at sklearn 7cb1868aa); 30/30 recheck. Part A deliberately not run (an encoder that loses to jina on code has no path to moving "dense ties grep"); asymmetric teacher-doc mode (mxbai docs + leaf queries, the card's strongest configuration) untested — the natural next pass for an offline-index budget. Codec follow-up (pareto.png): remax does not beat the card's own plain sign bits — at the shared 128 B, blog 0.503 vs 0.547 (n.s.), remex 1-bit level with vendor binary on both dists, so the rotation/centering machinery buys nothing on a model whose quantization robustness was distilled in; quantize-before-truncate reproduces on a second model (remex 2-bit @1024 beats the fp32 MRL floor d=64 at equal bytes, +0.073 blog p=0.015 / +0.117 code p<1e-4, and ties the uncompressed 4096 B vector); the Pareto frontier is composition — binary-asym d=512 at 64 B hits full-fp32 quality (64x compression free), MRL-fp32 is dominated everywhere; and kb-k-sweep's "dims beat stacks" inverts within remax here (d=512 k=2 > d=1024 k=1 at 128 B). 40/40 recheck. |
luna-onprem-tco/ | 2026-08-15 | done, two passes — API wins at 800 seats by 2.7x; one RTX 5090 wins above ~4 h/day of flat-out generation | RESULTS.md + ERRORS.md + model.py + hourly.py + params.json + recheck.py | Asked to price the raw electricity of running a GPT-5.6-Luna-equivalent locally in Montgomery County MD against Luna's API price. The literal question has a clean answer that decides nothing. Electricity at EIA's MD commercial rate (16.4 ¢/kWh) is $0.0035 per million input tokens on a GB200 NVL72 — Luna direct ($0.20/M) is 57× that and even Luna batch ($0.10/M) is 29×. But electricity is 3.1% of the cost of owning the hardware, so the ratio is load-bearing for nothing. The binding constraint is a memory floor. Capability parity is set by measured index, not parameter count (OpenAI publishes none, and the "27 B dense" figures in circulation are unreliable): DeepSeek V4 Pro 0813 at AA index 53 vs Luna max 52, MIT-licensed — which means 1.6 T params, ~800 GB NVFP4, and a ~$450 k 8×B200 node minimum, bought whole at 5% utilisation as at 85%. At 800 office seats (55% daily-active, staggered 07:00–18:30, 2.5 h nightly batch) three usage intensities give peak utilisation of 5% / 17% / 85% and API bills of $6.1 k / $29.3 k / $153 k against $279 k self-hosted — API wins by 3.5× / 2.7× / 1.2×. The overnight batch cannot rescue it, and not for a reason that depends on any usage estimate: fully saturated for 2.5 h the box emits 1.98 B input tokens/night where break-even needs 7.64 B — short by 3.9× at 100% saturation, across the entire 20–35% MFU sensitivity band. The window where the box could run flat out is also the window where the API is half price; the two compound. It flips on token intensity per node, never on seat count — scenario B stays API-side at 6,000 seats because past one node each added seat buys capex as fast as savings, while scenario C flips at ~2,000. Two errors in the model itself were caught and fixed structurally, both flattering self-hosting: a verdict of "self-host" returned at 845% peak utilisation (one node's cost vs a nine-node bill; now nodes = ceil(peak)), and a serving power floor applied to parked hours (+35% annual kWh). Spin-off finding: rack benchmarks do not transfer to single nodes — 8×B200 decode costs $0.083/M output against the NVL72 rack's $0.014/M on identical GPUs, 5.8×, because MoE decode scales with NVLink-domain size, while prefill is flat. Caveat that dominates all others: the session's egress proxy blocked WebFetch and curl to every primary source (openai.com, eia.gov, pepco.com, artificialanalysis.ai, inferencex.semianalysis.com), so every constant came from a search-engine summary rather than the page; params.json tags each row's confidence and recheck.py (104 checks) fails if one lacks a source. The workload scenarios are authored, not sourced — the published per-seat figures found sit ~7× above even the heavy case and were rejected as not credible, which is the single judgement call most worth challenging. Second pass (2026-08-16), single GPU: same price book, 1/1000th the scale — one RTX 5090 (600 W, $4,700 street median in an August-2026 shortage) running Qwen3.8-27B (released 14 Aug, Apache 2.0, 27.78 B dense), a far closer capability match than V4 Flash was at SWE-Bench Pro 61.7 vs Luna's 62.7. Electricity is $0.18/hr at Maryland's EIA-corrected residential 22.2 ¢/kWh against $0.90–1.49/hr of Luna for the same output — 5–11×, not 57×, because a consumer card serving one stream costs $0.19–0.26 per million output tokens against the 8×B200's $0.083 and the rack's $0.014. It answers the opposite way from the fleet model, for the same reason: break-even is 4.2–7.6 h/day of flat-out generation at street prices (2.3–4.2 at MSRP-era prices), so the shortage roughly doubles the break-even and the used-GPU market outweighs electricity, model choice and rate schedule combined. Two premise corrections generalise: a quoted 180–200 tok/s is 1.56× the hard bandwidth ÷ weight_bytes decode ceiling (121 tok/s; ~97 at 80% MBU), reachable only via this model's multi-token prediction, speculation or batching — kept as a branch rather than corrected away, with a roofline-respecting 95 tok/s branch beside it; and prefill and decode contend for one card, turning 190 tok/s into a sustained 123–167 once 1/(1/decode + fresh_ratio/prefill) is applied. Price-book finding worth carrying alone: Luna's cache writes cost 1.25× uncached input, so caching only pays above a 21.7% hit rate — below it, re-sending is cheaper. Two further errors logged, one flattering self-hosting (prefill contention omitted in the scratch pass) and one overstating a correction ("1.9× the ceiling" quoted against the 80%-MBU figure, not the ceiling). recheck.py now runs 154 checks. |
ttt-embed-quantized/ | 2026-08-14 | done — artifact committed; fp32 nDCG@10 0.7152, inside the expected band | RESULTS.md + ERRORS.md + encode.py + recheck.py + data/{Dm,Q}.npy + data/meta.json | One-time SciFact corpus encode for the TTT-Embed x remex/remax quantization experiment (#33), so claude.ai — <2 docs/s on 1 core, detached jobs reaped after ~100 s — never pays for it again. Not a hypothesis test: the deliverable is Dm.npy (5183, 256) fp32 + Q.npy (300, 256) fp32 + meta.json, encoded with jina-v5-nano model.q4.onnx @ v5-nano-8a7f00aa (SHA256-verified) at the 2026-07-08 codec eval's exact settings — dim=256, max_length=384, title + ". " + text, Document: /Query: prefixes, last-token pool, truncate-then-L2-normalize — so that eval's fidelity numbers carry over. fp32 nDCG@10 = 0.7152 (R@10 0.8346, R@100 0.9483), inside the issue's 0.60–0.72 band near the top; 14.7 min on 4 vCPU at 5.9 docs/s. Re-scored cold by recheck.py through a deliberately disjoint code path (sorted() over Python floats, explicit math.log2 DCG) reproducing 0.715232 vs 0.715232 to <1e-9, 17/17 checks, with negative controls that collapse to 0.003–0.004 — a sanity check that cannot go red is not evidence. The issue's HF-CDN warning did not reproduce: all three files landed first try on us.aws.cdn.hf.co, the host the spec calls un-allowlisted, confirming bekko-embedding-bench's per-environment reading — allowlist state is a fact about a container, not about a host, and the retry loop is kept only for the claude.ai path. The pinned encoder is knowingly superseded — the mirror's own PERFORMANCE.md and this repo's METHODS.md both say the authors' upstream q4 is smaller and more faithful — and was used anyway, because comparability with the prior eval requires identical weights, not better ones; flagged rather than silently upgraded. Prior art was found and deliberately declined: rotation-decorrelation already caches a jina_scifact_corpus.npy for this exact corpus and embedder, but it is corpus-only and its settings are unrecorded, so reuse would have risked a silently non-comparable matrix that every shape check would have passed. Caveats: 27% of docs hit the 384-token cap, so better than a quarter of the corpus is encoded from a prefix; no fp32-vs-q4 fidelity was re-measured on SciFact (the 0.975 cosine is inherited from NFCorpus/muninn); and pytrec_eval does not build here, so both scorers share an author. |
subagent-messaging/ | 2026-08-12 | done — 4 of 5 documented claims hold; the reply rule does not | RESULTS.md + ERRORS.md | Does the Claude Code SendMessage/ListAgents tool pair behave as its description says? Live test rather than reading: Opus 5 parent, Haiku 4.5 general-purpose peer instructed to report every envelope verbatim, 3 agent runs, ~121k subagent tokens. The reply rule is wrong. Both the tool doc and the harness footer appended to every delivered message say to reply by copying the incoming envelope's from into to; for subagents that value is the agent type, so the send returns No agent named 'general-purpose' is reachable. Two general-purpose peers emit two identical unusable from values — the attribute cannot distinguish senders even in principle, and the agentId from the spawn result is the only handle. Envelopes are asymmetric: <agent-message from=…> parent-side, bare <system-reminder> subagent-side, so a subagent cannot route a reply by inspecting what it received. ListAgents does not exist inside a subagent — ToolSearch("select:ListAgents") returns No matching deferred tools found, not an unloaded schema — so peers cannot be discovered from below and the topology is a star through the main conversation, not a mesh; peer-to-peer coordination requires the parent to hand out ids at spawn. Confirmed as documented: delivery enqueues to the receiver's next tool round and never interrupts (probe 2 landed after a sleep 20 finished, not during it), and resume-on-send works with context intact — but it is undetectable by the resumed agent (asked directly, it reported no gap or restart marker, "reads as one continuous conversation") and costs a full agent turn, ~40k tokens, each time. Also observed: the harness neutralizes instruction-shaped tags in agent→parent output (< → <\) and relabels them as findings rather than instructions. Prior art, two passes. Account-local found nothing (zero sendmessage hits across claude-workspace, scoped xr nothing above 0.371); account-wide xr surfaced claude-skills/orchestrating-agents at 0.539, a different mechanism (API agent pools) that carries a factual defect — v0.5.0 tells the reader the native runtime lacks inter-agent messaging and never mentions either tool. The published pass demoted two findings: cross-session messaging shipped 2026-08-07 in v2.1.224 with a thorough official page that states the queue-never-interrupt model almost verbatim, and claudefa.st already documents resume-with-intact-context and a better-measured cost chain (199k→324k over eight rounds). Still unpublished anywhere found: the from attribute, the envelope asymmetry, absent ListAgents in subagents, and resume being undetectable by the agent. And it gained a contradiction the account pass could not: claude-code#48160 (closed as duplicate) and ruflo#2028 (open) both report that subagents can receive but cannot originate SendMessage — this peer originated three sends to main successfully with no AGENT_TEAMS flag, lacking ListAgents instead, the exact inverse. Either fixed since, or environment-specific; CCotw vs local terminal is the uncontrolled confounder. Closest published match to the addressing finding is claude-code#42999 (closed as not planned), where a user-assigned name fails silently while the id works — adjacent but distinct, since from is not a name and fails loudly. Caveats: parent↔subagent only, one session; true cross-session peers (<cross-session-message>), Remote Control, named teammates, and Workflow-spawned agents all untested, and the from-as-address rule may well hold for them. |
lowbit-scan-crossover/ | 2026-08-09 | done — positive; the reported scale gate is a 4.1 ms constant, and bit planes beat the shipped kernel 2.4–5.2x | RESULTS.md + fit.py + layout.py + roofline.py + arms.py + steelman.py + xover.py + hamkern.c | Challenge the inevitability of BLAS beating low-bit storage at small corpora, per memory dab41dd6 ("compression is SCALE-GATED and below ~150k rows its win is negative"). It is not a scale gate. dab41dd6's own table fits t = 4.108 ms + 32.40 ns·n; the same numpy expression here fits −0.78 ms + 33.98 ns·n — per-row cost agrees to 5% across the two machines, and the entire crossover is the constant. n* = a/(b_f32 − b_ham) recovers the reported gate at ~68,000, derived rather than interpolated between two rows. The generalisable form: in t(n) = a + bytes·n/(BW·eff), n is a multiplier identical for every kernel, so two kernels can cross only if one has a > 0 — a reported crossover with a ≈ 0 on both sides is an artifact of the two n values bracketing it, and fit.py is the check. Nothing here crossed at any n from 100 to 1e6, cold or warm, at any ISA from SSE4.2 up, single-query or batch-1024 (4.2x at batch 1024, the narrowest point). Where the numpy time goes: np.bitwise_count is fine at 14.7 GB/s; .sum(axis=1) over a 4-wide inner axis is 1.9 GB/s and 62% of the kernel — a reduction shape, not bit-packing. Storing the words as contiguous bit planes recovers 5.2x at k=256 and 2.4x at the shipped d=512·k=4 config, pure numpy, no compiled dependency; the compiled kernel is 37x over BLAS warm, 19x cold, versus the 1.7x the shipped idiom gets. Both disconfirming arms failed: an adversarial challenging pass named AVX-512 VPOPCNTDQ as load-bearing and cold cache as untested — rebuilt at -march=x86-64-v3/v2 with zero vpopcnt in the object it costs 6%, and L3-evicted it is still 19x. So the original result is not attributable to the 1 vCPU container's core, cache, or instruction set. Also corrects the correction: dab41dd6 refuted remax-hamming-speedup's "beats BLAS at every N" using a different configuration — k=256 (4 words/row) vs the shipped d=512·k=4 (32 words/row) — and the narrow-reduction pathology is specific to the former; the shipped claim reproduces here (2.58x vs its published 2.43x at N=50k). Caveats: one machine; ARM/NEON untested; single-threaded; latency only, not latency-at-fixed-recall; the 4.1 ms constant's cause is unidentified, only its existence and size; and nothing transfers to the remex ADC path, whose 13.5x the family model predicts is gather latency — flat in n as 901e3c06 reports, but the bit-width leg is untested. Prior art — the C kernel is a rediscovery. remax/src/remax/_native.py already ships a __builtin_popcountll Hamming scan that remax.packing.hamming_distances already dispatches to, whose docstring already reports 25–35x over the NumPy path and already identifies the 100k–1M cache falloff; remax/core.py and QUERY_PATH_SPEED.md already record that a gather cannot use the popcount kernel (38–45x). Externally it is textbook — faiss's HammingComputer32 is four uint64s XORed and popcounted, and what gcc emits at v3 is Muła/Kurz/Lemire's Harley-Seal. hamkern.c is kept only as a fallback-free roofline reference. The mandated account-wide xr check was skipped — run afterwards it returns remax/packing.py at rank 1 and _native.py at rank 8, or rank 2 with -r remax, in 175 ms warm. A first attempt raised ModuleNotFoundError: remex and was written up as "xr is unavailable in this container" instead of fixed with pip install remex onnxruntime tokenizers (under a minute; remex is on PyPI from the same author). An ImportError in a mandated check is a missing dependency, not a broken check — that wrong diagnosis reached METHODS.md, RESULTS.md and a PR body before being caught. Logged in the duplication map, with the install recorded under Environment gotchas. What survives as new: the constant-term reconciliation, the .sum(axis=1) narrow-axis diagnosis (the bitwise_count path, not the LUT path _native.py analyses), the bit-plane numbers, the 6% -march measurement, and a wiring gap — remax_kb/_hamming.py already imports from remax.packing but never calls its native dispatch, so the compiled kernel is one import away from the shipped scan. Method note: a first pass labelled inversion was retrofitted onto a result already in hand and did not fire; the finding came from family traversal run properly afterward. |
account-index-corpora/ | 2026-08-09 | done — qualified; account-wide PR bodies +4.6% (a floor), tombstones +7.1% but 94% deleted data | RESULTS.md + clone_depth.py + corpora_scoped.json + results.json | Should the account-wide index carry the two corpora that measured as wins per-repo — deleted files (history-tombstone-index, 0/6 -> 6/6 on mechanism) and merged PR bodies (pr-decision-log, 6/8 -> 8/8 on rationale)? claude-workspace#197 names the risk correctly as size, not answer quality, since an off-class corpus already measured inert rather than harmful. Size is the half that costs seconds instead of a 22-minute sharded encode, so it was answered alone, via a new account.py corpora that chunks through the real build path and never loads the encoder. Re-run over all 65 repos on a runner in 2 min 18 s: PR bodies +4.6% (1,953 chunks over 1,444 merged PRs) and tombstones +7.1%, against a tree of 42,578 chunks that matches the published manifest exactly — so corpora is measuring the real index, not an approximation. The PR number is a floor: 12 of 65 repos returned HTTPError on /pulls while cloning fine, so the PAT reads contents but not pull requests on them, dropping claude-workspace's own 154 merged PRs out of the total. Both 3-repo estimates were wrong in opposite directions (tombstones 11.8% -> 7.1%, PRs 3.2% -> 4.6%) without moving the verdict. On 3 repos first: PR bodies +3.2% (419 chunks, 261 merged PRs), and the account meets the condition METHODS.md records for believing the remax result transfers — median body 1,577–3,197 chars, 11 effectively empty. Tombstones: +11.8% nominal and not worth it. The first run said +564% — 74,822 chunks against a 13,257-chunk tree — because a deleted file gets no stat() and no rglob, so every filter hcindex.discover applies to the tree (extension, skip_dirs, skip_names, exclude, the 1 MiB cap) has to be reapplied by hand; without them a 767,692-line deleted embedding dump enters a corpus the live index refuses. Filtered it is 47x smaller, but claude-workspace still contributes 1,484 tombstone chunks against a 232-chunk working tree, ~94% of it sub-1 MiB JSON data dumps and ~23 chunks of actual prose and source. Two account-scale effects the per-repo experiment could not see: the relocation guard must compare across repos (the 2026-07-28 migration deletes in one repo and lands in another — 540 files skipped only because the check was widened), and candidates are restricted by basename to keep it linear. --depth 50 resolved: it cost nothing (6.5s vs 7.3s vs 7.4s full, summed, no consistent sign) and would not have worked — git log --diff-filter=D sees only the grafted window, so depth 50 found 2 of muninn-utilities' 18 deletions; coverage is a function of commit rate, not of anything anyone chose. Now depth 1, or full history when tombstones are on. Caveats: this measures size only — the answer-quality half still needs a full encode and a benchmark that does not exist; and the PR total is understated until the PAT gains pull-request read on the 12 repos it cannot currently list. |
account-routing-tier/ | 2026-08-06 | done — qualified; 87-90% @k=3 of 9 repos, not safe as a default | RESULTS.md + run.py + results.json | Can a small always-loaded index of per-repo summary cards route a query to the right partition, so a whole-account index can live as per-repo release assets fetched on demand? The failure mode is unforgiving: a flat index that ranks badly still contains the answer further down, while a coarse tier that routes wrong makes it unreachable and returns a confident result from the wrong repo. 9 repos on disk, 25,899 fine chunks; the coarse tier is 0.1-0.35% of it, so storage was never the constraint. 30 queries about internals only (ascii_fold, the CSR builder, NVFP4 dequant, sklearn's CSR indptr) so cards must route on similarity to a summary that does not contain the answer; gold is an oracle (flat RRF over all chunks) rather than hand labels. Content cards beat front-matter cards at every k with 2.5x fewer chunks (@3 80% vs 73%, 26 vs 64 chunks); both together reach @1 53%, @3 87%, @5 97%. The diagnosis came from a failed fix: 4 of 8 initial misses wanted sklearn-bench whose card lacked a README because CARD_FILES held only README.md and scikit-learn ships README.rst — fixing that moved recall@1 47% -> 43% and the same four queries still missed. scikit-learn's README has zero occurrences of "gradient boosting", "one-hot", "cross validation", "sparse" or "estimator" against 296 and 311 files in-tree; front matter states identity, routing needs inventory. Splitting large repos into per-directory cards to fix a 350x card-capacity imbalance (sklearn 0.033 card-terms/chunk vs 11.8) backfired — @1 53% -> 47% for 3.7x the cards — because ranking a repo by its best card makes more cards more draws, inflating a split repo's maximum for reasons unrelated to relevance. Verdict: 13% of queries land in no fetched partition at k=3, silently; k=5 fetches over half the partitions and defeats the point at this scale. Usable only behind confidence-gated escalation — route to top-3, widen if the best fine score is weak — which turns a silent wrong answer into latency. What broke: a confidently wrong diagnosis (above); the harness sat inside the corpus it measures for the fourth time here, after being diagnosed in code-index-duplication and guarded in hybrid-code-index — knowing a failure by name did not prevent reproducing it twice more; the oracle is not reliable gold (one query routed correctly to claude-container-layers at rank 1 and was scored a miss because flat search disagreed), so these are agreement-with-flat numbers, not accuracy; and self-pollution is structural, since experiments is both an indexed repo and where these writeups live. |
pr-decision-log/ | 2026-08-05 | done — positive; 8/8 vs tree's 6/8, and the three corpora are orthogonal | RESULTS.md + run.py + prs.json | Are PR descriptions worth indexing as a searchable decision log? Code says what, commit messages say what changed, PR bodies say why — including what was rejected. Proposed as an alternative to hunk-level change indexing, and the volume argument alone favours it: 43 merged remax PRs is 87 chunks, +12% over the tree, where hunks would be thousands of near-identical neighbours (the pollution that cost repo-index 20% of its corpus). Eight "why" questions written from CLAUDE.md's claims rather than PR text, scored as marginal value over a tree whose CLAUDE.md already documents decisions. tree 6/8 -> tree+PRs 8/8. The two gains are cases where the tree carries the outcome but not the reason: [PR #65] Consolidation: −4,879 lines for why the bench harness left the wheel (CHANGELOG.md ranks first in both arms and records only that it happened), and [PR #61] Restore rotations_ assignment via write-through setter at rank 1. Tombstones add exactly nothing on rationale (6/8 -> 6/8) — which is the more useful finding: the three corpora are orthogonal, tree answering what, tombstones how did the deleted thing work, PRs why, and a corpus aimed at the wrong question class is inert rather than harmful, so they stack. Caveats: the generalization threat is severe and was recorded before running — every remax PR is Claude-authored, median body 2,727 chars, none empty, where most repos have one-line or blank bodies; this measures "PR bodies are worth indexing when written like this". n=8, one repo. One gold list held the bare substring "PR #" matching any PR chunk — fixed, and outcome-neutral (that query hits via CLAUDE.md at rank 1 in both arms), but it is the third answer-key defect of this shape in this line of work. The real objection is architectural: PR bodies are not in git, so indexing them makes network access and a token a hard dependency of a full rebuild — every other corpus here comes off the filesystem, which is why the indexer runs offline and in CI. A real version needs a cache with a staleness policy that degrades to tree+tombstones rather than failing the build. |
history-tombstone-index/ | 2026-08-05 | done — positive; 12/12 fused vs 5/12 working-tree-only | RESULTS.md + run.py + results.json | Does indexing deleted code add anything over a repo that already documents its rejections? A current-state index structurally cannot hold code that no longer exists — but that is only valuable if the knowledge left with the code. remax is the hard case: its CLAUDE.md mandates "a measured rejection is an asset — delete the driver, never the record", so removed apparatus leaves a prose writeup in bench/results/*.md. Tested against remax (144 commits, 10,042 deleted lines, 17 true deletions) because this repo cannot answer it — 73 commits and zero files deleted-and-never-restored. Recovered each dead file at its last living revision, headed with its removing commit's subject, and scored RRF(dense, stored-BM25) over three corpora. The convention works — for existence: 5/6 from prose records alone. It cannot work for mechanism: 0/6. A record is prose about a verdict; "the encoder, its CSR-builder and a BEIR benchmark were all built" does not tell you the signature, the batching, or what the tests asserted — that left with the file, and no writeup discipline short of pasting the code retains it. Tombstone-only scores 4/6 + 6/6; the union is 12/12, strictly better than either arm, for +19% corpus. Arms are complementary, not competing. What broke: relocations look exactly like deletions — five src/remax/bench/* files were moved to bench/*, not deleted, and indexing them inflated the tombstone corpus 27% and manufactured a false current-only mechanism hit (a live crossover.py satisfying a query whose gold was its deleted path). A current-state index scoring on a mechanism-only query was the tell that the answer key was wrong. Fixed by detecting relocation by content (>50% of non-trivial lines present in a live file), not by path or basename — basename would have wrongly dropped src/remax/bench/__init__.py, which is genuinely gone. The correction strengthened the result (1/6 -> 0/6). Caveats: n=12, one repo, queries written by someone who knew the answers, gold matched on filename substrings, and only whole-file deletions are indexed — removed hunks inside surviving files are probably the larger population and are untested. The defensible product claim is narrower than "index your git history": index what was removed and never came back; everything else in history is a near-duplicate of content already indexed. |
hybrid-code-index/ | 2026-08-05 | done — hybrid wins 24/24; two assumptions refuted | RESULTS.md + hcindex.py + bench.py + bench_incremental.py | repo-index was a markdown sidecar with a code afterthought and a single dense arm; the target is a general-purpose hybrid code indexer. Six arms over three query classes (rediscovery / keyword / duplication), each scored against an answer key that already existed for another purpose. rrf(dense, stored-BM25) scores 24/24, beating bm25 alone (23), dense alone (22) and every rg variant. Stored BM25 beats ripgrep as the lexical arm decisively (23/24 vs 17/24), and the gap is almost entirely duplication (8/9 vs 3/9): ripgrep returns a set, and 'find me a file like this one' is a ranking question no amount of term-counting recovers. Adding a third arm HURTS — rrf(all 3) drops to 22/24, because RRF is unweighted so a weak arm votes as loudly as a strong one. More retrieval arms is not monotonically better. The .json dilution was refuted: 79% of the corpus is generated results data and it is inert (24/24 with and without), so no build-time exclusion is warranted — unlike the outputs/ model-generation case, where near-duplicate prose competed directly with real answers. Volume does not predict pollution; similarity to real queries does. Storage is not free though: BM25 postings inflate to 6.36 MB / 138k terms with JSON vs ~1 MB without. Rebuild cost forced incremental: a full build is 537 s, vs 0.2 s for a one-file change (2735x), and the incremental result is verified bit-identical (max delta 0.000e+00) rather than an approximation — safe because the encoder is per-chunk independent and remex is data-oblivious. BM25 cannot be incrementalized the same way (IDF shifts for every term on any insert), which generalizes: any component fitted on the corpus — PCA, k-means, ITQ, PQ codebooks, IDF — breaks the equivalence that makes incremental safe. Incremental does not fix the committed-blob cost: 1.00 MB dense + 6.36 MB postings per rebuild is ~1.5 GB of git history at 200 rebuilds, which wants the artifact published as a release asset instead. Not done: graduating the winning arm into ask.py, moving the artifact out of git, a second-repo check, and history/tombstone indexing (untestable here — 73 commits and zero files deleted-and-never-restored). |
code-index-duplication/ | 2026-08-05 | done — positive; shipped into repo-index | RESULTS.md + run.py + results.json | Should repo-index/ index .py as well as .md? This repo's own bekko-embedding-bench already measured dense retrieval as not beating grep at NL->code localization (r@5 0.656 vs 0.596, n=59, ns) — so the usual reason to index code is a measured non-reason here. But localization is not the failure this repo has with code: METHODS.md's duplication map records three independent reimplementations of one bench harness plus three more near-identical pairs, all found by hand after the fact. That map is an answer key written before this experiment existed, for an unrelated purpose. Scored leave-one-out hit@5 over 831 flat 60-line windows from 190 .py files (flat not AST, because AST-vs-flat was noise at p=0.424 in the earlier bench): querying with a file's own text finds a documented sibling 9/9 at ranks 1-3, and content-only scores identically to with-path-header, so it is content matching and not filename matching — the confound that would have made the headline hollow. NL-description queries get 8/9; grep handed the most distinctive def name out of the query file gets 8/9. Dense ties grep, it does not beat it (n=9), but the arms need different things: grep's needs a draft containing a distinctive name, the NL arm needs no draft at all. The single NL 'miss' is the index returning _lib/pipeline.py/_lib/textnorm.py — which is where that code was extracted to, per the same map, so it is the better answer against a stale key. Three process failures, all self-inflicted and all in the measurement: (1) run.py embeds its own NL queries verbatim and so retrieved itself, top-5 on 4 of 9 — excluding it moved content-only NL 6/9 -> 8/9; (2) the first number described excluding only the query file, a configuration the shipped tool does not use, so --file and the harness were both changed to exclude the query's directory before any number was recorded; (3) after adding .py to repo-index, keyword agreement looked like it fell 10/10 -> 7/10, but all three 'regressions' were the index returning the definition instead of a prose mention (ascii_fold -> _lib/textnorm.py, GRID_VERSION -> grids.py) and the grep arm was still restricted to *.md — a baseline scoped narrower than the system under test reports improvements as regressions. Matched arm: 9/10, rediscovery unchanged at 5/5. Shipped: repo-index indexes .md+.py (0.14 -> 0.27 MB) and grew --file. |
bekko-embedding-bench/ | 2026-08-04 | done — split verdict; one prior reversed, one default upheld | RESULTS.md + instances.json + recheck.py | Handoff claude-workspace#185: benchmark hotchpotch/bekko-embedding-v1 (a8m/a25m, 384-d Matryoshka mmBERT) for two separate decisions. Part A: the reversal did NOT survive a bigger sample — n=6 → n=59. The first run said bekko beats identifier rg at r@5 0.806 vs 0.667 and cleared the pre-registered gate. Re-mined to n=59 (630 PRs harvested, 97 candidates with live gold), dense/a8m lands at 0.595 against grep's 0.596 — a dead tie — and worse at r@10. No dense-vs-grep comparison is significant in any of the four cells. The gate now passes or fails depending on which cell you pick, which is itself the finding. The identifier-poor stratum is still n=1 of 59 — 10x the sample bought zero additional instances, independently corroborating the ~0.3% base rate. So the 2026-07 retirement of the semantic tier stands; the n=6 reversal was noise. Two things do survive: a25m > a8m is real (+0.061, 13 wins to 1, p=0.0018), which reverses the n=6 call that a25m doesn't earn its cost; and RRF(rg, dense) is directionally best in every cell (r@10 up to 0.762 vs grep's 0.682, bootstrap CI excluding zero but sign test p=0.09) — suggestive, not established. Cost at n=59: dense 200k tokens vs rg -l 315k, i.e. dense is now 1.6x cheaper. A code-trained encoder does not rescue it either: jina-embeddings-v2-base-code (161M, 768-d, 30 languages) scores r@5 0.630 against general-text bekko-a25m's 0.656 — it loses to the general encoder, at 6x the encode cost (612 MB / 61.9 min vs 124 MB / ~10 min), with no comparison significant. The obvious confound was ruled out: path-only retrieval (no code content at all) scores 0.304/0.370, so the code body is genuinely carrying signal — the specialization just adds nothing on top of it. Every arm clusters within noise of grep, and RRF fusion is the only thing that consistently helps. Part B is a regime choice, not a dominance. Official jina v5 nano q4 wins 11 of 12 iso-byte cells and owns the top quality rung — but bekko-a8m encodes a query 12.9x faster on 1 vCPU (11.3 vs 146.4 ms, 11.2x tokens/s), which is the entire design point of a 7.7M-active-parameter model and which an iso-byte table cannot see. The measured ratio matches the ~12x FLOPs ratio (4x384x1152 vs 12x768x3072), so it is architectural, not a q4 artifact. Result is an iso-quality ladder: bekko-a8m up to blog R@10 0.575 at 11.3 ms, a25m to 0.598 at 35.0 ms, jina alone above 0.60 at 146.4 ms. Keep jina when quality-bound or compute is amortized; take a8m when the reader is a 1-vCPU container or the corpus is large. End-to-end through remax_kb.read.KB.search, though, only 2.3x of that 12.9x reaches the reader — a ~50-60 ms constant (_stacked_simhash_encode rebuilding k Haar rotations by QR per query, from manifest params that cannot change) is 87% of bekko's query. Caching it per opened index is one line, verified to give identical codes and hits, and restores 11.6-15.1x — a finding about remax_kb, not about bekko. Swapping the projection does not fix it: every option is a per-query construction cost of 14-76 ms against a ~6 ms encode, and remax_kb v2's default srht is the slowest of them (1.4-3.0x slower than Haar at every dim), deliberately, because it is seed-only and bit-for-bit reproducible by a JS reader. remax's own rht_rotation at rounds=2 does reproduce its documented 1.5-1.8x -- a different function. Projection choice is a portability decision; the latency is a caching decision. Still no remax_kb swap made — the call is the deployer's, and the code-distribution gap (0.983 vs 0.888) is where bekko was advertised strongest. Matryoshka trimming vs quantization: quantization wins directionally at every budget — against the vendor floor d=64 (256 B, R@10 0.520), remex 1-bit @384 is 48 B at 0.564 — and the vendor's own HAKARI table agrees (binary@384 −12.93% vs truncation-to-64d's −17.51%). But a paired-McNemar audit at the end found this corpus cannot establish it: 179 chunks from 11 blog posts, where one query is 0.56 pp, and SEVEN OF EIGHT headline claims are noise — including the one I led with (remex 2-bit beats the uncompressed vector: +0.011, p=0.625). Only truncation-to-d=64 costs recall survives (p=0.009). The direction is consistent and matches the vendor's much larger eval; the demonstration is theirs, not mine. The one overwhelming result is Part B's: jina beats bekko on code-distribution R@1, +0.168, 31 discordant wins to 1, p<1e-5 — that, not the twelve correlated iso-byte cells, is the real basis for 'do not swap'. Compute went to the wrong arm: 78 min encoding 41,500 sklearn chunks for a 6-instance code-search benchmark, while every embedding-quality conclusion rode on 179 chunks encoded in seconds. Earlier passes also quoted an off-spec d=12 tier (strawman, retracted) and mis-priced shared structure (over-correction, retracted — remex's codebook is 28 B and the rotation is seed-derived). The R@50 ceiling (26/179) is partly harness artifact, has a 7.3% true shared floor, and is recovered by BM25 (14/26) and RRF (best overall, R@10 0.615) — not by query expansion (3/26, and R@50 drops), reproducing the repo's muninn-rm3 negative. 2-bit beats 1-bit in all 8 cells, so bekko is a Jina-side embedder and the SPECTER2 one-bit-beats-two result does not transfer. Confirmed the artifact facts independently (404.3 MiB → 124.1 MiB at cosine 0.99992 to its own fp32, holding on both distributions) and failed to reproduce the card's 5.5x OpenVINO-over-ORT claim (20.1 vs 21.8 ch/s on 4 vCPU). |
remex-vs-higgs-ablation/ | 2026-08-02 | done — mixed; 2 of 4 pre-registered predictions failed, and one published mechanism refuted | RESULTS.md + tables.md + gate.log + axes.png / marginals.png / seeds.png | Issue #8: does remex (exact fp32 norm + dense Haar rotation + scalar Lloyd-Max) buy anything for retrieval-index compression over the QuIP# -> HIGGS -> TurboQuant lineage (randomized Hadamard + per-block scale + Gaussian-MSE-optimal grid)? Full 2x2x2 factorial, 11 arms x 6 bit widths x 4 corpora (d=100/768/784/1024) x 5 rotation seeds x 2 metrics, scored against fp32 exact search rather than qrels. Only axis C moves. Rotation is null (-0.0004 recall@10) and norm handling is null (+0.0007); the codebook is an order of magnitude larger (+0.0082 cosine / +0.0112 IP), peaking at +0.035 recall@10 at 2-3 bits and decaying to zero by 8 bits, with the effect fmnist784 sweep (raw pixels, norm CV 31%, d=784) ran; axis B is flat there too (+0.0005 cosine / +0.0009 IP), so exact-norm storage is null even off cosine-trained encoders — and the published mechanism for the 1-bit remex win was refuted (it predicts the effect fades as norm spread grows; fmnist has the most spread and the biggest remex win, and reverses under cosine where the norm is divided out). Two further mechanisms were measured and refuted; the effect is now carried as MEASURED but UNEXPLAINED. Process: the two-sided calibration gate caught Lloyd-from-random-init producing grids worse than scalar at 6-8 bits, and a scheduled adversarial review then found five more blocking defects -- a stale codebook served by a cache keyed on the problem rather than the method (8-bit vector arm 87% worse than scalar), a Lloyd-Max MSE identity evaluated off the fixed point (+16% at 8 bits, in the direction that makes the gate more permissive), a 'provably no worse' guarantee that was argued rather than enforced, a block/sub-vector divisibility bug hitting only the HIGGS-like arm, and a gate that never certified the grids behind any glove result. Scoring q.xhat without dividing by ` |
lattice-representation-hypothesis/ | 2026-07-31 | done — negative result (opening thesis refuted by the experiment's own adversarial + WordNet arms) | RESULTS.md + THEORY.md + fca.py + noise_reversal.png | A Paper Skygest Bluesky post pointing at arXiv:2603.01227, "The Lattice Representation Hypothesis of LLMs" (LLM embedding geometry encodes an FCA concept lattice; meet/join as half-space operations). Hypothesis: the concept algebra has a broken join — FCA's meet extent is a bare intersection (exact under half-spaces) while its join extent is the closure of a union, and the paper's Definition 7 writes that join as a literal set union, contradicting its own Appendix B. Measured at scale: 0 meet phantoms across 9,615,370 concept pairs, join overshoot mean 0.60. Then the experiment's own arms killed it. (i) Definition 7's "conic hull" clause makes the join exactly R(Y_A n Y_B) under linear independence — verified 96/96 by an independently written Minkowski-sum LP, strict only under conic dependence (43/96, 34/96). (ii) The "phantoms" are not errors: they are the least upper bound, and the gold label in the paper's own task (join of {dog, wolf} is canine, which contains foxes). (iii) The premise was mathematically wrong: meet = objs_of(B1 u B2) and join = objs_of(B1 n B2) are both plain half-space intersections (0 identity violations over 30 configs x 8 seeds); the join needs fewer constraints (0.83 vs 5.55). The only non-representable object is the plain set union, which isn't a lattice operation — and it is indeed recovered worst of the three. The one measured effect — join degrades more slowly than meet under probe error on Jaccard (5% flip: 0.939 vs 0.835) — carries a 12x target-size confound, flips sign on symmetric-difference error in one of two contexts (cross: join 0.110 vs meet 0.028), and its size-controlled version is reproduced more strongly by random-direction controls; the proposed "closure absorbs noise" mechanism is contradicted by a negative overshoot-vs-error correlation (-0.44). Surviving caveats, aimed at the paper's setting: WordNet noun hypernymy gives an extremely thin lattice (15 concepts from 150 objects x 13 attributes, 0/78 cross-cutting attribute pairs, 66% of meets empty, ~42% of joins the top element); learned attribute directions can be exactly antipodal (mutual coherence 1.0000, since living_thing and artifact are complementary), so the canonical form's linear-independence assumption is not secured by `d >> |
svgview/ | 2026-07-30 | working on Linux; Windows build never run on a real machine | README.md + src/ | andri.dk on Bluesky, arguing that launching a full browser to render PDF or SVG is "bonkers insane" — narrowed in his own reply to systems doing it internally. Tested the SVG half by building the alternative: a native Windows-first viewer wrapping resvg, ~600 lines. Measured here: 4.8 MiB executable, 12 MiB resident, 16 ms exec→window, 20 ms parse+render to a 1000 px PNG. Verdict: he is right about SVG and resvg had already done the hard part; the argument does not transfer to PDF, where the honest options are wrapping Chrome's own PDFium or accepting hayro's coverage gaps. Windows compiles in CI; the file dialog, icon, and association scripts are unverified. |
erdos-gyarfas/ | 2026-07-28 | open problem; partial results | README.md + note.html + tutte_coxeter_lemma.py | this session |
ms13-campaign/ | 2026-07-24 | closed (compute exhausted; open maths recorded) | SUMMARY.md (academic writeup) + NOGOS.md (ledger) + BLOGPOST.md | issue #169: campaign against Morell–Skutella Conjecture 1.3 (two-sided unsplittable-flow rounding). No counterexample. Produced instead: a reduction showing 1.3 restricted to 2-path instances is a linear-discrepancy question on network matrices with demand-scaled columns — a connection neither literature appears to draw (full-text greps of TVZ/Swamy/MSW25 find no mention of Doerr, "linear discrepancy" or "totally unimodular"); a theorem settling that question for k=3 (R = 3/4 exactly, unconditional, census complete through m=10, exact branch-and-bound on the 2 maximal classes); and a ledger of 20 refuted families/claims including two conjectures of our own (12.1 refuted at k=4, the staircase conjecture at k=8). Rediscoveries correctly identified as such: the tightness gadget is Morell–Skutella Fig. 3, the equal-demand bound is Doerr 2004. Enumeration shown dead by arithmetic (~2,070 h at k=4). Open: Q7′ (column-scaled Doerr bound, general k). Methodology notes: a false-positive "counterexample" caught at the certificate gate when two independent verifiers turned out to share one blind spot; every over-claim had the same shape (clean at k≤6, false at larger k). |
ssuf-beta/ | 2026-07-24 | done (scoped) | RESULTS.md + engine.py + calibration.py + family.py | claude-workspace#165: quantitative hunt for the SSUF cost-preserving violation constant β* following the Goemans/DGG conjecture disproof (16/15 < β* ≤ 2). Built an exact-rational β* engine (breakpoint-enumerated convex-hull membership LP via sympy's tested simplex, after a hand-rolled one failed its own sanity test). Calibration against the real Rybin instance was blocked — no arXiv writeup exists, and this session's WebFetch can't reach the source X thread (HTTP 402, no working mirror) — so calibrated instead against a fully hand-derived, independently-constructed triangle-conflict instance (β*=1/2, exact match) and swept a parametrized generalization of it — that family's β* has supremum exactly 1 (approached, never attained or exceeded), a clean negative result short of even the original refuted β=1 bound. Literature gate confirmed TVZ's planar +2·d_max bound and ring-loading bounds (1.1D/1.3D) from primary sources. Honest scope cut: no claim here reproduces or exceeds β*=16/15; gadget search, ms13 engine sweep, and the ring-loading secondary target were not attempted. |
discrepancy/ | 2026-07-24 | done (D(4) + n≥17 deferred) | RESULTS.md + growth.png | issue #166: certified discrepancy lower-bound records — Komlós per-size K(n) + Beck–Fiala small-t exact values, max-min engine with exact certificates. Literature gate first (logged on the issue): Kunisky's K ≥ 1+√2 record reframed Target A to per-size records; 2025 Bansal–Jiang resolution of Beck–Fiala for t ≥ log²n left small-t exact values open. Beck–Fiala: D(2)=2, D(3)=3 exactly — the CEGAR SAT search rediscovered the Fano plane from scratch and proved it's the minimum-ground-set (n=7) witness; D(2)=2 (triangle minimal); D(4,n≤9)=3 with D(4)∈{3,4,5} open (PG(2,3) computed weak: disc 2). Small-t truth sits on D(t)=t, far below 2t−3. Komlós: certified rational per-size records K(3)≥1.571, K(4)≥1.731 (beats Kunisky's own n=4 tree matrix 1.707), K(5)≥1.785, K(7)≥1.830; proved the "exact Δ vs published δ" gap on Kunisky's family is empty (Δ=δ, one-line proof) before wasting compute on it. All records verified by an independent second code path (verify_certificates.py); calibration gates G1–G9 green first. |
woodall/ | 2026-07-23 | phase 1 (verifier+calibration done, search deferred) | README.md | issue #163: SAT/MIP dijoin-packing verifier for Woodall's conjecture τ=3 counterexample search (Goemans-fall follow-on). Built a CEGAR SAT verifier (python-sat/cadical+glucose) for ν(D,u) and brute-force τ; transcribed Schrijver's (D1,u1) counterexample directly off a 600dpi render of Figure 6 in the Feofiloff survey PDF (not from memory). Calibration gate 1 (Fact 7.1, ν=1/τ=2) passes exactly, after catching and fixing a real transcription bug (one dashed arc misread) via cross-checking the graph's derived 3-fold symmetry and the paper's "4 critical cuts" claim against an initial read that gave only 3 and a spurious ν=2. Generalized D1 into a ring-of-length-2i family from its own orbit structure and validated the paper's odd/even parity claim (i=3,5 counterexample, i=2,4 not) exactly. A first light random-search pass (16,000 trials) found 0 candidates, confirming unstructured random generation is a weak filter-pass rate (0.15%) for τ≥3 — the real search needs the structured null-arc-resolution and Williams-catalog generators, explicitly deferred (not silently dropped) along with gate 2 (Cornuéjols-Guenin D2/D3 calibration). |
pdf-streaming-test/ | 2026-07-05 | done (shipped) | RESULTS.md + test_streaming.js | "Speed up the UX of austegard.com/web-utilities/pdf-text-extractor by streaming pages one-at-a-time; also check if parallelising improves throughput." Added a bounded worker pool + streaming display (pages append to the output pane in strict page order via a nextToFlush cursor as they complete) and a URL API knob (`&concurrency=1 |
atproto-pad-login/ | 2026-07-04 | done | RESULTS.md + pad_login.mjs + 04_connected.png | "you can log into ATProto as muninn — try this pad URL" → Drove austegard.com/bsky/pad.html end-to-end as muninn.austegard.com via Playwright/Chromium: opened the shared-pad URL, clicked Join, filled the login dialog with $MUNINN_BSKY_HANDLE + $MUNINN_BSKY_APP_PASSWORD, waited for rtc.login() to establish the PDS session and rtc.connect(peerDid) to send its WebRTC knock. Result: status flipped Local only · 0 peers → Connected · 0 peers (session up; the invited peer just isn't online sharing, so no data channel — a one-sided join can't complete). Load-bearing finding along the way — headless Chromium via $HTTPS_PROXY returns net::ERR_CONNECTION_RESET on every HTTPS goto unless launched with --ssl-version-max=tls1.2. Curl and Node fetch work; Chromium doesn't. NetLog shows the CONNECT tunnel returns 200 and Chromium's TLS 1.3 ClientHello (~1700 B, ECH extension included) goes out — then the socket is reset (SOCKET_READ_ERROR net_error=-101 os_error=104) before any TLS response. --ignore-certificate-errors, ignoreHTTPSErrors: true, and even --disable-features=EncryptedClientHello don't help — only pinning TLS 1.2 does. So the session's egress-gateway MITM terminator can't complete a modern Chromium TLS 1.3 handshake. Side finding: the docs claim the browser NSS trust store is pre-loaded with the proxy CA; ground truth is that /root/.pki/nssdb was empty — certutil -L returned only the header. Adding /root/.ccr/agent-proxy-ca.crt with certutil -A -t 'C,,' is needed for the proxy-terminated cert to verify after the RESET is worked around. |
session-relay/ | 2026-07-02 | done | RESULTS.md + relay.py + README.md | "How might this be accomplished?" → Joshua Shew's Bluesky post: one Claude session built a relay service so two other sessions could chat and share inbox best practices. Reproduced with ZERO deployed infrastructure — the shared Turso DB is the relay: one relay_messages table (channel, sender, AUTOINCREMENT seq as cursor) + a ~190-line CLI (init/post/poll/wait/history, 5xx backoff). Live run: two concurrent agents with disjoint seeds (muninn-a: docs/architecture.md lifecycle; muninn-b: experiment conventions) negotiated a joint 5-item "experiment survival checklist" over channel hub-coord — 4 messages, one round-trip each way, ~75s, clean CONSENSUS/ACK termination, with verifiable two-way knowledge transfer (final checklist contains facts neither seed had alone) and genuine critique (b fact-checked a against Error Patterns, restructured its draft). Protocol findings: table-global seq leaks cross-channel counts (cursor discipline handles it); shell quoting is the ergonomic tax (mandate the post ... - stdin path); front-loaded opener + explicit delegation → 1-round convergence. Transport is container-agnostic (HTTPS to Turso) — works across CCotw/Claude.ai session boundaries, unlike the /tmp-files original. Caveat: for two Muninn sessions the marginal value over shared memory is live negotiation, not knowledge transfer. |
omnigent-library-eval/ | 2026-06-29 | done | RESULTS.md | "Is this anything you could make use of?" → omnigent-ai/omnigent, the open-source meta-harness (5.3k★, 18 days old, alpha 0.3.0.dev0, Apache-2.0) that orchestrates 11 vendor harnesses (Claude Code/Codex/Cursor/Pi/…) with policies + sandboxing + multi-device. Verdict: NOT a claude-workspace dependency. It's a competing harness — a server+CLI that wants to OWN orchestration/model-routing/sandboxing, all of which CCotw already provides; the Python SDK is a client to a running omnigent server, not an importable lib; it can't even run here (no model key by design, wants Node 22/tmux/bwrap). But three borrowable wins: (1) free — omnigent/spec/skill_sources.py reads the SAME SKILL.md format with a claude family, so Muninn's whole skill library loads with zero porting; (2) the Polly orchestration prompts — though the cross-vendor review discipline they encode Muninn can ALREADY run in CCotw today (Gemini reviewer via the Cloudflare AI Gateway, gemini_generate()); omnigent only uniquely adds cross-vendor implementer agents (Codex/Cursor/Pi as full coding harnesses), a narrow win; (3) the ALLOW/DENY/ASK three-level policy engine (docs/POLICIES.md: blast_radius/spawn_bounds/cost_budget) as a reference design if Muninn ever formalizes its settings.json permissions. Rec: keep on radar as a place to run Muninn-the-agent outside CCotw; don't take the dep. No code executed (alpha server, no key). |
q4-official-vs-ours/ | 2026-06-28 | done | RESULTS.md + run_batched.py + results.txt | remax_kb#23: run the q4 head-to-head claude.ai couldn't (egress blocks both weight hosts) — does the model authors' own onnx/model_q4.onnx (137.8 MB, HF Optimum) match/beat our JinaQ4ONNXEmbedder build (~170 MB, remax_kb#14) on retrieval fidelity to fp32, deciding whether to upload ours to HF? NFCorpus, 2058 docs/100 queries, 4 vCPU. Result: ours is DOMINATED — official wins every axis at once: smaller (138 vs 170 MB, −19%) AND strictly more faithful to fp32 on all four metrics (nDCG@10 0.4291 vs 0.4250, per-doc cos 0.976 vs 0.974, recall@10-vs-fp32kNN 0.870 vs 0.862, Spearman ρ 0.980 vs 0.976). The hoped-for edge — our int8 embedding-table mop-up holding where Optimum's generic q4 degrades — is refuted; Optimum handles the EuroBERT Gather at least as well. Decision: do NOT upload ours; close the loop. embedders.py now steers users to the official asset. Side findings: docstring's "cosine 0.975 to fp32" holds (ours 0.9743 on this larger subsample); q4 is NOT faster than fp32 on CPU (~13–14 min each — int4 dequant offsets size; "~2x faster decode" didn't reproduce); patched the bench's one-shot encode(all_docs) OOM (~26 GB attn Expand) with numerically-identical mini-batching. Reinforces the prior-art-check lesson: the upstream repo already shipped a smaller, better q4. |
jina-remex-vs-remax/ | 2026-06-27 | done | RESULTS.md + score_fidelity.py + fidelity.png | "Can we apply the remex (not remax) quantization optimizations to our Jina q4 embedding — as a practical compressed-Jina format, not necessarily beating remax at a byte budget?" First clears up the two q4s: q4-the-model (JinaQ4ONNXEmbedder, weight quant, fp32-parity float output) is orthogonal to remex/remax, which compress the vectors; remex replaces the remax 1-bit step, not the q4 model. Recall-vs-qrels saturates both ways (muninn fp32 ceiling 0.90/1.00; NFCorpus fp32 floor 0.241 — embedder-limited), so switched to the saturation-proof metric: fidelity to fp32's own ranking (recall@k vs fp32-kNN chunk-level, Spearman ρ, recon cosine — remax's own bench metric). Result: remex (rotation+Lloyd-Max scalar) ≫ remax (1-bit SimHash) at every byte budget — NFCorpus n=120: remex ρ 0.92–1.00 vs remax 0.63–0.74; remex 4-bit @ d768 (384 B) near-lossless (ρ 0.998, R@10-vs-fp32 0.96); remex 2-bit @ 192 B (ρ 0.978) crushes remax d768/k2 @ 192 B (0.741); remex 1-bit @ 96 B beats every remax config. Headline: bits beat stacks — graded magnitude (Lloyd-Max) dominates more sign-bits (stacked SimHash); full-dim-low-bit > truncated-dim-high-bit. remex is the data-oblivious Haar rotation our own ITQ-rejection work (remax#46) endorsed. Rec: ship remex into remax_kb as the near-lossless/mid-byte codec (default 4-bit @ d768); needs a new SPEC binarizer type + ADC scan path (numpy+scipy), additive not drop-in. Caveats: muninn n=5; norms excluded from B/row (Jina unit-norm). |
remax-hamming-speedup/ | 2026-06-26 | shipped (PR) | RESULTS.md + bench.py + latency.png | remax_kb#15: the 1-bit Hamming scan (_hamming.hamming_scan, popcount LUT gather) was ~10× slower than a BLAS float-cosine search at small N, forfeiting the latency half of the 1-bit format's promise. Benchmarked the issue's candidates head-to-head (LUT vs bitwise_count u8/u64 vs ±1 BLAS matmul vs float cosine) over N=600→1M, single-thread BLAS. Winner is the cheapest candidate (approach 1): np.bitwise_count over a uint64 view of the XOR — ~10× over the current LUT and faster than BLAS float cosine at every N (e.g. N=10k: 0.58 ms vs LUT 7.83 ms vs cosine 2.08 ms), zero-copy, codes stay bit-packed (256 B/row). The ±1 matmul (approach 2) is 2–6× slower and costs 8–32× RAM (OOM at 500k); compiled SIMD (approach 3) is unnecessary. Issue's success criterion (Hamming ≤ cosine at N≥10k without losing storage) exceeded — holds at all N. Shipped: _hamming.py + read_v2.py swap to a shared _popcount_rows fast path with a numpy<2.0 LUT fallback (numpy≥1.24 floor preserved) + remax-free regression test (remax_kb PR). Bit-for-bit exact vs the old LUT across 8 realistic (dim,k) widths incl. non-multiple-of-8 byte rows. |
lfm25-230m-verify/ | 2026-06-25 | done | RESULTS.md + run.py | "test this Bluesky claim — RUN the model" (@sungkim post) on Liquid AI's LFM2.5-230M. Spec-checked the HF card, then loaded and ran the weights on container CPU to verify the runtime-checkable claims. Confirmed: 229.7M params, Lfm2ForCausalLM/model_type=lfm2, open weights, coherent CPU generation at 29.2 tok/s (fp32, 4 vCPU — the slow path vs card's quantized 42/213 tok/s). Training claims (19T tokens, distill-from-350M) aren't runtime-observable. Minor nuance: post/card say 32K context, loaded config carries 128K positional ceiling. Verdict: post accurate. |
lfm25-embedder-remax_kb/ | 2026-06-25 | done | RESULTS.md + lfm25_embedder.py | Follow-on from lfm25-230m-verify: if the 230M runs locally, does Liquid's LFM2.5-Embedding-350M work as a third remax_kb embedder — in-process CPU, no 847 MB Jina download, no Gemini API key? Wrote an LFM25Embedder (sentence-transformers, CLS-pool, 1024-d, query:/document:) on remax_kb's Embedder protocol. One real fix: the model's bidirectional remote code predates transformers' seq_idx shortconv kwarg (5.12.1 TypeError) — patched at our layer, not the model's cached files. Tiny-corpus 1-bit pipeline: 3/3 topical top-3 (matches Jina torch). Full-float head-to-head on the 73-post muninn corpus: LFM2.5 0.73/0.83 R@5/R@10 — below lexical 1.00/1.00 and Jina v5-nano 0.90/1.00, despite being the larger model; embeds slowly (1.1 chunks/s fp32, ~19.5 min). Verdict: viable where in-process/no-key/no-network is the hard constraint and retrieval is tolerant; Jina still wins on quality. Closing analysis: remax's 1-bit step is corpus-global (can't quantize up front); per-chunk-memory levers = fp16 buffer, streaming Welford mean, dim-truncation (non-MRL caveat), and int8 model quant for RAM+speed. |
muninn-rm3/ | 2026-06-25 | done | RESULTS.md + bench.py | The platform-less, agent-less site-search floor (no model to host, no Claude to expand the raw query). Pure RM3 vs plain BM25 on the muninn corpus (5-query phase0 gold). RM3 is a dud — identical R@5, hurts R@10 (1.00→0.90 whole-doc); skip it. But plain BM25 (whole-doc) = 0.833/1.00 — ties Jina-q4→remax (0.833/1.00) at ZERO inference/agent/per-query cost, all in the Worker. Only the agent-expansion path (1.00, unavailable to site search) beats it. Caveat: these are in-vocab acceptance queries; the residual Gemini buys is vocabulary-divergent (paraphrase) queries — BM25/RM3 can't bridge that lexical gap, no platform-free option does. Recommendation: drop Gemini for plain in-Worker BM25 IF muninn searches are keyword-ish; keep Gemini only if paraphrase-heavy. Decision input: actual query mix. |
muninn-embedder-bakeoff/ | 2026-06-25 | done | RESULTS.md + jina_remax.py + embed_one.py | Test our special-case Jina→remax as a Gemini replacement for muninn search (+ how it's practical). On the muninn corpus (5-query phase0 gold): Jina-q4 full-float = 0.900/1.000 (identical to fp32 — q4 is free); Jina-q4→remax 1-bit at d=512/k=4 (256B) = 0.833 R@5 / 1.00 R@10 (near the float ceiling; the shipped d=256/k=8 default is the weak config at 0.667 — dims beat stacks here too). So Jina→remax is a credible Gemini replacement; lexical (1.00) still edges R@5. Practicality: indexing is free (offline CI pack with Jina), only the online query-embed is hard — Worker can't host 170MB (10MB bundle/128MB mem), Workers AI has no Jina, so serve queries from a CF Container/endpoint running q4 (replaces the paid per-query Gemini call with cheap fixed compute; q4 is what makes that light). Migration = corpus re-pack (Jina space, d=512/k=4) + query-embed service swap; KV 1-bit mechanism unchanged. Sidebar: CF-native Workers-AI BGE alts underperform (bge-large 0.733/bge-base 0.667). Caveats: n=5; no live Gemini (Jina-0.90 float is the reference); embeddinggemma-300m HF-gated. |
rotation-decorrelation/ | 2026-06-25 | done | RESULTS.md + sweep.py | "How can I trust either end of the ITQ pendulum? Embedder-specific? Test the decorrelation angle." Controlled study, remax/bench metric (self-retrieval recall@10 vs float32 kNN), pure numpy on precomputed caches: SPECTER2 (specialized) + Jina-v5 (general), k-ladder {1,2,4,8}, simhash/itq/decorr, in-corpus vs transfer, 3 seeds. Resolution: the pendulum is 3 interacting axes — (1) k: ITQ wins k=1, decays/reverses on the ladder (#46's mechanism reproduced); (2) protocol: ITQ in-corpus overfits (gap ~0.02–0.03 SPECTER2, ~0.04–0.05 Jina; simhash/decorr zero gap); (3) embedder: ITQ's edge is 3× bigger on general Jina (+0.048 k=1) than specialized SPECTER2 (+0.015) — partly embedder-specific but mostly overfit. My NFCorpus 'win' = perfect storm (general × in-corpus × k=1); #46 = specialized × transfer × ladder. Honest config (transfer+ladder, k=8): simhash beats itq on BOTH (SPECTER2 −0.029, Jina −0.015). Decorrelation (α-mix) is a wash — ties simhash everywhere, no overfit gap. Verdict: keep parameter-free SimHash (#46 upheld, now for general embedders too); the open lead doesn't pay. |
recall-per-byte/ | 2026-06-25 | corrected (re-derived rejected work) | RESULTS.md + sweep.py | Generative-thinking move (random stimulus "river") reframed compaction toward information-per-stored-bit (the rotation). Recall-per-byte bake-off on cached NFCorpus fp32 Jina vectors (600 docs/120 q): remax StackedSignBit vs SimHash vs ITQ vs PQ. Apparent headline (ITQ/PQ@16B beat shipped remax@256B) does NOT hold: ITQ was already tested rigorously and rejected — remax#46/PR#47 (closed unmerged, SPECTER2 n=10k): learned ITQ loses to centered SimHash at every ladder rung, deficit grows with k. My k=1/600-doc "win" is the exact in-corpus overfit artifact #46 diagnosed (transfer rotations beat in-corpus ones); I never tested the stacked ladder where ITQ definitively loses. PQ not novel either (remex/TurboQuant owns codebook compression; my 256-centroid book on 600 docs flatters it). Lesson: recall() prior art before claiming a win. One genuinely open lead (from #46's parting question): a decorrelating joint multi-rotation objective — diverse-across-stacks, better-than-random per-stack — validated on SPECTER2 with transfer + ladder, explicit kill criterion. |
jina-int8-remax_kb/ | 2026-06-25 | done | RESULTS.md + quantize.py + bench.py | Pivot from quantizing the weak LFM2.5 to quantizing the strong embedder: int8-quantize Jina v5-nano's ONNX export to cut its 847 MB download. quantize_dynamic(QInt8) → 212 MB (4.0× smaller). Head-to-head on the 73-post muninn corpus (same stage_b.py methodology): fp32 0.90/1.00 @ 8.3 ch/s vs int8 0.83/1.00 @ 16.7 ch/s — 4× smaller, 2× faster, R@10 untouched; the R@5 dip is one query (Q3) sliding past rank 5. fp32 reproduces the prior 0.90/1.00 exactly (harness validated). int8 Jina dominates the LFM2.5 local option on every axis (R@5 0.83>0.73, 212<919 MB, 16.7≫1.1 ch/s). Standing rec stays lexical 1.00/1.00; int8 Jina is the real-vector fallback. Follow-up (NFCorpus, 600 docs/120 qrel-queries, full-float + 1-bit recall): overturns int8. Per-tensor dynamic int8 is domain-fragile — 0.445 per-doc cosine to fp32 on medical abstracts (vs 0.83 R@5 on muninn tech text; probe rules out seq-length: 0.44@256 vs 0.41@512). Blockwise 4-bit q4 matches fp32 (0.975 cosine, cos R@10 0.241 vs 0.242, 1-bit R@10 0.208 vs 0.222) at 170 MB < int8's 212 MB; q2 too far (0.73). Embedding-table workaround: MatMulNBits leaves EuroBERT's ~400 MB Gather fp32 (naive int4=465 MB > int8), so int8-mop-up the embedding → q4=170/q2=141 MB. 3-bit unsupported (ORT {2,4,8}). Answer to "is int8 the floor before 1-bit?": no — 4-bit blockwise is smaller, more faithful, domain-robust; floor is 4-bit; the int8 embedding table is now the size-dominant cost. Recommended quantized Jina = q4, not int8. **SHIPPED: q4 integrated into remax_kb as JinaQ4ONNXEmbedder (oaustegard/remax_kb#14) — embedder + deterministic scripts/build_q4_onnx.py + gated test; model.q4.onnx hosted (SHA-verified) on the jina-v5-nano-mirror release; fp32-parity query cosine 0.977; opt |
Truncated — view the full README on GitHub.
Contributors
Similar repos
Languages
Python
86.6%
HTML
5.2%
ASP.NET
3.0%
JavaScript
1.9%
Shell
1.3%
Lean
1.0%
README
Experiments
Random vibe-experiments — one-off explorations and data products. Each
subdirectory is self-contained: scripts, data, and a results file
(RESULTS.md for pipeline runs, README.md for build/recipe artifacts,
or the artifact itself when it speaks for itself, like an HTML page).
Trust conventions. ANCHORS.md registers every published
constant in use with its covered range, because a range gap is invisible from
inside a green run. Each experiment should carry an ERRORS.md (what was wrong,
how it was caught, which direction it pushed the conclusion — the base rate is
the most useful calibration number about a body of work) and a recheck.py
(a sub-5-minute fixture that checks the prose against the artifacts, so the
writeup and the data cannot drift apart between full rebuilds).
remex-vs-higgs-ablation/ carries all three and is the reference shape.
Results are reported as they came out. Several of these are negative results, one is a correction of earlier rejected work, and one turned out to reproduce a 2004 paper rather than extend it — all labelled as such rather than quietly dropped.
Migrated from oaustegard/claude-workspace/experiments/, which is a
session-boot repo and was the wrong home for 37 research projects.
Index
| Experiment | Started | Status | Results | Origin |
|---|---|---|---|---|
jev-tag-encoder/ | 2026-09-23 | done — one Jev call per document answering 256 "The document is about ." Nouls works as a zero-shot topic labeller and fails as an RRF leg beside dense; without dense, Jev carries an interpretable pipeline. 24 arXiv categories mapped to tags: micro-F1 0.636 [0.616, 0.659] with no labels, between gemini-embedding-2 probes on 50 (0.517) and 200 labels (0.714). A probe on the 256 raw probabilities beats the dense probe at 50 labels (0.566) and trails at 500 (0.697 vs 0.778); standardizing the probabilities first costs up to 0.22 F1. On SciFact the general tags alone score nDCG@10 0.18 to 0.34 and as a third RRF leg cost 0.087; a 256-tag taxonomy fitted to SciFact (KMeans clusters named by Gemini) lifts that to 0.386 alone and a 0.055 cost, still below BM25 + dense, and a perfect labeller of it (query-to-centroid cosine) scores 0.183. Round 4, dense ruled out: BM25 + λ·fitted-tag score (z-scored sum, not RRF) goes 0.662 → 0.699, and reranking its top 20 with one pairwise Noul ("The document provides evidence that supports or refutes the claim.") reaches 0.763, against RRF(BM25, dense) 0.774. A 3-bit code (sign bit plus logit-spaced confidence buckets) matches the floats on retrieval (−0.011); 1 bit keeps zero-shot F1 and loses 0.18 micro-AP. 7,449 calls, p50 0.59 s, ~$1.30 total, paced at 48/min under the AI Gateway's 50/min limit; Round 3's 5,483 direct TypeSafe calls ran at 600/min, p50 0.46 s; reruns flip 5 of 7,680 values across 0.5 | RESULTS.md + tags.txt + jev.py + data.py + encode_all.py + embed.py + fit.py + classify.py + retrieve.py + ablate.py + quant.py + quant_eval.py + probe_codes.py + build_domain_tags.py + tags_scifact.txt + encode_domain.py + retrieve_domain.py + rerank.py + recheck.py + vectors/*.parquet + results/*.json | #98: the claude.ai probe (c98b700) showed 256 Nouls fit one call; run the six-step spec at corpus scale from code. Oskar mid-run: "test a 2 bit and 3 bit version (using confidence buckets for the extra bits)"; then: does a per-domain tag set change the SciFact result? And with dense out, what is the best interpretable pipeline? Prior art: QA-Emb (arXiv 2405.16714), binary answers to task-generated questions. |
subagent-context-filter/ | 2026-09-22 | done — writing only the task and letting Jev pick the parent transcript's chunks works on fact-carrying delegations: 99/105 facts with ~20k tokens of selected chunks vs 102/105 with the full ~81k-token transcript (paired −2.6 to −2.9 points, CI reaching zero) and 82/105 with a brief the same model wrote. The brief's whole gap is 5 of 32 tasks where it pointed the subagent at a lookup instead of stating the fact; elsewhere every arm is within two facts. Filter median 1.1 s and $0.002 per delegation; the brief took 528 output tokens and 23 s. Paging at 4x view ties the condensed window on these ≤97-chunk sessions and fixed a 181-chunk case. Found: TypeSafe's edge WAF blocks some transcript content, surfaced by the AI Gateway as HTTP 402 "Payment error from model using BYOK"; the filter bisects around it. Shipped as the delegating-with-context skill + PreToolUse hook | RESULTS.md + ERRORS.md (5) + chunking.py + jevfilter.py + context_hook.py + preview.py + run_eval.py + adjudicate.py + analyze.py + recheck.py + results/summary.json (counts only; transcripts are private) | Oskar: "wrap subagents in a jev-aided transcript-parser which per transcript chunk determined whether that chunk should be included for a given subagent task", then "page in 30k token increments" and "updatedInput is too late though? This is why a skill is needed". 8 archived sessions, 32 tasks written blind by Sonnet labellers, five arms on a tool-less claude -p executor, regex scoring plus a blind adjudicator validated in-batch (TPR 100%, TNR 95%). Pilot and design in memories 90ac1ab8, 9557f376, cb532ad0. |
sparseup-tag-probe/ | 2026-09-21 | done — negative for the learned expansion. A linear map from SPARSEUP's 50k-dimension sparse vector to Muninn's 325 human tags exists (micro-AP 0.612 vs 0.043 prior), and word + char TF-IDF over the same text carries more of it: 0.664, paired difference −0.051 [−0.059, −0.044]; on the encoder's own 512-token window TF-IDF still leads by 0.044. P@5 ties. gte-small is below both (0.576). Binarizing the SPARSEUP vector costs 0.021 micro-AP and 0.008 P@5. Untrained, 16.4% of tag instances appear as an exact expansion token at k = 32 (18.9% with every BPE piece); 227 of 325 tags are two or more pieces. The pre-registered micro-F1@0.5 ordered the arms the other way and was measuring calibration. Round 2, the 2024 map step as the first layer: a binary vector over K spaCy noun phrases selected per fold by frequency reaches micro-AP 0.369 at K = 512, 0.453 at 2,048 and 0.481 with all 7,655; word unigrams beat phrases at every K (0.573 with all 16,316) and the embedding-dissimilarity filter costs up to 0.035. Round 3, retrieval: a binary vector over all 5,827 tags (5.7 active per memory) finds cited memories as well as gte-small (R@10 0.667 vs 0.648, tie), holds on the recency-controlled half, and fusing the two beats either (+0.059); PMI co-occurrence expansion never helps and hurts at weight >= 0.5; TF-IDF leads every single arm (0.808) on refs written after lexical recall. Round 4: gemini-3.5-flash-lite re-tagging 300 of those memories retrieves at R@10 0.331 with a frequent-tag hint prompt and 0.461 when asked for specific names, against 0.697 for the original tags; the original tags' retrieval lives in their rare, specific tags (df < 20 alone: 0.742; df >= 20 alone: 0.309). Round 5: shown the tags of its five nearest older memories, flash-lite reaches 0.565 and 0.34 Jaccard with the originals; copying those neighbours' tags with no model reaches 0.581 at k = 10, so the context, not the model, carries the gain. Round 6: with the cited memories' tags in the prompt, flash-lite reaches 0.629 forward (−0.067, CI touching zero) and 0.617 on the leak-free reverse direction against 0.674 for the originals and 0.626 for gte-small; a no-model union of context tags matches it at R@10 and loses on MRR | RESULTS.md + PLAN.md + ERRORS.md (5) + fixture.json (ids, cleaned tags, hashes; no memory text) + build_fixture.py + fetch_text.py + encode.py + arm1_recall.py + probe.py + select.py + make_tables.py + recheck.py + phrase_vocab.py + retrieval.py + tag_model.py + flashlite_tags.py + results/*.json | muninn-utilities#137, from Oskar's 2024 binary tag-vector idea meeting Tom Aarsen's SPARSEUP thread (memory 5312cc75): if a SPLADE-style encoder's dimensions are vocabulary tokens, do they line up with a human tag vocabulary? 3,457 memories (37 private excluded), 3,315 labelled, one shared stratified 5-fold, one-vs-rest LR, C swept {1…10000} and selected per arm by micro-AP because TF-IDF and gte-small rows are unit-norm and SPARSEUP rows are not. Eight predictions, six wrong on F1@0.5, four of those right once the threshold is removed. Two Sonnet workers wrote the arm and check scripts; a scheduled adversarial read found the threshold artefact and the truncation asymmetry. |
modernbert-bidirectionality/ | 2026-09-20 | done — two results that disagree, and the trained one wins. Ablation on the trained encoder: right-context dependence is redundant per layer (any one of ModernBERT-base's layers 1–21 causal costs ≤ 0.14 nats) but not confinable (the 8 global layers alone retain 33%), and local rather than anchored (an 8-token lookahead in every layer retains 79%; [CLS]/[SEP] alone is worse than nothing; punctuation ties the content-free masked positions). Adaptation from ettin-decoder-32m, 6M tokens per arm: bidirectional attention in the 4 global layers with the rest causal ends at 2.945 nats, full bidirectionality 2.996, the 8-token lookahead 3.164, causal control 4.181. The dQwen3.5 cut is at least as good as flipping every layer, and the untrained ablation ranked the two cuts backwards | RESULTS.md + PLAN.md + PLAN-adapt.md + PRIOR-ART.md + probe.py + masks.py + adapt.py + results/ | CCotw handoff from the claude.ai session (Muninn memory 879c3575, title only survived): does non-causal dependence concentrate in few layers and few positions?, motivated by dQwen3.5 making only the attention quarter of a Gated-DeltaNet hybrid bidirectional. Per-layer boolean masks injected by forward pre-hook; 120 arms on ModernBERT-base and 69 on ettin-encoder-32m over 9,728 masked wikitext tokens. The pre-registered all-causal reference collapsed to 23.5 nats (above uniform), so an in-distribution left-only reference was added after the smoke. Oskar then asked for the training test: five attention patterns adapted from the encoder's causal twin at equal budget, six predictions pre-registered, two wrong, both the ones the ablation had motivated. Training repairs a layer cut completely (0.43 untrained → 1.04 of the reachable benefit) and a position cut only partly (0.65 → 0.86). |
msd-context-classifier/ | 2026-09-20 | done, three rounds plus checks: the paper filter's 0.98 AUC was the negative sampling — against PubMed's own similar-articles it is 0.82, and the uncurated modern population (9,754 PMC full-text MSD papers) against its neighbours is 0.75 on SPECTER2 with every nonlinear classifier lower citation-graph features adding three points at most, and TF-IDF + LR within two points of the embedding (BM25 nearest-known-paper retrieval is below chance against the neighbours, which are the lexically nearest papers by construction); SPECTER2 is a recall filter, not a classifier: full-text search first, embeddings only over the non-open remainder. Round 1 (classifier): a seven-way classifier over mesoscale.com's site sections scores 0.91–0.96 on held-out pages and collapses to 0.21–0.35 on question-shaped inputs, routing 65–76% of questions to support; 140 model-written questions train a frozen gte-small probe to 0.78 on questions, 617 pages to 0.25. Round 2 (vocabulary, RESULTS-vocab.md): the site's 4,567 terms fragment to 3.4 BPE pieces; two epochs of continued pretraining on the 4,760-page crawl take ettin-32m's whole-term recovery from 0.32 to 0.69, whole-term masking to 0.83, 300 added tokens to 0.75 and −5% tokens/page; mean-pooled retrieval stays at chance and classifiers within two points of stock. Round 3 (paper filter, RESULTS-paper.md): "does this paper relate to MSD?" from 1,004 bibliography abstracts vs 2,466 PubMed negatives — every fine-tuned encoder AUC 0.98, 91–95% recall on the 89% of positives whose abstracts never name the platform at 8–14% false positives on topic-matched papers; stock and adapted ettin-32m within noise, ettin-150m no better. The vocabulary lives in methods sections PubMed does not serve; the abstract-level task is topic recognition | RESULTS.md + RESULTS-vocab.md + RESULTS-paper.md + three PLAN*.md + ERRORS.md (16 entries) + vocab/ (4,567-term list, 300 added tokens; adapted checkpoints on branch msd-vocab-weights) + train.py + dapt.py + mlm_ppl.py + retrieval_eval.py + paper_*.py + results/*.json | Oskar: "I'm interested in finetuning to allow for a niche bioscience context classification. Consider for example the public content for mesoscale.com"; "I was looking more for finetuning the vocabulary"; "One of our challenges is: does this paper relate to MSD?" Labels from URL paths and from MSD's own bibliography; corpora fetched by tracked background jobs; questions written by workers from descriptions and titles alone. Twenty-four pre-registered predictions across three rounds, nineteen wrong or half-wrong, all expecting adaptation and vocabulary to show up downstream in proportion to what was learned. Token-level masked-LM loss cannot see a vocabulary gap (one masked piece of IL-6 is predicted from the other two), so the diagnostic masks whole terms. BioClinical ModernBERT (53B biomedical tokens) ties plain ModernBERT-base on the catalog's terms. Ship: regex on the 11% of papers that name the platform, stock fine-tuned ettin-32m on the rest; next, real triage decisions and a full-text arm. |
encoder-platform-survey/ | 2026-09-20 | done — survey plus one measured arm. No open Jev-class model is a base to build on: every clone is under a week old, single-author, and compares its own fine-tune against Jev zero-shot; the one same-benchmark external test puts a 150M encoder at 48% against Jev's 91%. Jev itself is a prefill-only causal decoder (~10B active by black-box probe), Sonnet-accurate on TypeSafe's own evals (67.8% vs Sonnet 5 67.8%, Opus 5 73.1%). Recommendation: fixed-schema classifiers fine-tuned on frontier labels, on Ettin encoders (MIT, beat ModernBERT at every size), int8 ONNX on CPU | RESULTS.md + ERRORS.md + ladder.py + cut_heads.py + ladder_t{1,4}.json | Oskar: "Survey the field and determine a) if there's a solid version to base further work on or b) if we should endeavor to create one of our own and if so c) what platform." Use cases: model routing, compliance gates, first-round paper filters. Surveyed 19 clones (Latent.Space, awesome-jev, 17 Hub checkpoints), the four prior reviews in memory, TypeSafe's evals site and Archer Hume's 10k-call probe. Measured a CPU latency ladder of int8 encoder exports with the masked-LM head cut off (the first run timed the vocab head and inflated mmBERT-small 3x): ettin-17m 6 ms and ettin-32m 12 ms per 128 tokens on 4 vCPU, ModernBERT-base 46 ms, ModernBERT-large 135 ms; at 512 tokens 27 / 63 / 312 / 662 ms. Zero-shot dynamic-schema decisions need the pretraining a 10B decoder has and a 150M encoder does not — the same mechanism hypothetical-classification measured. All three named use cases are fixed-schema and frontier-labellable, which is where 150M encoders score 85–95% in-domain. First experiment proposed: the paper filter, 1,500 Fable-labelled abstracts with a 150-row human check, zero-shot GLiClass and a frozen-embedding probe as floors, recall at 95% precision and coverage at a 5% error budget as the metrics. |
noncommutative-composition/ | 2026-09-17 | done — negative. The commutator K = h(AB) − h(BA) of a meaning-changing composition (milk chocolate / chocolate milk) is 1.2–1.5× the commutator of an order-inert one (milk and chocolate) at mid/late layers of SmolLM2-135M, Qwen2.5-0.5B and Qwen2.5-1.5B, under the registered 1.5× on every model and layer; random unrelated nouns in the same carriers give 1.0–1.3×, and reversing two adjacent nouns in a bag-of-words frame moves the residual as much as reversing a compound. The direct analogue of the SL(n) paper's Flickr ablation (P5) comes out sign-reversed on all three models: the additive model fits the compound better (R² 0.76–0.91) than the conjunction (0.67–0.80). 6 registered predictions per model: P1 WRONG, P4 WRONG, P5 WRONG everywhere; P2 RIGHT; P3 RIGHT but passed by any vector against a 3-d subspace; P6 PARTIAL/WRONG | RESULTS.md + run.py + results/*.json + 6 plots | experiments#97, the Between-the-Spokes follow-up to arXiv:2609.15083 (SL(n) representation learning): does an LLM's residual stream compose knowledge noncommutatively in the paper's sense, generating directions that commutative addition cannot reach, more so where English order carries meaning? Three carriers × three paraphrases × 20 compound pairs + 20 random-noun pairs, residual read at the final : token at every layer, 12 three-word compounds over all 6 orders, 1000-draw paired permutation tests, a real-vs-random interaction test. Every reversal is noncommutative by about the same amount whatever the nouns mean; the compounding-specific excess is a 1.1–1.3× modulation on the Qwen models (p 0.01–0.08) and nothing on SmolLM2, carried by one of the three carriers (I saw a {A} {B}: alone clears 1.5×). Commutators are pair-specific (PC1 13–21%) in every carrier, so P2 says nothing about semantics. Three design flaws surfaced and are reported both ways: the CONJUNCTION basis has no carrier for and (a post-hoc frame-mean term narrows P5 without flipping it), P3's 70% threshold is passed by construction at d ≥ 576, and a one-token read is dominated by the token before it. The 1.5B run was OOM-killed once for holding 3780 × 151,936 log-probs beside the fp32 model. |
rht-operator-native/ | 2026-09-17 | done — positive. A C kernel applying remex's randomized Hadamard plan directly is 2–140x faster than the materialized matrix under sgemm in every measured cell from d=1024 up, on x86, ARM and Apple Silicon, over four CI runs; d=768 wins except batches of 128–256 on x64, where the serial cutoff is too high; d=384 is a split. Its output hashes identically on all four machines where dense output differs on every one. Reproducible codes also need a one-line codebook change: boundaries taken from float64 centroids drift with NumPy's SIMD level | RESULTS.md + ERRORS.md + rht_kernel.c + rhtop.py + ci_run.py + ci/*.json + make_tables.py + recheck.py | From a conversation about arXiv:2609.15083 (SL(n) representation learning) and what remex's rotation could still yield. remex#72 closed a NumPy operator form as won't-do and named a compiled FWHT as the untested lever. Measured on the 1-vCPU container plus three GitHub runners via .github/workflows/rht-operator-native.yml, which commits each machine's JSON back. Nine errors logged, four of which biased the comparison against the operator. Proposed remex changes are filed as remex#86 and not applied. |
deepseek-batch/ | 2026-09-13 | recipe, unrun. Overnight batch coding harness for a local model, shaped by the one number that decides viability: an agent turn is prefill-dominated, so at 30k context 800 t/s prompt processing is ~75 s/turn and ~20 tasks a night while 40 t/s is ~13 min/turn and two. Oskar's M5 Max 128 GB at 2-bit does 17 t/s decode, sitting on antirez's published ds4 ceiling of 16, and ~40 t/s prefill against his ~800 — a 20× gap worth more than any harness design. Both published 2-bit V4.1-Flash numbers are self-reports, and evanwtf/local-llm#321 is open asking for exactly this machine's reproduction | README.md + deepseek-batch.sh + example.task | Oskar, offloading paid-token Claude work to a local model: "I want to be able to schedule batch like work for this model so that it can run during all hours when I'm not working on the computer myself." mini-swe-agent, one process per task, over ds4's OpenAI-compatible server, chosen because its message list only appends — no compaction anywhere in DefaultAgent.run(), so the prompt prefix stays byte-stable and ds4's prefix cache can work. Aider was the first choice and was dropped: last push 2026-05-22, 114 days stale, 1,865 open issues, and its built-in --auto-test loop is the weaker shape anyway per harness-bench (8/12 feeding tests back vs 11/12 letting the agent run them). OpenCode rejected on its own issue #16589 (stalls unattended); OpenHands rejected despite being very much alive, because its context condenser rewrites the middle of the prompt and forces a full re-prefill. The design point is the gate, not the agent: at 2-bit (one write-up puts top-token agreement at 78%) an unsupervised model produces confident garbage at scale, so every task carries a machine-checkable acceptance command, the baseline is checked before the agent runs, and the full suite runs after for collateral damage. Also corrects a 2026-09-10 piece asserting V4.1-Flash cannot run on 128 GB at any quantization — false on the day it published. Nothing measured locally; bash -n is the only check that has passed. |
query-rewrite-lexical-arm/ | 2026-09-10 | done. BM25 alone scores 73.62 HIT@10 on EnterpriseRAG-Bench where arXiv:2609.05637 reports 39.22 for the dense baseline it calls deliberately strong and 55.74 for its all-method oracle. That ordering is not new — the benchmark's own paper (Sun et al., arXiv:2605.05253), which arXiv:2609.05637 cites, already publishes BM25 at 68.4% document recall against 46.0% for text-embedding-3-large; this run replicates it to within 0.34 with a matched analyzer. The finding is the omission: bm25, lexical, sparse, hybrid, keyword and tf-idf appear zero times in the paper. BM25 rescues 63.2% of the dense arm's misses against 9.0% the other way; fusing dense in changes nothing (−0.43, p=0.639) and an S4 rewrite leg on top costs 2.13 (p=0.048), the sign PLAN.md predicted from hybrid-code-index. The rewrite still rescues 12.1% of the hybrid's misses, inside the paper's own 10.1–14.8% band | RESULTS.md + PLAN.md + ERRORS.md + bm25_sparse.py + run_arm_sparse.py + run_arm_dense.py + analyze.py + overlap.py + make_tables.py + recheck.py + results/ | Muninn, from reviewing arXiv:2609.05637 (ServiceNow, EMNLP 2026) at Oskar's request. The paper varies six query-rewriting strategies under one fixed dense pipeline and reads a +12.5 HIT@10 enterprise gain as complementarity between rewrites; no configuration anywhere retrieves lexically. Pre-registered before measurement, including a weakened thesis after the paper's own Appendix Table 7 was found to account for its result without any missing arm. Thirteen arms, 470 questions, 511,962 documents, no LLM judge — expected_doc_ids are benchmark-supplied. Six errors logged, one of them an OOM I caused by running an analysis beside a live arm. |
threejs-lion/ | 2026-09-10 | done | index.html (interactive, one file) + render.png + live page | spike — Oskar: "Using Three.js reproduce this image", a generated stock cartoon lion on black. Procedural: spheres for body and skull, ~3,000 swept-tube hair strands in one buffer per layer, a lathed body under the mane so gaps read as depth, a canvas-gradient environment for the falloff. Verified by headless Chromium renders against the reference at each step. The source image's second tail tuft is a generation artifact and was dropped. Moved here from claude-workspace PR #268. |
toc-path-remax_kb/ | 2026-09-09 | done — negative. Prepending a chunk's markdown heading path to its indexed text does not improve remax_kb: fused Recall@1 0.592 -> 0.586, a difference of −0.0063 [−0.0325, +0.0188] over 799 queries, with every fused contrast at every k straddling zero. The harness is not blind — with the heading path itself as the query the same arm gains +0.2450 [+0.1950, +0.2950] Recall@1. The two modalities move in opposite directions and cancel: dense MRR −0.0200 [−0.0378, −0.0019], BM25 Recall@10 +0.0113 [+0.0013, +0.0225]. Heading-bounded chunking (43% more chunks) is actively worse, fused Recall@3 −0.0300 [−0.0551, −0.0063] | RESULTS.md + PLAN.md + ERRORS.md + build_corpus.py + make_prompts.py + consolidate_queries.py + run_arms.py + rescore_modalities.py + analyze.py + subgroup.py + recheck.py + results.json | Muninn, from assessing arXiv:2609.03874 (STAIR/SearchTome, IBM Research India). The paper reports Recall@1 82.6 against BM25's 59.5 and reads the gap as structure helping retrieval, but every baseline indexes section content while only the proposed system is shown section titles, and no table varies that axis. This runs the cheap version — heading path spliced into the text you already index, no finetuning — on a retriever not built for it. Four arms: production default_chunker, a source-path leakage control, the heading path at byte-identical boundaries (an instrumented chunker reproduces default_chunker exactly on all 95 files, asserted at build), and heading-bounded chunks. 799 queries written by ten subagents shown only chunk bodies, never the headings and never the purpose. Pre-registered before measurement, including that a negative delta was a real expected outcome; PLAN.md called the mechanism and got the signs backwards, predicting BM25 would suffer when dense did. Also closes the title-boosting question lexical-kb/RESULTS.md deferred, and argues for BM25F field weighting over concatenation if it is revisited. Three errors logged, one caught by recheck.py overstating the result before commit. |
spd-hungarian-decoder/ | 2026-09-08 | done — on 1,785 MSLR-WEB10K slates of 50 items, sorting the score matrix's first column ranks as well as solving the assignment problem over the whole matrix (NDCG@10 0.5186 vs 0.5194, +0.0008 [−0.0028, +0.0044]), holding across 3 seeds and 4 Sinkhorn temperatures; the Hungarian solve does beat the row-argmax-plus-repair strategy the paper names as its alternative, +0.0076 [+0.0039, +0.0113]; the trained matrix carries 80.9% of its Frobenius energy in one singular value (96% for a linear-probe head), and on a rank-1 matrix the optimal assignment is provably argsort | RESULTS.md + PLAN.md + prep_data.py + train.py + decoders.py + run_eval.py + controls.py + adversarial.py + adversarial2.py + solver_bench.py + out/ | Oskar, on arXiv:2609.01807 (Laftchiev et al., Meta; v1 hLLM, v2 SPD): "This seems testable and test-worthy". Every ablation in the paper varies the head, the training signal or the backbone; none varies the decoder, so this registers and runs the missing one. Pre-registered readings for both outcomes, plus a positive control on a synthetic matrix with real two-dimensional structure (D1 ahead by 0.04–0.09 NDCG@10 there, so the harness can detect a Hungarian advantage) and a negative control confirming D1 is exactly argsort on rank-1 input, 500/500. The N×K formulation does beat a scalar score plus a sort (+0.0070 [+0.0030, +0.0110] NDCG@10 against the same encoder, same attention, ListMLE) — what it buys is not delivered by the solver. Two side findings: near-rank-1 matrices are LAPJV's bad case (3,072 ms at N=2,000 against 225 ms for random), and the empirical exponent is 2.16, not the O(N³) the paper quotes. Backbone replaced by an MLP over 136 LTR features, so nothing here reaches the paper's prefill-hidden-state or latency claims. |
neomme-remex-quant/ | 2026-09-07 | done — SciFact text + ViDoRe DocVQA and ShiftProject page images; NeoMME ships no quantized vectors; remex 1-bit token codes for its late-interaction head (5.1 KB/doc, 32x smaller) score nDCG@10 0.7070 on SciFact against 0.7198 fp32 and beat the fp32 dense head (4.1 KB/doc, 0.5527) by +0.154 [+0.116, +0.192]; on the dense head remex 4-bit on a truncated Matryoshka dim wins every byte budget from 64 B to 512 B | RESULTS.md + ERRORS.md + neomme_quant.py + encode.py + bench.py + report.py + recheck.py + results.json | Third corpus, ShiftProject (1,000 pages, 100 queries, an easy set at R@10 0.99): the 1-bit token index is +0.016 [−0.006, +0.040] over fp32 at 48 KB/page, pool2 + 1-bit +0.003 at 24 KB, and the dense SciFact ordering returns (4-bit d=256 over 1-bit d=1024 by +0.040 [+0.003, +0.077]); across all three corpora, 1-bit tokens are within noise of fp32 at 32× smaller, 4-bit dense at full width matches fp32, and 1-bit dense at full width loses everywhere. Second corpus, DocVQA (500 OCR-free scanned pages, 2,921 tokens each): the 1-bit token index is −0.004 [−0.015, +0.007] from fp32 at 46.7 KB/page against 1.5 MB, pooling is free there too (pool2 −0.004), so pool2 + 1-bit lands at 23 KB/page, 64× under fp32, for −0.004; the token head beats the dense head by +0.130 on pages. The dense 128–512 B frontier is flat within noise on DocVQA; 1-bit at full width and fp32 at 128 dims both lose about 0.025 on both corpora. Oskar, from Tom Aarsen's Bluesky thread on H Company's NeoMME release: "check if they already ship a quantized vector version, and if not implement a remex/remax style vector quantization for it". Prior-art check first: ten Hcompany/neomme repos, safetensors only, no ONNX/int8/binary siblings, no derivative repos; the native size levers are Matryoshka 128/256/512/1024 on the dense head and HierarchicalTokenPooling on the token head, and both are arms in the bench. neomme_quant.py gives both heads a remex (multi-bit Lloyd-Max, RHT) and remax (1-bit, asymmetric float query) index and scores MeanMaxSim in rotated space, self-tested against sentence_transformers.util.mean_maxsim. Quantizing tokens beats pooling them per byte (remex 2-bit at 16x vs pool2 at 2x: +0.006 [−0.005, +0.016]). The dense→late top-100 pipeline the thread suggests caps at 0.6854 because dense R@100 is 0.864 vs late's 0.943; a full scan over 1-bit tokens beats the all-fp32 pipeline at 3% of its bytes. remex 1-bit and remax k=1 asym are the same code up to the rotation draw, and their ±0.01 gap is the bench's seed floor. Encode 39 min on 4 vCPU (transformers main needed; 5.16.1 has no NeoMME). Text only, one corpus, one seed. |
latent-calculator/ | 2026-09-06 | done — a frozen SmolLM2-135M answers arithmetic at 0.90 exact match with a calculator wired in between layers 16 and 17 and no tokens in either direction (add 0.99, sub 0.98, cmp 1.00, 12-digit mul 0.66); frozen alone 0.001, the result pasted as text 0.000 exact; asking needed a head that attends over the prompt tokens (0.993 query recovery vs 0.10 from one vector) and reading needed the result re-injected at every answer step with digits most-significant first (0.60 → 0.75 → 0.91) | RESULTS.md + PREDICTIONS.md + demo.html + demo.py + make_demo_page.py + data.py + model_utils.py + probe.py + query_head.py + train_port.py + eval.py + run_all.sh + run_phase2.sh + test_latent_calculator.py + results/ | Oskar, the depth-3 port from the in-model-tools design space (memory 88477132): can a tool go between layers, latent in and latent out, for an exact result rather than a retrieval. Frozen Pleias Monad 56.7M and SmolLM2-135M; trained parts 1.3M params (attention query head, step-conditioned result encoder). Three designs: one-vector query + single injection (phase 1, D1/D3 confirmed, D2/D5 refuted, D4 refuted on the held-out length because consumption failed first); attention query head (asking solved in distribution, fails on unseen operand lengths by position-counting); streamed injection, then most-significant-first digit slots (reading 0.91, generalizes to the held-out length at 0.85 with true operands). Residual add beats a KV slot on both models; delayed-by-one injection is free behind a space token and blind behind a word. Monad follows the same curve at 0.50. 29 tests, resumable drivers, ~9 h CPU. |
harness-bench/ | 2026-09-06 | done — on 12 Aider Polyglot exercises at a fixed model (Haiku 4.5), a single-shot edit harness scores 3/12, one retry carrying the test output 8/12, and an agent that runs the hidden suite itself 11/12; 8 recovered, 0 regressed, exact McNemar p=0.0078 | RESULTS.md + PLAN.md + ERRORS.md + recheck.py + harness/ + results/ | Oskar, from dollspace.gay's post that a self-built harness beat the leader on a harness benchmark: "Can you test our own harness à la...". Pre-registered the null and a ceiling stop, because orchestrated-coding-pareto and the PR #76 pilot both went vacuous at 14/14 — polyglot cleared it at 3/12, the first arm comparison here with room to measure. Every task certified both ways (reference PASS, stub FAIL) before any agent ran, which caught a shared CARGO_TARGET_DIR scoring a todo!() stub as 23 passed. Four of the five retry recoveries were a non-compiling import, capitalisation, operand sign order and a reset() that did not reseed. go/zebra-puzzle survives all three arms. Not comparable to the leaderboard's aider+gpt-5.2 0.880: different model, 12 tasks against 225. |
harness-bench/ | 2026-09-06 | done — Opus 5 single-shot scores 182/203 = 0.897 (Wilson 95% CI [0.847, 0.931]) on Aider Polyglot, statistically indistinguishable from the leaderboard's aider+gpt-5.2 0.880 over 225; separately, at a fixed Haiku 4.5 the harness loop moves 3/12 → 8/12 → 11/12, 8 recovered 0 regressed, exact McNemar p=0.0078 | RESULTS.md + PLAN.md + ERRORS.md + recheck.py + harness/ + results/ | Oskar, from dollspace.gay's post that a self-built harness beat the leader on a harness benchmark: "Can you test our own harness à la...", then "the whole point was testing your harness to see how you compare". Pre-registered the null and a ceiling stop, because orchestrated-coding-pareto and the PR #76 pilot both went vacuous at 14/14 — polyglot cleared it at 3/12. Every task certified both ways (reference PASS, stub FAIL) before any agent ran: 203 of 225 admitted, 6 rejected as refactoring exercises whose stub already passes and 16 whose reference will not build here, so the denominator is 203 and the exclusions are not random. Three harness defects found and fixed mid-run, each of which would have measured the prompt rather than the model: a shared CARGO_TARGET_DIR scoring a todo!() stub as 23 passed; prompts rendering only the first of a cpp exercise's two solution files, so the header the test includes was invisible; and regenerating prompts before resetting the work tree, which fed a run its own prior solutions. Per-language: javascript 0.958, python 0.941, cpp 0.909, java 0.897, rust 0.833, go 0.806. |
brief-compression/ | 2026-09-06 | done — negative; telegraphic briefs to Sonnet/Haiku subagents save ~110 brief tokens and cost 1.5k–6.3k receiver tokens, 2–3x tool calls, and 4 of 10 Sonnet runs misread lc "svc_" as case-insensitive; plain prose was the only style with zero failures | RESULTS.md + ERRORS.md + PREDICTIONS.md + recheck.py + gen_b.py / score_b.py + data/ | Oskar, from two screenshots of GPT Astra briefing its Luna sub-agents in telegraphese: would Muninn gain by briefing Sonnet and Haiku the same way? Two file tasks, three brief styles carrying identical information (prose 287 tok, structured 218, telegraphic 178 on Task B; o200k proxy), Haiku 4.5 and Sonnet 5 as Agent-tool subagents, 96 runs. Stripping inter-word spaces, which the screenshots appear to do, saves zero tokens on o200k (53 vs 53). Task A (9 checks) hit ceiling at 33/36. Task B (13 checks, n=10 per cell): prose 10/10 on both receivers; telegraphic 8/10 Haiku, 6/10 Sonnet (pooled 14/20 vs 20/20, Fisher p=0.02); structured 9/10 Haiku, and 2/10 Sonnet on a wording defect in one field of the structured brief (10/10 with it excluded, ERRORS.md #1). Every Sonnet-telegraphic miss is the same miss: !startswith lc "svc_" read as case-insensitive, with rows_kept matching the case-insensitive count to the row. Telegraphic receivers made 4.1 vs 2.0 (Haiku) and 2.9 vs 1.0 (Sonnet) tool calls per run and took 2–2.6x the wall time, re-reading data to resolve abbreviations. The brief's token count is the wrong quantity to optimise; the receiver's total is, and it moved the other way. |
alta-superposition/ | 2026-09-06 | done — on ALTA, every compiled program (looped SUBLEQ, looped parity, feed-forward parity) compresses below its live count with shared directions under a trajectory-SVD continuation, and cliffs at its live count with orthogonal survivors under the code-SVD continuation LAC used; the iteration-vs-feed-forward split does not appear | RESULTS.md + PREDICTIONS.md + alta_common.py + train_code.py + run_all.sh + plot_results.py + test_alta_superposition.py + curves.png | Oskar, testing whether LAC's no-sharing result (llm-as-computer experiments/superposition/RESULTS-A.md) is about iteration, on a second compiler: ALTA (Shaw et al. 2024) with its weights frozen and only a residual code trained by the LAC hinge objective and continuation. Trajectory-SVD arm: subleq 95 of 121 live (interference 0.47), parity_seq 17 of 31 (0.46), parity_ff 5 of 7 (0.38). Code-SVD arm: 121, 31, 7 with zero off-diagonal, the LAC pattern on all three. B1 refuted, B2 confirmed, B3 refuted under trajectory SVD; the LAC learned-code run should be repeated with the trajectory rule. jax + tensorflow-cpu, 18 tests, ruff clean; per-width checkpoints, resumable driver. |
rasp-numeric-select/ | 2026-09-05 | done — a numeric select_at for RASP: one attention head of head_dim 2 attends to a computed integer address through the parabolic key (2j, -j^2), winner/runner-up gap exactly 1, and the compiled weights carry no compile-time length bound | RESULTS.md + rasp_ns.py + compile_ns.py + programs.py + margin.py + test_rasp_ns.py + gap_vs_beta.png | Oskar, extending the LAC line (oaustegard/llm-as-computer) after torchlean-lac/: RASP/Tracr, ALTA and B-RASP[pos] all reach a position through a categorical predicate over one-hot keys, so a computed address costs one residual dimension per reachable position. select_at(addr) reaches it by key geometry instead. Three programs a categorical compiler cannot express without a bounded position table — y[i] = x[2i+1], y[i] = x[x[i]], y[i] = x[i - x[0]] — compile to 13, 11 and 19 residual dimensions and match the interpreter exactly at n = 3, 8, 32, 128, 257 and 1000, having never been shown a length; the same gather through a categorical position table costs 55 dimensions at n_max = 8 and 391 at n_max = 64 and is wrong above its n_max. Softmax attention matches average-hard attention within 0.5 from beta 3.5 (gather), 3.0–6.25 (chase, rising with n because the read values do) and ~9 (shift, whose range gate costs about 6 extra units of beta). numpy only, 34 tests, ruff clean. |
torchlean-lac/ | 2026-09-05 | done — LAC's parabolic addressing proved in TorchLean's spec layer, and its float32 capacity ceiling (exact through address 4096, fails from 4097) proved over TorchLean's FP32 arithmetic; the hard-argmax read cannot reach the graph IR | RESULTS.md + LAC/Core.lean + LAC/Check.lean + LAC/Capacity.lean + capacity_numerics.py + recheck.py + ERRORS.md | Oskar, after the TorchLean survey: "Can we describe/craft LAC in TorchLean?" Five lemmas about score j i = 2*j*i - j*j elaborate against Mathlib, including score_gap (adjacent integer addresses differ by at least 1 in score); paraKey/paraQuery/Mem C/rowScores elaborate against a fully built NN (4352 jobs, clean). #print axioms gives the three standard Lean axioms. Blocked: NN/IR/Graph.lean's OpKind has no argmax, gather, or one-hot, so no autograd and no IBP/CROWN on hard-argmax LAC; and Spec.Tensor.argmax has no characterization lemma anywhere in NN/ (the three argmax theorems in the tree are all about argmaxClassifier), so read-exactness needs the loop invariant proved first. The binary32 capacity ceiling (j^2 > 2^24) is proved in LAC/Capacity.lean against FP32 = NF binaryRadix fexp32 rnd32: read_exact_below_ceiling (every intermediate of the float32 score is exact for j ≤ i ≤ 4096) and read_ties_above_ceiling (addresses i-1 and i score identically for 4097 ≤ i ≤ 5792, the first binade), plus not_both_representable_above_ceiling for every binade and four bit-level IEEE32Exec instances closed by kernel decide. numpy float32 confirms the pipeline fails on exactly 4097–5793 and then only sporadically. |
eml-prove2me/ | 2026-09-05 | done — 11 EML witness theorems verified on Prove2Me; lower bounds posted open; mission proposal drafted | README.md + Definitions/ + Theorems/ + Solutions/ + mission_description.md + scripts/ + published.json | Oskar, after the Anthropic FLT formalization post: "Try the EML table as a Prove2Me mission. Sign yourself (or me if needed) up for an account." Account muninn registered under his email; EmlComplexity definition (Tree, size, real eval, valid, Attains, Complexity) published; the eleven real-branch witnesses from eml-sr benchmarks/eml_complexity.md (e:1, e-1:2, 0:3, e-2:7, -1:8, 2:9, ln2:12, 3:14, 1/2:17, -3:20, 4:21) proved by exhibiting the enumeration's tree and rewriting with log_exp/exp_log under pre-proved positivity facts, all ACCEPTED by the server's Lean (v4.33.1, Mathlib 0df444a); complexity_two, not_attains_four_below_twenty_one, complexity_four posted as open theorems; proposal 9a4eba3f with 13 milestones awaits the human confirm-and-submit step. Lean tactic lessons in the README. |
static-code-embed/ | 2026-09-03 | done — no cheap adaptation of a static table reaches rg, let alone bekko; fusing one with rg lowers rg | RESULTS.md + scripts/ + results.json + data/vocab_*.json | Oskar, on the zvec-grep evaluation: "isn't part of the attraction ease of fine tuning?" then "Run it!". Same n=59 sklearn harness as bekko-embedding-bench; rg reproduces 0.596/0.682 exactly and bekko-a25m re-encodes to 0.651/0.706. Sibling of potion-code-quant/ (same day, separate session; that one covers quantization and index compatibility, this one adaptation) — its vanilla potion r@5 0.436 is reproduced here independently. Five adaptations of potion-code-16M-v2, all paired against vanilla: MNRL on 4,223 docstring→code pairs (val loss 1.64→1.14, r@5 0.435, Δ −0.001); +12,276 corpus identifiers as whole-word rows, sum-initialised so epoch 0 equals vanilla (0.439), then MNRL (0.418 / r@10 0.563, n.s.); the same with mean-initialised rows 0.334, p=0.021 worse — one row replaces k pieces and long identifiers lose k-fold pooling weight; a hand-rolled Model2Vec distillation of bekko-a25m (23.6k tokens, PCA 92%, SIF) 0.249, +MNRL 0.350; model2vec.distill.distill() of the same teacher (268,850 rows) 0.052, +MNRL 0.135. bekko-a25m vs potion +0.214 r@5, 26/2, p<0.001; every static arm vs rg p≤0.004; potion RRF with rg 0.537 vs rg 0.596. Static encode 6 s vs bekko 20.6 min for 11,439 chunks. Caveats: identifier-poor stratum n=1; one learning rate for all arms; ast chunking only. |
potion-code-quant/ | 2026-09-03 | done — potion trails bekko-a8m by 0.16 r@5; no static table is a drop-in query encoder for a transformer-built index | RESULTS.md + run.py + distill.py + fit_table.py + specter2_fit.py + check_centered.py + results_*.json | Oskar: test the Model2Vec static code model (minishlab/potion-code-16M-v2, a 63k x 256 fp16 token table, mean-pool) against our embedders and see if its output quantizes; then, can a static table produce embeddings compatible with an index a transformer built, including a remax/remex-quantized SPECTER2 set. On bekko-embedding-bench's n=59 sklearn file-discovery task: potion fp16 r@5 0.436 vs bekko-a8m 0.595 (Δ −0.159, p<0.001); bekko's 96 B remex 2-bit sidecar (0.590) beats potion's uncompressed 512 B vector. potion encodes 80x faster (1.3 vs 104 ms/query) and its own output quantizes about like bekko's: remex 2-bit −0.015 (n.s.), remax 1-bit −0.075 (p=0.10); Model2Vec's native int8 table is free and its dimensionality= truncation equals a post-hoc slice. Token-level Model2Vec distillation of bekko (pca_dims=None) is not space-compatible: cosine 0.26 to the teacher, a 0.957 common component across all student vectors, r@5 0.017 querying bekko's index. Fitting the table by ridge regression to the teacher's sentence vectors gets cosine 0.76 on held-out code (0.60 on NL queries) and r@5 0.36 against bekko's index (teacher 0.60; remex 2-bit index 0.39, remax 1-bit 0.24), 9 MB table. SPECTER2 (remax bench 10k set, seed 99): the fitted 26 MB table reads cosine 0.96 raw but 0.71 after centering (SPECTER2's shared mean gives raw pairwise cosine 0.85), sign agreement 0.75 after the index's rotation, and r@10 0.288 against the teacher's remax 1-bit index where the teacher gets 0.645; float index 0.264, remex 2-bit 0.117. The quantizer on the index side is not the limiting factor; the student/teacher residual is. |
embedding-inversion/ | 2026-09-01 | done — a vec2text-shaped inverter on bekko-a8m behaves as the paper describes (verifier selection, then one large correction round, then convergence) and at 40k pairs / t5-small / 4 vCPU recovers the exact string 2.4% of the time from the float vector and 0.9% from the 384-bit sign code; not retrieval, not an inverter either, a paraphraser that lands on topic | RESULTS.md + ERRORS.md + recheck.py + results_float.json / results_bin1.json | Oskar: "How would you craft a reverse embedding model?", then "implement that PoC". Pre-registered four predictions before the first stage ran; three held, and the one that decides whether this is an inverter (beat nearest-training-string by 20 points of exact match) failed in both arms. Zero-step base scores cosine 0.55 on its own training set and 0.547 on dev, so the shortfall is underfitting: vec2text's base starts near 0.9 after 5M pairs, this one at 0.55 after 40k. The arm trained on the sign code loses to the float-trained arm even at matching the code's own bits (69.2% vs 71.8%). Exact match is zero past 10 words in both arms. Also recorded: a container restart kills a nohup driver about a minute after the turn ends, and a harness-tracked Monitor keeps the container up between turns; a Monitor with persistent: true still dies at 30 min. |
ms13-k4/ | 2026-09-01 | done — theorem: R_max(4) = 4/5 exactly (Conjecture 12.2 and Q7′ at k=4) | RESULTS.md + bbk.py + splits2.py | Reopens ms13-campaign's open question at the k it called a 2,070-hour no-go. The tree census is unnecessary: maximal row-set types are the split systems of binary trees on 2k leaves with leaves paired into chords (Buneman), 4 shapes × 105 pairings at k=4, 14 maximal types, reproduces the campaign's k=3 census exactly. Fail-first branching in the campaign's own exact rational B&B (next rounding = fewest live (row, side) options at the parent LP optimum) proves R ≤ 4/5 per type in 3–64 s where lexicographic order had not finished type 0 after 900k LPs. All 14 maximal types have R = 4/5 exactly, each attained at unit demands; unequal demands never beat Doerr at k=4. Lower bounds re-verified by a code-disjoint evaluator; two-sided calibration at v=3/4 finds margin-1/20 witnesses on all 14. Hand lemma: the spider type is ≤ k/(k+1) for all demands. Open: k ≥ 5, a proof (the gap is named: weighted rows lose the two-valued error structure that Hoffman–Kruskal + Carathéodory needs). |
bts-coordinates/ | 2026-09-01 | negative (mechanism), headline withdrawn on its own null | RESULTS.md + PLAN.md | Transplant of the growing-feature-set mechanism in Large Discovery Models (arXiv:2608.15669 §4.2) onto the between-the-spokes cross-field prior-art problem, against the SLT objection in memory a8b97f70 that novelty on a new representational direction is unreachable by flat embedding search. Retro-eval targets from the ms13 campaign (Doerr 2004, cross-field; MSW25, same-field) over arXiv-built pools of 1101 and 1241 titles, one encoder across all arms so only the basis changes. The growing-coordinate arm does not beat its own frozen ablation (one win each over four informative comparisons, sign test p = 1.0; both arms also beaten by a static ranking that costs zero reads). The first pass's headline — a blind-named axis ranking the target 5th of 1101 — was destroyed by the null control the pre-registered adversarial pass demanded: 12 random pool titles used as axes give a median best-of-12 rank of 10, so best-of-k is an order statistic. Surviving effect is narrow: mean-over-named-probes beats both the raw query and random probes in 5/5 configurations, i.e. multi-probe query expansion, which METHODS.md records losing three times before. Also found: paraphrasing only the target's title moves its rank by up to 383 places, larger than every measured effect; a fabricated negative control outscores the real target (0.7719 vs 0.7295), so absolute similarity says nothing about whether an answer exists; and issue #179's own P1 test case leaks the answer's vocabulary, so a stripped variant was written. Upstream of all of it, a subagent with zero tool calls, given the problem with all cross-field vocabulary removed, returned the target's verbatim title as its top query in under two minutes. Infra: arXiv keyword search works again (it killed PR #180 in July) and HF weights are reachable from CCotw. |
hypothetical-classification/ | 2026-08-31 | done — the pattern beats every model-free baseline (0.564 vs 0.417 acc@1 on WANDS) and loses to shipping the vocabulary by 14 points (0.701); the prompt must anchor on the vocabulary's REGISTER, never on novelty, or a Haiku subagent that obeys scores a fifth of the no-model control. Shipped: muninn-utilities#127, claude-skills#782 | RESULTS.md + ERRORS.md + recheck.py | Oskar: "Don't classify. Hallucinate!" (softwaredoug.com, 2026-08-10) — "it just takes a class corpus, a cheap model and a cheap embedder". Measured on WANDS (860 labels, 468 queries, one gold label) and on Muninn's own tag vocabulary (1,273 tags, 250 memories). The source post's novelty-anchored prompt is the single largest variable and is wrong: gemini-3.5-flash-lite half-ignores "novel, never-seen-before" and writes Salon & Styling Chairs, while a Haiku 4.5 subagent obeys it, writes Hydraulic Styling Thrones, and scores 0.100 acc@1 against a 0.500 no-model control. Register-anchored, the same subagent takes the best acc@3 of any arm (0.750). A boundary claim — that the pattern halves against direct embedding on long documents — was published into both downstream PRs from the novelty prompt and then withdrawn: under the register prompt that arm goes 0.200 → 0.500, past the control, and the union of both rankings reaches 0.676/0.848/0.876. Also measured: batching 40 items per call is free at 1/17 the input tokens, char-ngram TF-IDF snaps nearly as well as MiniLM (0.528 vs 0.564) and beats it outright where documents contain their own label words, and a general-purpose Haiku subagent spends 32,539 tokens to output the word ok. On the in-browser question: Pleias Monad (57M) and Baguettotron (321M) package fine (35 MB and 236 MB at q4f16, onnx-community builds) and earn nothing — 0.425/0.400 acc@1 as label writers and 0.325/0.350 as likelihood rerankers, against a 0.500 no-model control, with the gold label in the encoder's top-10 for 82.5% of queries. What the cheap model contributes is a prior over how taxonomies name things, which is the first thing a shrinking model loses. A fully client-side classifier is real and it is the encoder alone: gte-small int8 is 33 MB and scores 0.455 acc@1 / 0.594 acc@3 with no API call. |
hyde-recall/ | 2026-08-31 | negative — HyDE query expansion over Muninn's FTS5 corpus is a wash at matched depth (0.263 vs 0.250 R@10, n=80); plain recall(n=40) scores 0.525 for free and dominates every arm | RESULTS.md + ERRORS.md | Built on a misreading of the request above (HyDE, arXiv 2212.10496, rather than hypothetical classification) and kept because it closes a question METHODS.md records as open. HyDE's filter is a dense encoder's lossy bottleneck; BM25 has none, so a hallucinated term either matches nothing or drags in an off-topic document. Generating without corpus exemplars scores below the baseline (0.175 vs 0.250) — the same register failure the sibling experiment measures at scale. The confidence gate nl2sh-dense flags as unmeasured is now measured: term-coverage gating reaches 0.300 against a 0.338 oracle ceiling and still loses 5 queries. Third consecutive loss for query expansion on this account's corpora, after nl2sh-dense and muninn-rm3. The real finding is that Muninn's default recall depth of 10 leaves half the reachable targets unretrieved at zero cost. |
halo-ccotw/ | 2026-08-24 | done — HALO's engine, WASM sandbox and trace tooling all run in a CCotw container; the RLM needs an OPENAI_API_KEY the environment does not carry, and the multi-session corpus it wants does not exist because release writes are 403 for this session type | RESULTS.md + cc_to_halo.py + validate_dataset.py | Oskar: "Assess the viability of a solution like this in the Claude Code on the web environment" — context-labs/halo. Measured rather than reasoned about: pip install halo-engine is clean, the bundled deno 2.7.14 and ripgrep 15.1.0 wheels run, and the Deno+Pyodide sandbox boots numpy/pandas/pydantic in 7.5s cold (24 MB of deno cache, ~7s per invocation since each run is a fresh subprocess). POST to OpenAI and OpenRouter /v1/chat/completions returns the provider's own 401, not a proxy 403, so a key supplied through the environment is all the engine needs; there is no way to route its calls through the harness, since the Agent tool is a model tool call rather than an HTTP endpoint a subprocess can reach. cc_to_halo.py converts Claude Code transcripts to OpenInference spans — one 11-minute session became 161 spans with all three of the index's health counters at zero, and view_trace correctly reports it oversized at 391 KB against a 150 KB budget. Two conversion traps: durations are derived (each record has one timestamp, so LLM-span duration is an upper bound while tool spans are true wall-clock), and prompt tokens are three Anthropic fields summed — counting only input_tokens reports 106 for a session that processed 13.7M. The corpus is the real blocker: persist-transcript.sh has archived nothing since 2026-04-05, because /tmp/.workstation-booted is gone and, underneath that, the proxy returns 403 on every release write and Contents-API PUT for this session type. claude-workspace#246 fixes the discovery and makes the 403 audible. |
coherence-remex/ | 2026-08-24 | done — one mechanism in daniloc/coherence is worth having (the meta-oracle, which reads a test's own AST and refuses a via test claim whose test loops a hand-written list); the surrounding apparatus is not. Adopted into remex as a live trial: remex#80 | RESULTS.md + ERRORS.md + recheck.py + artifacts/ | Oskar: "Read this full thread" (daniloc.xyz, 2026-08-22 — models write fine code at file level; the failure is complexity across files, swamped by volume no human reads; cites Conant & Ashby on regulators), then "attempt implementation against one of our repos to assess it with a live use case. Maybe remex?". Tool ground-truthed rather than read: builds clean on Node 22, 936/936 tests pass, verify on its own repo is 83 claims all green over a 643-decision dogfood ledger — but 66 of 66 of its own boundary claims use via guard, the escape hatch that skips the meta-oracle, which its own Known Limits calls "a laundering channel for hand-lists dressed as guards". Zero via test, zero parity, zero conforms to in its own specs, so the flagship check is unit-tested and never exercised through its own enforcement path. That made a trial on foreign code the only assessment available. On remex it ran on Python out of the box (shipped tree-sitter grammar, no adapter) and immediately flagged test_pq_and_npz_round_trip_rotation — @pytest.mark.parametrize("rotation", ["haar", "rht"]) against a three-member ROTATION_CODES — as "iterates a LITERAL domain… a sampling oracle, not totality". Four perturbations, each reverted: "hadamard2": 3 into ROTATION_CODES with no construction, and "hadamard2" into Quantizer.ROTATIONS alone, both leave the entire 267-test suite green while only the live-domain oracle goes red by name; the other two are co-detected and labelled as the weaker refutations they are. Adoption cost is a refactor, not a spec edit: the second literal domain had no registry to loop — the packable widths were spelled five times as bits in (5, 6, 7) across three modules plus twice as [1, 2, 3, 4, 8] in tests — so closing the claim meant extracting SUPPORTED_BITS in remex itself. Also found wrong: the parity arm false-fails a correct oracle that aliases its domain (for name in sorted(persistable) refused; for name in ROTATION_CODES accepted), against a README that claims the analyzer never false-fails; and redundancy found two disagreeing README benchmark tables while missing the five-site source duplication the claim machinery had just caught. Ends at 7 claims/7 green/3 anchored invariants each with a recorded refutation, 288 Python tests passing. Verdict: transplant the LIVE/LITERAL/NO-ITERATION classification into verifying-claims; leave the 40+ subcommands, the 217 KB README (~54k tokens it tells agents to read) and the doctrine/gyroscope/premise-lease vocabulary. |
hyparam-survey/ | 2026-08-24 | done — hypvector is remax_kb's architecture in a Parquet container, and the one thing it has that we do not is a build-time IVF whose cells are physically contiguous on disk, so a reader range-fetches 28 of 112 clusters instead of downloading the index | README.md + NOTES.md + ERRORS.md + results.json + evidence/ (nine cited source passages, re-derivable from the sha256-pinned npm tarball) + recheck.py | Oskar: "This org's work is seriously impressive! Spelunk and pick up tooling/references that may aid us in our work. The vector search on parquet part is maybe something to consider for remax-kb?". Surveyed 27 public repos; the vector library is npm-only (hypvector@0.2.2 ships unminified src/, the GitHub repo is private). Ran the sweep independently on 50k synthetic 384-dim vectors: their constants.js claim that residual misses are a rerankFactor limit and not a probe limit reproduces — scanning all 112 clusters instead of 28 moved recall@10 from 50% to 49% — and their default rerankFactor: 10 reaches only 50% on this corpus, 93% at 100. remex.IVFCoarseIndex already partitions a corpus but is deliberately data-oblivious and keeps an 8-byte-per-vector permutation; hypvector reorders rows physically and stores per-cluster counts instead. |
model-register-drift/ | 2026-08-23 | done — Opus 5 is third-cleanest of six models on declaude_lint.py and roughly twice the next model's rate once the register entries regex cannot reach are counted by hand | RESULTS.md | Oskar: "people are not all that happy with your Opus 5 substrate's attitude and personality... I find its idiolect insufferable". Six bare CCR sessions (create_session, explicit model id, no source_url, so no repo, no boot, no identity) wrote the same 700–900 word post with no voice instruction from anyone. The Agent tool could not run this — its model parameter takes four aliases and cannot address 4.8 or 4.6. Scored twice: declaude_lint.py normalised per 1000 words, then every one of the 42 entries by hand. The two rankings invert. Opus 5 sits at 5.04 tics/1k mechanically (3rd of 6) and 25.1 violations/1k adjudicated (1st, next is 16.4), with 85% of them in the aphorism/verdict family — entries 3, 7, 12, 13, 37, 38, 39, which SKILL.md already names as the third its regexes miss. Four of six headers are verdicts; six paragraphs end on a quotable line. Within the Opus line the adjudicated rate rises 11.4 → 15.3 → 25.1; the Sonnet line does not (16.4 → 13.4). n=1 per model, one prompt, one judge — only the Opus 5 gap is wider than adjudicator noise. |
nl2sh-cli/ | 2026-08-23 | done — the shippable half of the nl2sh line as a CLI: hybrid search over shell documentation with no model at all, plus an optional generator through ten backends | README.md + nl2sh/ (search.py, backends.py, cli.py, config.py, vendor/) | Oskar: "let's create a product that optionally invokes a model, either running locally via ollama/llm.cpp/mlx etc, or remotely. If no model is chosen the user just gets the hybrid search". backend='none' is the default and the product — ranked documentation, no weights, no network, no key — rather than a degraded mode. Local: ollama, llamacpp, llama-server, LM Studio, mlx, transformers. Remote: any OpenAI-compatible endpoint, Anthropic, Gemini. Remote costs no dependency because all of it is HTTP and urllib speaks it, so the extras cover only in-process runtimes. Every probe returns Availability(ok, detail) with an actionable reason instead of a bare False, the same rule compose_layers._tag_exists was fixed for. Defaults come from the measurements: instantiate_anchored because it wins every column at 1B, with a warning rather than a silent switch when a model's name looks small, since that prompt collapses to 0.146 at 270M. prompts.py and extract_params.py are vendored with pinned hashes because every quoted number came from those exact bytes. Search reaches 0.506 and not the research's 0.555: the query adapter that separates them gained +0.184 on the 207 utilities it trained on and lost 0.039 on unseen ones, so it stays out. Nothing here executes a command; it prints one. |
caps-emphasis/ | 2026-08-23 | done — capitalising a directive does nothing on Baguettotron: mid-sentence CAPS is +0.003 log-odds [-0.147,+0.168], and violation rate is 42/43 in every surface form including no directive at all; where CAPS does move the number it tracks the extra tokens capitalising costs, not the case (+0 token bin CI spans zero). Markdown bold looked 20x stronger, but most of that was entropy and the rest reverses sign when the bold is moved into the reasoning register, where SYNTH actually puts it. Corpus reason: SYNTH holds one genuine capitalised directive in 22,100 documents; bold outnumbers emphatic caps 23:1. Largest effect measured was not typographic — restating the constraint in the reasoning register cuts the ironic rebound by ~0.8 log-odds | RESULTS.md + Q5_CORPUS.md + METHOD.md + PRIOR_ART.md + ERRORS.md | #45 — does writing part of a prompt in capitals change what a model does, and by what mechanism |
tc-interference-weights/ | 2026-08-22 | done — review: three checkable problems in Anthropic's interference-weights note, all re-derived from its own figure data | RESULTS.md + check_claims.py + mirror.sh | Oskar: "get this in its entirety, including the charts, then review" — Turner, Wu & Batson, Characterizing interference weights in a tiny language model, Transformer Circuits Thread, 21 Aug 2026. A 1L transformer plus a transcoder, expanded into 331M virtual weights, scored by Fisher effectiveness (2nd-order KL, all 331M weights over 537M tokens) and helpfulness (mean loss change under ablation, 7,765 sampled weights over 1B tokens). Its genuinely new contribution stands: " IN "→" utions " is the first interference weight demonstrated inside a trained transformer with a loss measurement attached — largest raw virtual weight from that token, target never once follows source in training, ~3 OOM below top effectiveness, negative helpfulness. Three problems, each reproducible from shared/data/figures/*.js via check_claims.py. (1) The stated pruning exception names the wrong family. Text: Fisher beats raw |w| "for every density and individual weight family (except for negative Features→Logits weights)". threshold_data.js says Fisher wins that family at all 19 densities by 3–10×; the family where it actually loses is Tokens→OV→Logits negative-only, worse at 8/19 densities and up to 2.22× across density 0.05–0.40. (2) The main conclusion does not survive the paper's own effect-size appendix. The Discussion rests "the model is still dense in this basis" and "no saliency scheme will perform much better" on 47.6% of weights having positive mean helpfulness. Its ROPE appendix, population-weighted over all 331M: at ε→0, 15.1% positive / 64.3% uncertain; at the paper's own yardstick ε = budget/N_total = 1.5e-8 nats/token, 90.1% practically zero and 2.9% positive — ~9.6M weights, roughly 3× the transformer's parameter count, landing next to its own helpfulness-mass estimate (2.43% density holds 90% of positive mass). Three routes converge on 2–3%; only the sign-of-the-mean route reaches the Discussion, which never cites ROPE. (3) Helpfulness and Fisher effectiveness are the same quantity on the tail. Expanding the paper's own formula to second order gives helpfulness(w) = −w·∂L/∂w + fisher(w) + O((sw)³) — the Optimal Brain Damage saliency it cites in related work but never connects to its headline. fisher ≥ 0 always and grows as w² while the gradient term grows as w, so weights above a crossover in |sw| are forced to measure as helpful. Measured on the helpful branch: log-log slope 0.89–1.06, r 0.956–0.990, median h/fisher 0.58–1.22 in all six families over 6+ OOM. So "the model puts its most effective weights in helpful directions" is partly an identity between the two metrics, not solely evidence about training pressure — and it inverts the claim that "a sharper metric has little room to improve", since the first-order term the two metrics don't share is exactly what separates helpful from harmful. Checked and clean: the 7,765 sample is family-stratified (~1,210–1,379 each) while families range 16.8M–104.9M weights; population-weighting moves h>0 only 47.6% → 48.4%, so the headline is not biased by that. Pruning numbers verified (70% pruned → +0.0107 nats, 85% → +0.0702). Minor: the dead fraction of one sample is reported three ways — 12.7% (table), 13.1% (histogram), 13.07% (implied by the mass figure's counts). Artifact: mirror.sh reproduces a 107 MB offline mirror — main page, 21 interactive figures with data and vendor bundles, the 34-page figure gallery, the 6.2 MB feature_vis page. Two assets 403 on the live site (shared/data/token_splits.js, shared/data/token_vocab_split.js, Feature 157 figure), broken upstream. |
avo-supervisor-specter2/ | 2026-08-22 | done — best bits=4/haar/seed=2026/two-stage(300) R@10=0.7630 vs bits=4/rht/seed=42 baseline 0.7510 (+0.012), 30 candidates, 3 strategy classes, 2/3 plateau switches; the Stop-hook loop itself did not run — this session is a child session with no supervisor hook wired, so candidates ran via a manual driver applying the same plateau rule instead | RESULTS.md + run_candidates.py + ledger.json | claude-workspace#233 — first real search run off the supervisor infrastructure assessed there |
avo-supervisor-assessment/ | 2026-08-22 | done — go: exit-2 Stop hook blocks and injects verbatim (tested live), hook wall clock 38 ms, single-candidate remex fitness 1.4–1.7 s, in-repo JSON ledger beats Turso 0.2 ms vs 0.5–0.7 s; Fable 5 session with create_session(model=…) makes the three-arm ablation spawnable | RESULTS.md + artifacts/ | claude-workspace#233 — assess AVO-style supervisor loop before building |
nl2sh-instantiate/ | 2026-08-21 | done — the bake-off in MODELS.md revises this headline: a zero-shot Gemma 3 1B beats the fine-tuned 270M on every column (routing 0.799 vs 0.610, usable 0.793 vs 0.470), and the instantiation framing WINS at 1B (0.848 vs 0.799) once the model stops imitating the source-line format. The 270M finding below stands, scoped to 270M: framing the task as substitution does not move routing on a 270M model (p=0.76) but collapses token-repeat loops 0.183 → 0.049, and execution scoring puts the real functional accuracy at 0.055 against a 0.427 routing headline | RESULTS.md + prompts.py + run_gen.py + train.py + score.py + funceq_ext.py + funceq_alfa.py + alfa_prep.py + bench.py | Issue #52, stage 2: retrieval finished at 0.555 gold-in-sources against an 0.640 oracle ceiling, so the loss is the generator's. The issue argues the job is instantiation — the user's literals substituted into a documented example — because §6 of nl2sh-dense measured an exemplar worth +0.189 routing while the choice of exemplar was worth zero. Oskar: "let's start with the 270 million Gemma three". Model, sources, distractors, decode and seed held fixed; only the instruction varies. Zero-shot the substitution prompt loses 0.146 to 0.500, and the loss is a format artifact: on 0.774 of rows the model answers in the shape of the source lines it was shown, bullet included. One epoch erases that entirely (0.774 → 0.000), the same shape as stage 1's 0.026 → 0.706. Past training the prompts route the same — 23 wins to 20, p = 0.76 — and separate on garbage: loops fall 0.183 → 0.049 and usable gains +0.092 (31–16, p = 0.040). nl2sh-selfhist/MODELS.md had named degeneracy the real ceiling after repetition_penalty=1.3 bought a similar reduction at a cost of 0.118 routing; this buys it free. The published benchmark reproduces the direction on rows built to execute (westenfelder/NL2SH-ALFA, MIT, arXiv:2502.06858): 0.854 → 0.866 routing, 0.144 → 0.085 degeneracy on the non-find slice — after subtracting its 0.393 always-find prior, which eats most of the 0.911 headline. And the first functional number for this line of work is 0.055: with the funceq fixture built from every path the gold commands name, 36 of 164 cyber rows are decidable and 0.250 of those are equivalent — utility_ok overstates by ~8x, corroborating the issue's hand-read estimate of "nearer 0.05". The cyber corpus caps at 0.22 coverage no matter how wide the fixture; ALFA is where execution decides. Recovered from a lost container: the session that ran the grid wedged on a run_in_background poll loop and was reclaimed with everything uncommitted — scripts and tables came back from its transcript and re-ran to identical numbers. |
gemma-proxy-tuning/ | 2026-08-20 | open — vocabulary precondition holds, logit scale does not; experiment not run | RESULTS.md + tokenizer_diff.py + tokenizer_diff.json | Oskar, after establishing that speculative decoding cannot merge two models: "how similar are Gemma 3 and Gemma 4? Could we fine-tune the smallest Gemma 3 and apply it to the Gemma 4 ~30B?" Proxy-tuning (Liu et al., COLM 2024) and emulated fine-tuning (Mitchell et al.) are the same equation, softmax[s_base + alpha*(s_expert - s_antiexpert)], and both need the three logit vectors to share an index and a scale. Index: yes. Both tokenizers are 262,144-piece BPE with a byte-identical 514,906-rule merge list; 255,938 ids (97.63%) carry the same token and zero ordinary text tokens moved — all 6,187 that did are special or reserved. The disagreeing ids form exactly two contiguous ranges, [46,106] (61 <unusedNN> shifted by 7) and [255999,262143] (6,145 Gemma 3 image/turn tokens replaced by Gemma 4 `< |
needle-depth-growth/ | 2026-08-20 | done — the surgery is 30 lines and lossless; the training is pretraining, not a fine-tune | README.md + grow.py | Oskar, reading the Cactus paper next to needle-bsky: "can a model like needle be retro trained to add MORE layers, à la Monad?". Mechanically yes, and verified. Needle 2's stack is an nn.scan, so every per-layer tensor carries a leading axis of num_layers — block weights inside the scanned collection, MHC lane parameters as explicit (L, ...) arrays — and nothing downstream hardcodes 27 (decode.py sizes the KV cache from cfg.num_layers, export.py loops over it). Growing is a concat along axis 0 plus a config bump. grow.py does it with identity-initialised blocks and the grown model's logits are byte-identical (`max |
needle-tool-naming/ | 2026-08-20 | done — negative: tool names and tool descriptions are two near-equal channels, and better names buy nothing | RESULTS.md + PREREG.md + ERRORS.md + recheck.py + results_*.json | Oskar, after reading Cactus's attention-only paper (which is Needle 2's architecture — the shipped class is literally SimpleAttentionNetwork): "is it worth further experimentation?". needle-bsky left profile at 0.250 and identity at 0.333 in every arm, unmoved by a schema rewrite and a fine-tune — even though the discriminating fact was already in the description verbatim (get_profile says "follower count"; the query "how many followers does pfrazee.com have" still returns get_followers at 0.80). That is the opposite of the failure the paper predicts, which localizes the SAN deficit to low-context tokens. Hypothesis: names outrank descriptions. Six variants over the same 18 tools, same 62 queries, same scoring code, predictions committed before the first run. The hypothesis failed: names-only 0.444 vs desc-only 0.407, predicted gap ≥0.15, measured 0.037 at p=0.82. What replaced it is cleaner — the two channels are near-equal and partly redundant: deleting descriptions costs 16.7pp (p=0.035), deleting names costs 20.4pp (p=0.019), deleting both leaves 0.074 against a 0.056 chance floor, and against that floor names are worth 0.370, descriptions 0.333, both together 0.537 — a quarter of each carried by the other. Rule-written names buy exactly nothing (separated = canon = 0.611 flat and 0.778 oracle, p=1.00 both) and cost the gate: confidence separation collapses 0.191 → 0.101 because mean confidence on wrong calls rises 0.392 → 0.480. Name capture is nonetheless real and visible where it acts — mechanically rotating names onto neighbours moves profile 0.250 → 0.750, with "how many followers does pfrazee.com have" → get_profile @ 0.81 and "look up the account jay.bsky.team" → get_profile @ 0.90, descriptions unchanged by one byte — it is just too small (4 queries of 54) to surface as a main effect. Mirror case in the same arm: resolve_identity wearing the name check_network_outage_status is still chosen correctly on its description alone, at confidence 0.584 → 0.167 — conflict keeps the answer and destroys the belief. The two heads read different things: retrieval cost (oracle − flat) is 0.056 with opaque names and real descriptions but 0.185 with real names and no descriptions and 0.278 with rotated names, so the contrastive head reads descriptions and is damaged more by a wrong name than by a missing one — though needle-bsky's +26pp auto→tuned rewrite survives the oracle at +20.4pp, so most of that win was decode, not retrieval. Answers the deployment question directly: no, there is no naming route to declaring all 18 — retrieval costs 16.7pp under the best naming available and separated reproduces that gap to three decimals, leaving the already-measured ladder (regex→≤5 at 0.722/316 ms, regex-only at 0.833/0.022 ms, two-model agreement at 0.880) unchallenged. Side benefit: stripping all 18 descriptions cuts the median turn 808 → 532 ms, the only lever here that trades the other way. Harness and version check: canon reproduced needle-bsky's tuned-min exactly — 0.611, identical calls and confidences — on cactus-needle 2.0.7 against that experiment's 2.0.6. Also censused the shipped checkpoint while scoping this: 45,211,383 params, of which 18.6% are Engram hash-indexed n-gram tables (a parametric store the paper never mentions, added back after the paper concluded a SAN lacks one), and LoRA reaches 28.31M of 45.21M — out_proj, the paper's write path, is adapted, so the fine-tune negative was never a frozen-write-path story. Caveats: n=54 routable (one query = 1.85pp), the profile flip is 4 queries, separated is one author's rule applied once, and descriptions were stripped rather than degraded. |
nl2sh-dense/ | 2026-08-20 | done — a 25.6 MB encoder plus a page-level index lifts gold-in-sources 0.262 → 0.390 (p=0.0003); query reformulation is a clean negative; and the old 34-row eval would have called the whole thing a regression | RESULTS.md + sample_cyber.py + cyber_nl_ext.json + results_*.json | Issue #48: the on-device shell helper is retrieval-bound — Gemma 3 270M routes 0.706 with the gold page in context and 0.206 with real BM25, which surfaces it 26% of the time. The eval was extended first. sample_cyber.py drew 149 more commands from the same Zenodo/UCI corpus under the same tiered protocol and gen_nl.py wrote their NL, taking the independent eval from 34 to 164 leak-free requests over 132 distinct gold utilities (constant prior 0.012). That turned out to be the load-bearing step: on the original 34 rows the winning retriever lowers routing 0.206 → 0.147 while raising sources 0.235 → 0.382, and on the 130 new rows both rise — the old eval's headline would have been that better retrieval makes the system worse. A 23.5 MB encoder matches a 164.5 MB one. all-MiniLM-L6 int8 scores 0.341 in sources against bekko-a8m's 0.354 (one query on 164) at a seventh the disk, so the issue's 157-MB footprint worry does not bind. Page-level indexing is the free half of the win: grouping 31,169 chunks into 6,397 pages lifts BM25 alone 0.262 → 0.323 with no encoder at all, and composed with the dense arm reaches 0.390, p = 0.0003, 27 wins to 6. But feeding whole pages to the model does nothing (0.159 vs 0.165) — the index wants pages, the prompt does not. Reformulation lost, confirming muninn-rm3's prediction: RM3 costs 0.036 and dense-PRF costs 0.030, even though dense-PRF surfaces fcrackzip on the exact query the issue quotes. The abstention gate's margin >= 5 fails because 5 is in BM25 score units, not because it is a difference — a quantile-set absolute margin transfers as well as top2/top1 does; the ratio's advantage is needing no calibration sample. RRF is the wrong substrate for the gate (margin AUC 0.47–0.53, a coin flip) because it discards score magnitudes. End to end the retrieval gain is decisive and the routing gain is not: 0.128 → 0.165/0.183, p = 0.26/0.11 at n=164. A follow-up answers 'should we fine-tune the embedder' with no: one identity-initialized d x d matrix on frozen query vectors — 4,588 NL2Bash pairs, 40 seconds on 4 CPU cores, 4.2 MB, document vectors untouched — takes gold-in-sources 0.384 → 0.463 (p=0.024) and routing 0.128 → 0.201 (p=0.058), but the whole gain is on the 207 utilities the training data covered (+0.184 seen, −0.039 unseen), and a rank-64 adapter with 16x fewer parameters reproduces the same split — so the limit is utility coverage, not capacity, and a fine-tune would hit the same wall with more room to memorize. The largest lever turned out to be the corpus, not the retriever: a flash-lite pass over all 6,397 pages adding goal-level phrasings (Pleias-Redline-style) takes BM25 alone 0.311 → 0.427 and end-to-end routing 0.128 → 0.226 (p=0.0052), composing with the adapter for 0.555 / 0.250 — validated against a human-authored control (+0.098, p=0.0001) because both the eval's NL and the corpus's came from Gemini, and localised by a fidelity split (+0.224 on pages whose generated intents added vocabulary the page lacked, +0.086 where they only echoed it). Ships as a greppable handbook with the tldr-pages CC-BY-4.0 attribution this repo had been missing. |
nl2sh-retrieval/ | 2026-08-19 | done — the 350M model works after 25 minutes of fine-tuning (0.923 where a constant scores 0.000), retrieval is now the bottleneck, and an adversarial pass returned OVERSTATED on the first draft of the retrieval numbers | RESULTS.md + EXTRACTION.md + results_*.json | Oskar's six-component architecture — LLM-composed regex, a small model for intent, regex parameter extraction, man pages with remax-kb-style hybrid indexing, ICL grounding, and error logging — built and gated overnight. The gate is the result. Pleias-RAG-350M is trained to quote sources literally, which is exactly the operation monad-bsky measured its 56M sibling failing (51% identifier copying), so the bet was that converting generation into extraction rescues a tiny model. Handed the gold utility's tldr example every time: 0 usable commands in 40, verbatim rate 0.000, 6.5 s on 4 CPU cores. It quotes descriptions into cited prose, never a command — monad-bsky's zero-shot result one generation later. It degrades with source count (6/8 at 3 sources, 4/8 at 5, worse at 15), so k≈3 is an architectural constraint. Two harness findings cost a false negative: a prompt assembled from the special-token list alone gives a 0.000 parse rate that mimics incapacity (it must end with `< |
nl2sh-scoping/ | 2026-08-19 | done — the terminal-helper problem is utility selection over a long tail, not flag composition; and man pages are the coverage backbone, not the fallback | README.md + results.json + doc_corpus.json | Oskar, after the gh-mcp-regex-fit cascade: "how about a model + regex combo for something much bigger: bash + zsh commands". Scoping measurement run before building, to choose between a cascade (needs a thick head) and retrieval (needs a long tail the model does not know). Shell history was unavailable — a week-old laptop — and the substitute is arguably the better corpus: a helper is asked about what you would look up, not what you type most, and NL2Bash was scraped from forums and tutorials. One caveat governs everything: 60.3% of NL2Bash leads with find, so it is a correctness corpus and not a usage distribution; quote the non-find column. The head is thin and the tail is long: top 10 utilities cover only 29.1% of non-find requests, you need ~50 for 70%, and 176 utilities appear exactly once — a weak case for rules that answer and a strong one for rules that narrow. The difficulty is utility selection, not flag composition: 72.6% of non-find commands carry at most one flag. A self-correction worth the entry: the first draft argued for tldr over man pages, conflating RAG over man pages with putting a man page in the context window. Measured properly, whole pages run to 47k tokens but .TP option entries are median 56, 93.4% under 350, and tldr covers 96% of the top-50 utilities against only 50% of the used-once tail — where the requests and the model's ignorance both are. Both corpora, tiered by whether a chunk is quotable or needs composition. Unused lead: SYNOPSIS is a grammar, the same object needle-bsky compiled constrained decoding from. |
nl2sh-selfhist/ | 2026-08-19 | done — an eval where neither side is Claude's drops the fine-tuned model from 0.92 to 0.62; a real 16k-command corpus with a 0.189 constant prior replaces NL2Bash's 0.603 find-skew | RESULTS.md + corpus_probe.py + gen_nl.py + run_independent_eval.py + results_*.json | Oskar: "you've issued hundreds of commands tonight, you could use your own history" then "SURELY there are bash history logs to be found online". This session's 289 Bash calls proved unusable as a benchmark (26 general-shell, ~15 of one 'print lines X-Y' shape, and the description field is not persisted), but the search turned up the Zenodo/UCI hands-on cybersecurity training corpus (record 8136017, CC-BY-4.0): 16,065 real bash commands, 696 utilities, constant prior 0.189 (ls) against NL2Bash's 0.603 (find). Documentation coverage on that real distribution: 87.7% of invocations, 24.4% of utilities, 9.8% of the used-once tail, with the uncovered tail mostly undocumentable (ll a shell alias at 114 uses, ./ssh2john.py a local script, mfsconsole a typo) — arguing for reading shell config and $PATH over a bigger corpus. Found a builder bug: 379 tldr alias pages dropped as stubs rather than resolved to their target (whoami->id), a nearly-free +1.2-point fix. The capstone is the eval that finally has neither side authored by the model under test: the cyber corpus supplies real commands, and gen_nl.py has Gemini write the request for each (instructed not to name the utility; 4 of 38 leaked). The fine-tuned RAG model routes 0.618 leak-free (n=34) against its 0.923 on the NL2Bash gate — a 0.30 drop that is the cost of templated phrasing, the find-heavy distribution, and NL2Bash naming the answer 34.7% of the time. Failures are distractor-utility hallucination (pgmbentley, calligrastage) and abstention. This closes the eval-authorship problem that ran through the whole thread: every prior number measured against a self-authored or utility-naming eval is an upper bound, not a capability. Still utility-routing, not functional equivalence, which would be lower again. |
gh-mcp-regex-fit/ | 2026-08-18 | done — fitted routing rules lose to hand-written ones; the answer is neither, it is a cascade: precise rules first, a scored fallback that can also abstain second (+0.136 wild accuracy at zero abstention cost) | RESULTS.md + results*.json + wild.jsonl | Oskar, after monad-bsky: "what other tool use and routing use cases ought to sit behind a trained regex heuristic like that of regex_only.py" — then "build the fitting harness against the GitHub MCP catalog". monad-bsky left its 20 regex rules hand-written after reading the eval's failures, so 0.833 was fitted to an unknown degree. This searches for the rules instead, on the real 58-tool GitHub MCP catalogue (50 with upstream schemas, 79 routing targets once the seven method-enum dispatchers are counted), built from github/github-mcp-server's own committed schema snapshots. fit.py induces an ordered decision list by greedy precision-constrained covering (CN2/RIPPER shape) over structural cues, catalogue-vocabulary tokens and IDF-weighted schema-overlap features, abstaining rather than falling back. Result is negative and clean: every fitted arm loses. Best fitted reaches 0.239 on a held-out phrasing family and 0.351 on hand-authored queries against 0.546 / 0.486 for rules written by hand (McNemar p=1.7e-44 on family B, p=0.058 on the 74-row wild set); no regularisation knob — Laplace scoring, min-coverage 8, dropping bigrams — moves it. The diagnosis is vocabulary, not entity memorisation: the learned rules are sensible (tok:diff -> get_diff) but never fire on "what code does this PR actually change", whereas a human writes `\b(diff |
monad-specdec/ | 2026-08-18 | done — negative: Monad drafting for Baguettotron runs at 0.90x baseline at best; depth sets decode latency, so a 5.7x smaller model is only 2.1x faster | RESULTS.md + specdec.py + results.json + analysis.json + depth_scaling_both.json | Oskar: "could we use monad as a speculative decoder for baguettotron?". Both PleIAs, both LlamaForCausalLM on SYNTH, but different tokenizers — 7,397 of Monad's 8,192 tokens exist as strings in Baguettotron's 65,536-piece vocabulary and only 157 share an id — so drafts are verified by string-level exact match (Timor et al. 2025) rather than against target logits. The loop is lossless: every run is token-identical to plain greedy. It is also slower at every draft length: 0.90x at γ=1, 0.42x at γ=8. Three causes compound. (1) Latency tracks depth, not parameters: Monad is 64 layers to Baguettotron's 80 and decodes at 57.0 ms vs 119.9 ms, so c = 0.476 rather than the 0.18 its parameter count suggests. Truncating each layer stack gives 0.804 ms/layer at width 256 against 1.264 ms/layer at width 576 — a ratio of 1.57 where a compute-bound decode would show 2.25² = 5.06, so fixed per-layer overhead dominates and a narrower model of equal depth is barely cheaper. (2) A smaller vocabulary needs more draft steps: 3.25 chars/token vs 4.12 means 1.27 draft steps per target token, lifting the effective c to 0.602. (3) Acceptance lands just under break-even: 0.546 measured against a 0.602 break-even at γ=1. Monad is a good predictor — Baguettotron's own first-N layers agree with its full stack on only 2.1%/7.3%/25.0% at 20/40/60 layers — it is just not a cheap one. A viable draft would be ~24 layers at width 256 sharing the target's tokenizer (c ≈ 0.21, ~1.3x at γ=2), assuming acceptance holds, which it would not fully. CPU, 4 threads, fp32, batch 1, 5 prompts. |
monad-bsky/ | 2026-08-18 | done — a fine-tuned 56M generalist reaches ~2/3 of a purpose-built 45M tool-caller; the gap is transcription, not choice | RESULTS.md + ERRORS.md + params.json + recheck.py + results_*.json | Oskar: "do the same but fine-tune Pleias' Monad on the same task". Direct continuation of needle-bsky — same 18 Bluesky tools, same 62-query eval, same scoring code (imported, not copied), same 800 training rows from the same generator and seed. Monad is 56M params, 64 layers x 256 hidden, trained on SYNTH, with no decode grammar and no confidence head. Zero-shot it routes nothing — 0/54 routable, 0/62 parseable, it analyses the instruction instead of answering it. A full fine-tune (3 epochs, 108 min on 4 CPU cores) takes it to 0.481 routable at epoch 2 against Needle's 0.611 base and 0.722 best config; paired McNemar vs Needle-LoRA p=0.043, vs Needle two-stage p=0.0037. The gap is transcription, not routing: over the 41 eval arguments that appear verbatim in the query, Monad reproduces 0.512 against Needle's 0.780 base / 0.902 LoRA — austegard.com comes back as afethew.com, jetstream as jetforek — and more training makes copying worse (0.561 at one epoch). The obvious explanation is wrong and was retracted before publishing: both models carry 8,192-piece vocabularies and segment these strings identically (111 vs 109 pieces over ten identifiers), so the cause is the training objective — Needle's base weights, never exposed to this data, already copy at 0.780. It also invents undeclared tool names on 6.5-14.5% of queries, which a decode grammar makes impossible by construction. Constructive fix measured: keeping Monad's tool choice and refilling arguments by regex lifts args 0.296 -> 0.370 against a 0.444 ceiling, and what it cannot fix is free-text search terms, which have no structure to extract. Two invented numbers caught in one draft (Needle's vocab size, its layer count) — see ERRORS.md. Synergy (synergy.py, eight combinations, pure post-processing over both experiments' committed rows): one works. Where the two models independently name the same tool, that answer is right 0.880 across 0.455 coverage, against 0.741 for Needle's own confidence head at matched coverage — and the two signals compose (0.929 at 0.255). Agreement needs no confidence head, which is exactly what fine-tuning Needle destroys; the price is running both, 11x latency. Clean negative: calibration does not transfer — Needle's confidence separates its own correctness (0.584 right / 0.392 wrong) but is flat-to-inverted for Monad (0.486 / 0.532), and worsens with threshold. The other five combinations do not pay for a second model: union ceiling 0.806 vs 0.710, name-snapping buys ~3.7pp not 14, split roles (Monad chooses / Needle transcribes) reaches 0.407 args against 0.685 for Needle doing both, fallback rescues 1-2 queries, per-category dispatch tops out at 0.758 and is fitted. Four follow-ups (cascade.py, classifier.py, regex_only.py). A retry cascade (Needle confidence, then agreement, then escalate) reaches 0.613 coverage at 0.842 precision where the confidence gate alone gives 0.323/0.800 — cascading beats any single gate. Rewriting the ask adds 1-3 of 24 escalated queries and lowers precision; the english→Monad→Needle pipeline is unavailable because Monad corrupts handles inside its own think trace. Scoring Monad as an 18-way classifier over the declared names removes hallucinations and yields a softmax confidence but drops routable to 0.241 from 0.481 (verified not a harness bug via independent forward passes). And the null model nobody had measured: 20 regex rules with no model route at 0.833 routable, 0.022 ms — beating Needle's two-stage 0.722 and its oracle ceiling 0.778, holding at 0.824 on unseen template queries. The models' remaining edge is refusal (0.625 vs 0.183) and a confidence score. |
needle-bsky/ | 2026-08-18 | done — schema wording is worth +26pp; the confidence gate only works if you declare no optional arguments | RESULTS.md + ERRORS.md + params.json + recheck.py + evalset.jsonl + results_*.json | Oskar, overnight: "implement Cactus Needle in your compute environment and have it set up as an interface in front of some tools — maybe the Bsky tools in muninn-utilities and/or the ATProtoing skill". Needle 2 is a 45M-parameter, 14 MB tool-calling model for phones and microcontrollers; this puts it in front of an 18-tool Bluesky read surface drawn from the browsing-bluesky and atprotoing skills, with a CLI (route / ask / repl) and a 62-query eval set. Base-model top-1 routing is 61–70% across a 2×2 of schema wording × argument arity; only the wording contrast is significant (auto 0.444 → tuned 0.704 on 54 routable queries, paired McNemar p=0.0072). Schema arity moves the confidence gate: declaring an optional argument the query does not license makes the model fill it anyway, and since the head scores the whole call, a correct routing decision lands at 0.0004. Drop optional arguments and the same gate becomes monotone and usable — 38% coverage at 0.762 precision, 20% at 0.909, 13% at 1.000, against tuned's 2% coverage for the same precision. Confirmed on a single-tool catalogue where misrouting is impossible: 30 queries, confidence falls 0.199 → 0.111 → 0.068 as unlicensed arguments are declared, sign test p=0.043 / 0.043 / 0.00032. Two more numbers for anyone deploying it: an oracle five-tool catalogue is worth +11 to +17pp (retrieval, not selection, is where most remaining errors are — best measured arm 0.815), and declaring a sixth tool costs 3.6× the per-turn latency (284 ms → 1034 ms) and then nothing more out to 18, because retrieval is a fixed per-turn cost above five. tool_index_path does not help — it caches tool embeddings, not the per-turn query embed. Acting on both: splitting the 18 into five groups of ≤5 and routing in two steps works, but only if stage 1 is not a model — a Needle turn over group descriptions scores 0.370 routable (24pp worse than the flat 18) while ~20 lines of regex over structural cues scores 0.722 (+11pp over flat, against a 0.778 five-tool ceiling). Needle's contrastive retrieval head is much better at picking 5-of-18 than Needle-the-model is at picking 1-of-5 categories. Also found: the engine holds one global session per process, so alternating agents re-runs needle_init every turn and a loaded .cact can never be unloaded. The LoRA arm is a negative: 800 templated rows and ~2h of CPU moved routing 0.611 → 0.667 (paired McNemar p=1.0), left profile (0.25) and identity (0.33) — the two categories the templates specifically covered — completely unmoved, regressed off-topic refusal 0.625 → 0.375, and replaced every confidence score with None, since fine-tuning does not update the confidence head. Extraction is a second negative: across 22 attempts over live and constructed posts, every single one scored below 0.05 confidence (max 0.0434) — at any threshold that makes the routing gate useful, all of it escalates. Route with this model; do not extract with it. |
orchestrated-coding-pareto/ | 2026-08-16 | done — orchestration arms never activated; token economics, not accuracy, set the frontier | RESULTS.md + ERRORS.md + params.json + recheck.py + tasks/ + data/ | Continuation of luna-onprem-tco (PR #37): is a big orchestrator driving a fleet of Luna-class workers Pareto-optimal for coding? Built a 14-task bank (precise specs, hidden pytest suites validated against references before any model saw them) in three escalating tiers, run one-shot at haiku/sonnet/opus tiers in the CCotw Workflow harness with per-arm output-token metering, plus two orchestration arms (haiku+raw-test-feedback, opus-diagnoses→haiku-fixes) seeded from haiku failures. Quality saturated everywhere: haiku 14/14 = opus 14/14, sonnet 13/14 (accepted 1.0.0-01 as semver) — the ceiling survived two difficulty escalations ending in a 20-opcode stack-VM and a character-exact table formatter, so both orchestration arms went vacuous: zero haiku failures to orchestrate over. The measured story is verbosity: haiku emitted 6.7× opus's output tokens (280,717 vs 42,019; 11× on tier-3), which more than cancels its 5× per-token discount — haiku-solo $0.101/task vs opus-solo $0.079/task at equal quality; at Anthropic prices the cheap tier is Pareto-dominated by frontier-solo. Repricing haiku's measured token profile at Luna direct ($0.20/$1.20) gives $0.024/task (4.2× cheaper than opus-solo), DeepSeek-Flash $0.0057 (13.8×) — the thesis lives or dies on the fleet tier's sticker price and verbosity discipline, not on orchestration structure. Tie-back to PR #37: an orchestrated fleet is batchable (Luna batch halves it again) but at ~20k output tokens/task, saturating the 7.64 B-token/night self-host break-even needs ~380k tasks/night — self-hosting still doesn't pencil. Caveats carried in the writeup: all tasks are single-module fully-specified stdlib work (the saturation claim does NOT extend to ambiguous/multi-file/long-context coding); the harness's effort: medium has no documented mapping on Haiku 4.5, so some verbosity may be tunable; Luna prices inherit PR #37's secondary provenance. Tier-1's difficulty was misjudged and caught mid-run by early grading after Oskar questioned it — disclosed as an adaptive extension; 3 of 14 hidden suites had authoring bugs caught by reference validation before any model ran. Follow-up (same day, Oskar's effort question): re-run at effort: low cut haiku's tokens only 26% (verbosity is intrinsic, not the knob) but broke the ceiling — 12/14 — which finally activated the orchestration arms on the 2-failure seed: raw pytest feedback and opus-diagnosis both went 2/2 in one round, so the measured orchestrator premium over mechanical test feedback is zero quality at +$0.017/task; the best cheap pipeline (haiku-low + test-retry, 14/14, $0.080/task) ties opus-solo at Anthropic prices and wins 4.1x at Luna prices. |
mdbr-leaf-mt-bench/ | 2026-08-16 | done — no swap of the remax_kb default; leaf-mt-int8 ties bekko-a8m at 5.2x smaller | RESULTS.md + ERRORS.md + recheck.py + results_*.json | "Another embedding model to evaluate": MongoDB/mdbr-leaf-mt, 23M params / 1024-d, distilled from mxbai-embed-large-v1, #1 on MTEB v2 (Eng) ≤30M — run through bekko-embedding-bench's Part B harness (179-chunk blog + 179-chunk sklearn-AST self-retrieval, incumbents re-encoded on the same splits so every verdict is paired). The billing does not transfer: jina v5 nano q4 wins both distributions, decisively on code R@1 — 0.888 vs 0.581, Δ −0.307, 2 wins/57 losses, p<1e-5 — the same cell, with nearly the same margin, that settled the bekko verdict; blog R@10 −0.067 (p=0.036), and iso-byte truncation widens the gap (leaf@64 vs jina@64 code: −0.251). Where it does land: the compute-bound rung. The int8 export (23.7 MB) is a paired statistical tie with bekko-a8m in all four cells (all p>0.07) at 5.2x smaller and 8.0 vs 10.8 ms same-session 1-vCPU query — the smallest credible remax_kb embedder measured in this family, 19.2x faster per query than jina. Export quirk worth knowing: leaf's int8 is faster than its own fp32 (7.3 vs 15.4 ms) with no significant retrieval cost — unlike bekko, whose transformer-int8 ships _not_recommended — while q4 is dominated on every axis (slower than fp32 at 1 thread, only export with a directional code dip). Cross-run anchors all reproduced exactly (jina 0.631/0.978, bekko 0.575/0.888, corpus 11,380 chunks/674 files at sklearn 7cb1868aa); 30/30 recheck. Part A deliberately not run (an encoder that loses to jina on code has no path to moving "dense ties grep"); asymmetric teacher-doc mode (mxbai docs + leaf queries, the card's strongest configuration) untested — the natural next pass for an offline-index budget. Codec follow-up (pareto.png): remax does not beat the card's own plain sign bits — at the shared 128 B, blog 0.503 vs 0.547 (n.s.), remex 1-bit level with vendor binary on both dists, so the rotation/centering machinery buys nothing on a model whose quantization robustness was distilled in; quantize-before-truncate reproduces on a second model (remex 2-bit @1024 beats the fp32 MRL floor d=64 at equal bytes, +0.073 blog p=0.015 / +0.117 code p<1e-4, and ties the uncompressed 4096 B vector); the Pareto frontier is composition — binary-asym d=512 at 64 B hits full-fp32 quality (64x compression free), MRL-fp32 is dominated everywhere; and kb-k-sweep's "dims beat stacks" inverts within remax here (d=512 k=2 > d=1024 k=1 at 128 B). 40/40 recheck. |
luna-onprem-tco/ | 2026-08-15 | done, two passes — API wins at 800 seats by 2.7x; one RTX 5090 wins above ~4 h/day of flat-out generation | RESULTS.md + ERRORS.md + model.py + hourly.py + params.json + recheck.py | Asked to price the raw electricity of running a GPT-5.6-Luna-equivalent locally in Montgomery County MD against Luna's API price. The literal question has a clean answer that decides nothing. Electricity at EIA's MD commercial rate (16.4 ¢/kWh) is $0.0035 per million input tokens on a GB200 NVL72 — Luna direct ($0.20/M) is 57× that and even Luna batch ($0.10/M) is 29×. But electricity is 3.1% of the cost of owning the hardware, so the ratio is load-bearing for nothing. The binding constraint is a memory floor. Capability parity is set by measured index, not parameter count (OpenAI publishes none, and the "27 B dense" figures in circulation are unreliable): DeepSeek V4 Pro 0813 at AA index 53 vs Luna max 52, MIT-licensed — which means 1.6 T params, ~800 GB NVFP4, and a ~$450 k 8×B200 node minimum, bought whole at 5% utilisation as at 85%. At 800 office seats (55% daily-active, staggered 07:00–18:30, 2.5 h nightly batch) three usage intensities give peak utilisation of 5% / 17% / 85% and API bills of $6.1 k / $29.3 k / $153 k against $279 k self-hosted — API wins by 3.5× / 2.7× / 1.2×. The overnight batch cannot rescue it, and not for a reason that depends on any usage estimate: fully saturated for 2.5 h the box emits 1.98 B input tokens/night where break-even needs 7.64 B — short by 3.9× at 100% saturation, across the entire 20–35% MFU sensitivity band. The window where the box could run flat out is also the window where the API is half price; the two compound. It flips on token intensity per node, never on seat count — scenario B stays API-side at 6,000 seats because past one node each added seat buys capex as fast as savings, while scenario C flips at ~2,000. Two errors in the model itself were caught and fixed structurally, both flattering self-hosting: a verdict of "self-host" returned at 845% peak utilisation (one node's cost vs a nine-node bill; now nodes = ceil(peak)), and a serving power floor applied to parked hours (+35% annual kWh). Spin-off finding: rack benchmarks do not transfer to single nodes — 8×B200 decode costs $0.083/M output against the NVL72 rack's $0.014/M on identical GPUs, 5.8×, because MoE decode scales with NVLink-domain size, while prefill is flat. Caveat that dominates all others: the session's egress proxy blocked WebFetch and curl to every primary source (openai.com, eia.gov, pepco.com, artificialanalysis.ai, inferencex.semianalysis.com), so every constant came from a search-engine summary rather than the page; params.json tags each row's confidence and recheck.py (104 checks) fails if one lacks a source. The workload scenarios are authored, not sourced — the published per-seat figures found sit ~7× above even the heavy case and were rejected as not credible, which is the single judgement call most worth challenging. Second pass (2026-08-16), single GPU: same price book, 1/1000th the scale — one RTX 5090 (600 W, $4,700 street median in an August-2026 shortage) running Qwen3.8-27B (released 14 Aug, Apache 2.0, 27.78 B dense), a far closer capability match than V4 Flash was at SWE-Bench Pro 61.7 vs Luna's 62.7. Electricity is $0.18/hr at Maryland's EIA-corrected residential 22.2 ¢/kWh against $0.90–1.49/hr of Luna for the same output — 5–11×, not 57×, because a consumer card serving one stream costs $0.19–0.26 per million output tokens against the 8×B200's $0.083 and the rack's $0.014. It answers the opposite way from the fleet model, for the same reason: break-even is 4.2–7.6 h/day of flat-out generation at street prices (2.3–4.2 at MSRP-era prices), so the shortage roughly doubles the break-even and the used-GPU market outweighs electricity, model choice and rate schedule combined. Two premise corrections generalise: a quoted 180–200 tok/s is 1.56× the hard bandwidth ÷ weight_bytes decode ceiling (121 tok/s; ~97 at 80% MBU), reachable only via this model's multi-token prediction, speculation or batching — kept as a branch rather than corrected away, with a roofline-respecting 95 tok/s branch beside it; and prefill and decode contend for one card, turning 190 tok/s into a sustained 123–167 once 1/(1/decode + fresh_ratio/prefill) is applied. Price-book finding worth carrying alone: Luna's cache writes cost 1.25× uncached input, so caching only pays above a 21.7% hit rate — below it, re-sending is cheaper. Two further errors logged, one flattering self-hosting (prefill contention omitted in the scratch pass) and one overstating a correction ("1.9× the ceiling" quoted against the 80%-MBU figure, not the ceiling). recheck.py now runs 154 checks. |
ttt-embed-quantized/ | 2026-08-14 | done — artifact committed; fp32 nDCG@10 0.7152, inside the expected band | RESULTS.md + ERRORS.md + encode.py + recheck.py + data/{Dm,Q}.npy + data/meta.json | One-time SciFact corpus encode for the TTT-Embed x remex/remax quantization experiment (#33), so claude.ai — <2 docs/s on 1 core, detached jobs reaped after ~100 s — never pays for it again. Not a hypothesis test: the deliverable is Dm.npy (5183, 256) fp32 + Q.npy (300, 256) fp32 + meta.json, encoded with jina-v5-nano model.q4.onnx @ v5-nano-8a7f00aa (SHA256-verified) at the 2026-07-08 codec eval's exact settings — dim=256, max_length=384, title + ". " + text, Document: /Query: prefixes, last-token pool, truncate-then-L2-normalize — so that eval's fidelity numbers carry over. fp32 nDCG@10 = 0.7152 (R@10 0.8346, R@100 0.9483), inside the issue's 0.60–0.72 band near the top; 14.7 min on 4 vCPU at 5.9 docs/s. Re-scored cold by recheck.py through a deliberately disjoint code path (sorted() over Python floats, explicit math.log2 DCG) reproducing 0.715232 vs 0.715232 to <1e-9, 17/17 checks, with negative controls that collapse to 0.003–0.004 — a sanity check that cannot go red is not evidence. The issue's HF-CDN warning did not reproduce: all three files landed first try on us.aws.cdn.hf.co, the host the spec calls un-allowlisted, confirming bekko-embedding-bench's per-environment reading — allowlist state is a fact about a container, not about a host, and the retry loop is kept only for the claude.ai path. The pinned encoder is knowingly superseded — the mirror's own PERFORMANCE.md and this repo's METHODS.md both say the authors' upstream q4 is smaller and more faithful — and was used anyway, because comparability with the prior eval requires identical weights, not better ones; flagged rather than silently upgraded. Prior art was found and deliberately declined: rotation-decorrelation already caches a jina_scifact_corpus.npy for this exact corpus and embedder, but it is corpus-only and its settings are unrecorded, so reuse would have risked a silently non-comparable matrix that every shape check would have passed. Caveats: 27% of docs hit the 384-token cap, so better than a quarter of the corpus is encoded from a prefix; no fp32-vs-q4 fidelity was re-measured on SciFact (the 0.975 cosine is inherited from NFCorpus/muninn); and pytrec_eval does not build here, so both scorers share an author. |
subagent-messaging/ | 2026-08-12 | done — 4 of 5 documented claims hold; the reply rule does not | RESULTS.md + ERRORS.md | Does the Claude Code SendMessage/ListAgents tool pair behave as its description says? Live test rather than reading: Opus 5 parent, Haiku 4.5 general-purpose peer instructed to report every envelope verbatim, 3 agent runs, ~121k subagent tokens. The reply rule is wrong. Both the tool doc and the harness footer appended to every delivered message say to reply by copying the incoming envelope's from into to; for subagents that value is the agent type, so the send returns No agent named 'general-purpose' is reachable. Two general-purpose peers emit two identical unusable from values — the attribute cannot distinguish senders even in principle, and the agentId from the spawn result is the only handle. Envelopes are asymmetric: <agent-message from=…> parent-side, bare <system-reminder> subagent-side, so a subagent cannot route a reply by inspecting what it received. ListAgents does not exist inside a subagent — ToolSearch("select:ListAgents") returns No matching deferred tools found, not an unloaded schema — so peers cannot be discovered from below and the topology is a star through the main conversation, not a mesh; peer-to-peer coordination requires the parent to hand out ids at spawn. Confirmed as documented: delivery enqueues to the receiver's next tool round and never interrupts (probe 2 landed after a sleep 20 finished, not during it), and resume-on-send works with context intact — but it is undetectable by the resumed agent (asked directly, it reported no gap or restart marker, "reads as one continuous conversation") and costs a full agent turn, ~40k tokens, each time. Also observed: the harness neutralizes instruction-shaped tags in agent→parent output (< → <\) and relabels them as findings rather than instructions. Prior art, two passes. Account-local found nothing (zero sendmessage hits across claude-workspace, scoped xr nothing above 0.371); account-wide xr surfaced claude-skills/orchestrating-agents at 0.539, a different mechanism (API agent pools) that carries a factual defect — v0.5.0 tells the reader the native runtime lacks inter-agent messaging and never mentions either tool. The published pass demoted two findings: cross-session messaging shipped 2026-08-07 in v2.1.224 with a thorough official page that states the queue-never-interrupt model almost verbatim, and claudefa.st already documents resume-with-intact-context and a better-measured cost chain (199k→324k over eight rounds). Still unpublished anywhere found: the from attribute, the envelope asymmetry, absent ListAgents in subagents, and resume being undetectable by the agent. And it gained a contradiction the account pass could not: claude-code#48160 (closed as duplicate) and ruflo#2028 (open) both report that subagents can receive but cannot originate SendMessage — this peer originated three sends to main successfully with no AGENT_TEAMS flag, lacking ListAgents instead, the exact inverse. Either fixed since, or environment-specific; CCotw vs local terminal is the uncontrolled confounder. Closest published match to the addressing finding is claude-code#42999 (closed as not planned), where a user-assigned name fails silently while the id works — adjacent but distinct, since from is not a name and fails loudly. Caveats: parent↔subagent only, one session; true cross-session peers (<cross-session-message>), Remote Control, named teammates, and Workflow-spawned agents all untested, and the from-as-address rule may well hold for them. |
lowbit-scan-crossover/ | 2026-08-09 | done — positive; the reported scale gate is a 4.1 ms constant, and bit planes beat the shipped kernel 2.4–5.2x | RESULTS.md + fit.py + layout.py + roofline.py + arms.py + steelman.py + xover.py + hamkern.c | Challenge the inevitability of BLAS beating low-bit storage at small corpora, per memory dab41dd6 ("compression is SCALE-GATED and below ~150k rows its win is negative"). It is not a scale gate. dab41dd6's own table fits t = 4.108 ms + 32.40 ns·n; the same numpy expression here fits −0.78 ms + 33.98 ns·n — per-row cost agrees to 5% across the two machines, and the entire crossover is the constant. n* = a/(b_f32 − b_ham) recovers the reported gate at ~68,000, derived rather than interpolated between two rows. The generalisable form: in t(n) = a + bytes·n/(BW·eff), n is a multiplier identical for every kernel, so two kernels can cross only if one has a > 0 — a reported crossover with a ≈ 0 on both sides is an artifact of the two n values bracketing it, and fit.py is the check. Nothing here crossed at any n from 100 to 1e6, cold or warm, at any ISA from SSE4.2 up, single-query or batch-1024 (4.2x at batch 1024, the narrowest point). Where the numpy time goes: np.bitwise_count is fine at 14.7 GB/s; .sum(axis=1) over a 4-wide inner axis is 1.9 GB/s and 62% of the kernel — a reduction shape, not bit-packing. Storing the words as contiguous bit planes recovers 5.2x at k=256 and 2.4x at the shipped d=512·k=4 config, pure numpy, no compiled dependency; the compiled kernel is 37x over BLAS warm, 19x cold, versus the 1.7x the shipped idiom gets. Both disconfirming arms failed: an adversarial challenging pass named AVX-512 VPOPCNTDQ as load-bearing and cold cache as untested — rebuilt at -march=x86-64-v3/v2 with zero vpopcnt in the object it costs 6%, and L3-evicted it is still 19x. So the original result is not attributable to the 1 vCPU container's core, cache, or instruction set. Also corrects the correction: dab41dd6 refuted remax-hamming-speedup's "beats BLAS at every N" using a different configuration — k=256 (4 words/row) vs the shipped d=512·k=4 (32 words/row) — and the narrow-reduction pathology is specific to the former; the shipped claim reproduces here (2.58x vs its published 2.43x at N=50k). Caveats: one machine; ARM/NEON untested; single-threaded; latency only, not latency-at-fixed-recall; the 4.1 ms constant's cause is unidentified, only its existence and size; and nothing transfers to the remex ADC path, whose 13.5x the family model predicts is gather latency — flat in n as 901e3c06 reports, but the bit-width leg is untested. Prior art — the C kernel is a rediscovery. remax/src/remax/_native.py already ships a __builtin_popcountll Hamming scan that remax.packing.hamming_distances already dispatches to, whose docstring already reports 25–35x over the NumPy path and already identifies the 100k–1M cache falloff; remax/core.py and QUERY_PATH_SPEED.md already record that a gather cannot use the popcount kernel (38–45x). Externally it is textbook — faiss's HammingComputer32 is four uint64s XORed and popcounted, and what gcc emits at v3 is Muła/Kurz/Lemire's Harley-Seal. hamkern.c is kept only as a fallback-free roofline reference. The mandated account-wide xr check was skipped — run afterwards it returns remax/packing.py at rank 1 and _native.py at rank 8, or rank 2 with -r remax, in 175 ms warm. A first attempt raised ModuleNotFoundError: remex and was written up as "xr is unavailable in this container" instead of fixed with pip install remex onnxruntime tokenizers (under a minute; remex is on PyPI from the same author). An ImportError in a mandated check is a missing dependency, not a broken check — that wrong diagnosis reached METHODS.md, RESULTS.md and a PR body before being caught. Logged in the duplication map, with the install recorded under Environment gotchas. What survives as new: the constant-term reconciliation, the .sum(axis=1) narrow-axis diagnosis (the bitwise_count path, not the LUT path _native.py analyses), the bit-plane numbers, the 6% -march measurement, and a wiring gap — remax_kb/_hamming.py already imports from remax.packing but never calls its native dispatch, so the compiled kernel is one import away from the shipped scan. Method note: a first pass labelled inversion was retrofitted onto a result already in hand and did not fire; the finding came from family traversal run properly afterward. |
account-index-corpora/ | 2026-08-09 | done — qualified; account-wide PR bodies +4.6% (a floor), tombstones +7.1% but 94% deleted data | RESULTS.md + clone_depth.py + corpora_scoped.json + results.json | Should the account-wide index carry the two corpora that measured as wins per-repo — deleted files (history-tombstone-index, 0/6 -> 6/6 on mechanism) and merged PR bodies (pr-decision-log, 6/8 -> 8/8 on rationale)? claude-workspace#197 names the risk correctly as size, not answer quality, since an off-class corpus already measured inert rather than harmful. Size is the half that costs seconds instead of a 22-minute sharded encode, so it was answered alone, via a new account.py corpora that chunks through the real build path and never loads the encoder. Re-run over all 65 repos on a runner in 2 min 18 s: PR bodies +4.6% (1,953 chunks over 1,444 merged PRs) and tombstones +7.1%, against a tree of 42,578 chunks that matches the published manifest exactly — so corpora is measuring the real index, not an approximation. The PR number is a floor: 12 of 65 repos returned HTTPError on /pulls while cloning fine, so the PAT reads contents but not pull requests on them, dropping claude-workspace's own 154 merged PRs out of the total. Both 3-repo estimates were wrong in opposite directions (tombstones 11.8% -> 7.1%, PRs 3.2% -> 4.6%) without moving the verdict. On 3 repos first: PR bodies +3.2% (419 chunks, 261 merged PRs), and the account meets the condition METHODS.md records for believing the remax result transfers — median body 1,577–3,197 chars, 11 effectively empty. Tombstones: +11.8% nominal and not worth it. The first run said +564% — 74,822 chunks against a 13,257-chunk tree — because a deleted file gets no stat() and no rglob, so every filter hcindex.discover applies to the tree (extension, skip_dirs, skip_names, exclude, the 1 MiB cap) has to be reapplied by hand; without them a 767,692-line deleted embedding dump enters a corpus the live index refuses. Filtered it is 47x smaller, but claude-workspace still contributes 1,484 tombstone chunks against a 232-chunk working tree, ~94% of it sub-1 MiB JSON data dumps and ~23 chunks of actual prose and source. Two account-scale effects the per-repo experiment could not see: the relocation guard must compare across repos (the 2026-07-28 migration deletes in one repo and lands in another — 540 files skipped only because the check was widened), and candidates are restricted by basename to keep it linear. --depth 50 resolved: it cost nothing (6.5s vs 7.3s vs 7.4s full, summed, no consistent sign) and would not have worked — git log --diff-filter=D sees only the grafted window, so depth 50 found 2 of muninn-utilities' 18 deletions; coverage is a function of commit rate, not of anything anyone chose. Now depth 1, or full history when tombstones are on. Caveats: this measures size only — the answer-quality half still needs a full encode and a benchmark that does not exist; and the PR total is understated until the PAT gains pull-request read on the 12 repos it cannot currently list. |
account-routing-tier/ | 2026-08-06 | done — qualified; 87-90% @k=3 of 9 repos, not safe as a default | RESULTS.md + run.py + results.json | Can a small always-loaded index of per-repo summary cards route a query to the right partition, so a whole-account index can live as per-repo release assets fetched on demand? The failure mode is unforgiving: a flat index that ranks badly still contains the answer further down, while a coarse tier that routes wrong makes it unreachable and returns a confident result from the wrong repo. 9 repos on disk, 25,899 fine chunks; the coarse tier is 0.1-0.35% of it, so storage was never the constraint. 30 queries about internals only (ascii_fold, the CSR builder, NVFP4 dequant, sklearn's CSR indptr) so cards must route on similarity to a summary that does not contain the answer; gold is an oracle (flat RRF over all chunks) rather than hand labels. Content cards beat front-matter cards at every k with 2.5x fewer chunks (@3 80% vs 73%, 26 vs 64 chunks); both together reach @1 53%, @3 87%, @5 97%. The diagnosis came from a failed fix: 4 of 8 initial misses wanted sklearn-bench whose card lacked a README because CARD_FILES held only README.md and scikit-learn ships README.rst — fixing that moved recall@1 47% -> 43% and the same four queries still missed. scikit-learn's README has zero occurrences of "gradient boosting", "one-hot", "cross validation", "sparse" or "estimator" against 296 and 311 files in-tree; front matter states identity, routing needs inventory. Splitting large repos into per-directory cards to fix a 350x card-capacity imbalance (sklearn 0.033 card-terms/chunk vs 11.8) backfired — @1 53% -> 47% for 3.7x the cards — because ranking a repo by its best card makes more cards more draws, inflating a split repo's maximum for reasons unrelated to relevance. Verdict: 13% of queries land in no fetched partition at k=3, silently; k=5 fetches over half the partitions and defeats the point at this scale. Usable only behind confidence-gated escalation — route to top-3, widen if the best fine score is weak — which turns a silent wrong answer into latency. What broke: a confidently wrong diagnosis (above); the harness sat inside the corpus it measures for the fourth time here, after being diagnosed in code-index-duplication and guarded in hybrid-code-index — knowing a failure by name did not prevent reproducing it twice more; the oracle is not reliable gold (one query routed correctly to claude-container-layers at rank 1 and was scored a miss because flat search disagreed), so these are agreement-with-flat numbers, not accuracy; and self-pollution is structural, since experiments is both an indexed repo and where these writeups live. |
pr-decision-log/ | 2026-08-05 | done — positive; 8/8 vs tree's 6/8, and the three corpora are orthogonal | RESULTS.md + run.py + prs.json | Are PR descriptions worth indexing as a searchable decision log? Code says what, commit messages say what changed, PR bodies say why — including what was rejected. Proposed as an alternative to hunk-level change indexing, and the volume argument alone favours it: 43 merged remax PRs is 87 chunks, +12% over the tree, where hunks would be thousands of near-identical neighbours (the pollution that cost repo-index 20% of its corpus). Eight "why" questions written from CLAUDE.md's claims rather than PR text, scored as marginal value over a tree whose CLAUDE.md already documents decisions. tree 6/8 -> tree+PRs 8/8. The two gains are cases where the tree carries the outcome but not the reason: [PR #65] Consolidation: −4,879 lines for why the bench harness left the wheel (CHANGELOG.md ranks first in both arms and records only that it happened), and [PR #61] Restore rotations_ assignment via write-through setter at rank 1. Tombstones add exactly nothing on rationale (6/8 -> 6/8) — which is the more useful finding: the three corpora are orthogonal, tree answering what, tombstones how did the deleted thing work, PRs why, and a corpus aimed at the wrong question class is inert rather than harmful, so they stack. Caveats: the generalization threat is severe and was recorded before running — every remax PR is Claude-authored, median body 2,727 chars, none empty, where most repos have one-line or blank bodies; this measures "PR bodies are worth indexing when written like this". n=8, one repo. One gold list held the bare substring "PR #" matching any PR chunk — fixed, and outcome-neutral (that query hits via CLAUDE.md at rank 1 in both arms), but it is the third answer-key defect of this shape in this line of work. The real objection is architectural: PR bodies are not in git, so indexing them makes network access and a token a hard dependency of a full rebuild — every other corpus here comes off the filesystem, which is why the indexer runs offline and in CI. A real version needs a cache with a staleness policy that degrades to tree+tombstones rather than failing the build. |
history-tombstone-index/ | 2026-08-05 | done — positive; 12/12 fused vs 5/12 working-tree-only | RESULTS.md + run.py + results.json | Does indexing deleted code add anything over a repo that already documents its rejections? A current-state index structurally cannot hold code that no longer exists — but that is only valuable if the knowledge left with the code. remax is the hard case: its CLAUDE.md mandates "a measured rejection is an asset — delete the driver, never the record", so removed apparatus leaves a prose writeup in bench/results/*.md. Tested against remax (144 commits, 10,042 deleted lines, 17 true deletions) because this repo cannot answer it — 73 commits and zero files deleted-and-never-restored. Recovered each dead file at its last living revision, headed with its removing commit's subject, and scored RRF(dense, stored-BM25) over three corpora. The convention works — for existence: 5/6 from prose records alone. It cannot work for mechanism: 0/6. A record is prose about a verdict; "the encoder, its CSR-builder and a BEIR benchmark were all built" does not tell you the signature, the batching, or what the tests asserted — that left with the file, and no writeup discipline short of pasting the code retains it. Tombstone-only scores 4/6 + 6/6; the union is 12/12, strictly better than either arm, for +19% corpus. Arms are complementary, not competing. What broke: relocations look exactly like deletions — five src/remax/bench/* files were moved to bench/*, not deleted, and indexing them inflated the tombstone corpus 27% and manufactured a false current-only mechanism hit (a live crossover.py satisfying a query whose gold was its deleted path). A current-state index scoring on a mechanism-only query was the tell that the answer key was wrong. Fixed by detecting relocation by content (>50% of non-trivial lines present in a live file), not by path or basename — basename would have wrongly dropped src/remax/bench/__init__.py, which is genuinely gone. The correction strengthened the result (1/6 -> 0/6). Caveats: n=12, one repo, queries written by someone who knew the answers, gold matched on filename substrings, and only whole-file deletions are indexed — removed hunks inside surviving files are probably the larger population and are untested. The defensible product claim is narrower than "index your git history": index what was removed and never came back; everything else in history is a near-duplicate of content already indexed. |
hybrid-code-index/ | 2026-08-05 | done — hybrid wins 24/24; two assumptions refuted | RESULTS.md + hcindex.py + bench.py + bench_incremental.py | repo-index was a markdown sidecar with a code afterthought and a single dense arm; the target is a general-purpose hybrid code indexer. Six arms over three query classes (rediscovery / keyword / duplication), each scored against an answer key that already existed for another purpose. rrf(dense, stored-BM25) scores 24/24, beating bm25 alone (23), dense alone (22) and every rg variant. Stored BM25 beats ripgrep as the lexical arm decisively (23/24 vs 17/24), and the gap is almost entirely duplication (8/9 vs 3/9): ripgrep returns a set, and 'find me a file like this one' is a ranking question no amount of term-counting recovers. Adding a third arm HURTS — rrf(all 3) drops to 22/24, because RRF is unweighted so a weak arm votes as loudly as a strong one. More retrieval arms is not monotonically better. The .json dilution was refuted: 79% of the corpus is generated results data and it is inert (24/24 with and without), so no build-time exclusion is warranted — unlike the outputs/ model-generation case, where near-duplicate prose competed directly with real answers. Volume does not predict pollution; similarity to real queries does. Storage is not free though: BM25 postings inflate to 6.36 MB / 138k terms with JSON vs ~1 MB without. Rebuild cost forced incremental: a full build is 537 s, vs 0.2 s for a one-file change (2735x), and the incremental result is verified bit-identical (max delta 0.000e+00) rather than an approximation — safe because the encoder is per-chunk independent and remex is data-oblivious. BM25 cannot be incrementalized the same way (IDF shifts for every term on any insert), which generalizes: any component fitted on the corpus — PCA, k-means, ITQ, PQ codebooks, IDF — breaks the equivalence that makes incremental safe. Incremental does not fix the committed-blob cost: 1.00 MB dense + 6.36 MB postings per rebuild is ~1.5 GB of git history at 200 rebuilds, which wants the artifact published as a release asset instead. Not done: graduating the winning arm into ask.py, moving the artifact out of git, a second-repo check, and history/tombstone indexing (untestable here — 73 commits and zero files deleted-and-never-restored). |
code-index-duplication/ | 2026-08-05 | done — positive; shipped into repo-index | RESULTS.md + run.py + results.json | Should repo-index/ index .py as well as .md? This repo's own bekko-embedding-bench already measured dense retrieval as not beating grep at NL->code localization (r@5 0.656 vs 0.596, n=59, ns) — so the usual reason to index code is a measured non-reason here. But localization is not the failure this repo has with code: METHODS.md's duplication map records three independent reimplementations of one bench harness plus three more near-identical pairs, all found by hand after the fact. That map is an answer key written before this experiment existed, for an unrelated purpose. Scored leave-one-out hit@5 over 831 flat 60-line windows from 190 .py files (flat not AST, because AST-vs-flat was noise at p=0.424 in the earlier bench): querying with a file's own text finds a documented sibling 9/9 at ranks 1-3, and content-only scores identically to with-path-header, so it is content matching and not filename matching — the confound that would have made the headline hollow. NL-description queries get 8/9; grep handed the most distinctive def name out of the query file gets 8/9. Dense ties grep, it does not beat it (n=9), but the arms need different things: grep's needs a draft containing a distinctive name, the NL arm needs no draft at all. The single NL 'miss' is the index returning _lib/pipeline.py/_lib/textnorm.py — which is where that code was extracted to, per the same map, so it is the better answer against a stale key. Three process failures, all self-inflicted and all in the measurement: (1) run.py embeds its own NL queries verbatim and so retrieved itself, top-5 on 4 of 9 — excluding it moved content-only NL 6/9 -> 8/9; (2) the first number described excluding only the query file, a configuration the shipped tool does not use, so --file and the harness were both changed to exclude the query's directory before any number was recorded; (3) after adding .py to repo-index, keyword agreement looked like it fell 10/10 -> 7/10, but all three 'regressions' were the index returning the definition instead of a prose mention (ascii_fold -> _lib/textnorm.py, GRID_VERSION -> grids.py) and the grep arm was still restricted to *.md — a baseline scoped narrower than the system under test reports improvements as regressions. Matched arm: 9/10, rediscovery unchanged at 5/5. Shipped: repo-index indexes .md+.py (0.14 -> 0.27 MB) and grew --file. |
bekko-embedding-bench/ | 2026-08-04 | done — split verdict; one prior reversed, one default upheld | RESULTS.md + instances.json + recheck.py | Handoff claude-workspace#185: benchmark hotchpotch/bekko-embedding-v1 (a8m/a25m, 384-d Matryoshka mmBERT) for two separate decisions. Part A: the reversal did NOT survive a bigger sample — n=6 → n=59. The first run said bekko beats identifier rg at r@5 0.806 vs 0.667 and cleared the pre-registered gate. Re-mined to n=59 (630 PRs harvested, 97 candidates with live gold), dense/a8m lands at 0.595 against grep's 0.596 — a dead tie — and worse at r@10. No dense-vs-grep comparison is significant in any of the four cells. The gate now passes or fails depending on which cell you pick, which is itself the finding. The identifier-poor stratum is still n=1 of 59 — 10x the sample bought zero additional instances, independently corroborating the ~0.3% base rate. So the 2026-07 retirement of the semantic tier stands; the n=6 reversal was noise. Two things do survive: a25m > a8m is real (+0.061, 13 wins to 1, p=0.0018), which reverses the n=6 call that a25m doesn't earn its cost; and RRF(rg, dense) is directionally best in every cell (r@10 up to 0.762 vs grep's 0.682, bootstrap CI excluding zero but sign test p=0.09) — suggestive, not established. Cost at n=59: dense 200k tokens vs rg -l 315k, i.e. dense is now 1.6x cheaper. A code-trained encoder does not rescue it either: jina-embeddings-v2-base-code (161M, 768-d, 30 languages) scores r@5 0.630 against general-text bekko-a25m's 0.656 — it loses to the general encoder, at 6x the encode cost (612 MB / 61.9 min vs 124 MB / ~10 min), with no comparison significant. The obvious confound was ruled out: path-only retrieval (no code content at all) scores 0.304/0.370, so the code body is genuinely carrying signal — the specialization just adds nothing on top of it. Every arm clusters within noise of grep, and RRF fusion is the only thing that consistently helps. Part B is a regime choice, not a dominance. Official jina v5 nano q4 wins 11 of 12 iso-byte cells and owns the top quality rung — but bekko-a8m encodes a query 12.9x faster on 1 vCPU (11.3 vs 146.4 ms, 11.2x tokens/s), which is the entire design point of a 7.7M-active-parameter model and which an iso-byte table cannot see. The measured ratio matches the ~12x FLOPs ratio (4x384x1152 vs 12x768x3072), so it is architectural, not a q4 artifact. Result is an iso-quality ladder: bekko-a8m up to blog R@10 0.575 at 11.3 ms, a25m to 0.598 at 35.0 ms, jina alone above 0.60 at 146.4 ms. Keep jina when quality-bound or compute is amortized; take a8m when the reader is a 1-vCPU container or the corpus is large. End-to-end through remax_kb.read.KB.search, though, only 2.3x of that 12.9x reaches the reader — a ~50-60 ms constant (_stacked_simhash_encode rebuilding k Haar rotations by QR per query, from manifest params that cannot change) is 87% of bekko's query. Caching it per opened index is one line, verified to give identical codes and hits, and restores 11.6-15.1x — a finding about remax_kb, not about bekko. Swapping the projection does not fix it: every option is a per-query construction cost of 14-76 ms against a ~6 ms encode, and remax_kb v2's default srht is the slowest of them (1.4-3.0x slower than Haar at every dim), deliberately, because it is seed-only and bit-for-bit reproducible by a JS reader. remax's own rht_rotation at rounds=2 does reproduce its documented 1.5-1.8x -- a different function. Projection choice is a portability decision; the latency is a caching decision. Still no remax_kb swap made — the call is the deployer's, and the code-distribution gap (0.983 vs 0.888) is where bekko was advertised strongest. Matryoshka trimming vs quantization: quantization wins directionally at every budget — against the vendor floor d=64 (256 B, R@10 0.520), remex 1-bit @384 is 48 B at 0.564 — and the vendor's own HAKARI table agrees (binary@384 −12.93% vs truncation-to-64d's −17.51%). But a paired-McNemar audit at the end found this corpus cannot establish it: 179 chunks from 11 blog posts, where one query is 0.56 pp, and SEVEN OF EIGHT headline claims are noise — including the one I led with (remex 2-bit beats the uncompressed vector: +0.011, p=0.625). Only truncation-to-d=64 costs recall survives (p=0.009). The direction is consistent and matches the vendor's much larger eval; the demonstration is theirs, not mine. The one overwhelming result is Part B's: jina beats bekko on code-distribution R@1, +0.168, 31 discordant wins to 1, p<1e-5 — that, not the twelve correlated iso-byte cells, is the real basis for 'do not swap'. Compute went to the wrong arm: 78 min encoding 41,500 sklearn chunks for a 6-instance code-search benchmark, while every embedding-quality conclusion rode on 179 chunks encoded in seconds. Earlier passes also quoted an off-spec d=12 tier (strawman, retracted) and mis-priced shared structure (over-correction, retracted — remex's codebook is 28 B and the rotation is seed-derived). The R@50 ceiling (26/179) is partly harness artifact, has a 7.3% true shared floor, and is recovered by BM25 (14/26) and RRF (best overall, R@10 0.615) — not by query expansion (3/26, and R@50 drops), reproducing the repo's muninn-rm3 negative. 2-bit beats 1-bit in all 8 cells, so bekko is a Jina-side embedder and the SPECTER2 one-bit-beats-two result does not transfer. Confirmed the artifact facts independently (404.3 MiB → 124.1 MiB at cosine 0.99992 to its own fp32, holding on both distributions) and failed to reproduce the card's 5.5x OpenVINO-over-ORT claim (20.1 vs 21.8 ch/s on 4 vCPU). |
remex-vs-higgs-ablation/ | 2026-08-02 | done — mixed; 2 of 4 pre-registered predictions failed, and one published mechanism refuted | RESULTS.md + tables.md + gate.log + axes.png / marginals.png / seeds.png | Issue #8: does remex (exact fp32 norm + dense Haar rotation + scalar Lloyd-Max) buy anything for retrieval-index compression over the QuIP# -> HIGGS -> TurboQuant lineage (randomized Hadamard + per-block scale + Gaussian-MSE-optimal grid)? Full 2x2x2 factorial, 11 arms x 6 bit widths x 4 corpora (d=100/768/784/1024) x 5 rotation seeds x 2 metrics, scored against fp32 exact search rather than qrels. Only axis C moves. Rotation is null (-0.0004 recall@10) and norm handling is null (+0.0007); the codebook is an order of magnitude larger (+0.0082 cosine / +0.0112 IP), peaking at +0.035 recall@10 at 2-3 bits and decaying to zero by 8 bits, with the effect fmnist784 sweep (raw pixels, norm CV 31%, d=784) ran; axis B is flat there too (+0.0005 cosine / +0.0009 IP), so exact-norm storage is null even off cosine-trained encoders — and the published mechanism for the 1-bit remex win was refuted (it predicts the effect fades as norm spread grows; fmnist has the most spread and the biggest remex win, and reverses under cosine where the norm is divided out). Two further mechanisms were measured and refuted; the effect is now carried as MEASURED but UNEXPLAINED. Process: the two-sided calibration gate caught Lloyd-from-random-init producing grids worse than scalar at 6-8 bits, and a scheduled adversarial review then found five more blocking defects -- a stale codebook served by a cache keyed on the problem rather than the method (8-bit vector arm 87% worse than scalar), a Lloyd-Max MSE identity evaluated off the fixed point (+16% at 8 bits, in the direction that makes the gate more permissive), a 'provably no worse' guarantee that was argued rather than enforced, a block/sub-vector divisibility bug hitting only the HIGGS-like arm, and a gate that never certified the grids behind any glove result. Scoring q.xhat without dividing by ` |
lattice-representation-hypothesis/ | 2026-07-31 | done — negative result (opening thesis refuted by the experiment's own adversarial + WordNet arms) | RESULTS.md + THEORY.md + fca.py + noise_reversal.png | A Paper Skygest Bluesky post pointing at arXiv:2603.01227, "The Lattice Representation Hypothesis of LLMs" (LLM embedding geometry encodes an FCA concept lattice; meet/join as half-space operations). Hypothesis: the concept algebra has a broken join — FCA's meet extent is a bare intersection (exact under half-spaces) while its join extent is the closure of a union, and the paper's Definition 7 writes that join as a literal set union, contradicting its own Appendix B. Measured at scale: 0 meet phantoms across 9,615,370 concept pairs, join overshoot mean 0.60. Then the experiment's own arms killed it. (i) Definition 7's "conic hull" clause makes the join exactly R(Y_A n Y_B) under linear independence — verified 96/96 by an independently written Minkowski-sum LP, strict only under conic dependence (43/96, 34/96). (ii) The "phantoms" are not errors: they are the least upper bound, and the gold label in the paper's own task (join of {dog, wolf} is canine, which contains foxes). (iii) The premise was mathematically wrong: meet = objs_of(B1 u B2) and join = objs_of(B1 n B2) are both plain half-space intersections (0 identity violations over 30 configs x 8 seeds); the join needs fewer constraints (0.83 vs 5.55). The only non-representable object is the plain set union, which isn't a lattice operation — and it is indeed recovered worst of the three. The one measured effect — join degrades more slowly than meet under probe error on Jaccard (5% flip: 0.939 vs 0.835) — carries a 12x target-size confound, flips sign on symmetric-difference error in one of two contexts (cross: join 0.110 vs meet 0.028), and its size-controlled version is reproduced more strongly by random-direction controls; the proposed "closure absorbs noise" mechanism is contradicted by a negative overshoot-vs-error correlation (-0.44). Surviving caveats, aimed at the paper's setting: WordNet noun hypernymy gives an extremely thin lattice (15 concepts from 150 objects x 13 attributes, 0/78 cross-cutting attribute pairs, 66% of meets empty, ~42% of joins the top element); learned attribute directions can be exactly antipodal (mutual coherence 1.0000, since living_thing and artifact are complementary), so the canonical form's linear-independence assumption is not secured by `d >> |
svgview/ | 2026-07-30 | working on Linux; Windows build never run on a real machine | README.md + src/ | andri.dk on Bluesky, arguing that launching a full browser to render PDF or SVG is "bonkers insane" — narrowed in his own reply to systems doing it internally. Tested the SVG half by building the alternative: a native Windows-first viewer wrapping resvg, ~600 lines. Measured here: 4.8 MiB executable, 12 MiB resident, 16 ms exec→window, 20 ms parse+render to a 1000 px PNG. Verdict: he is right about SVG and resvg had already done the hard part; the argument does not transfer to PDF, where the honest options are wrapping Chrome's own PDFium or accepting hayro's coverage gaps. Windows compiles in CI; the file dialog, icon, and association scripts are unverified. |
erdos-gyarfas/ | 2026-07-28 | open problem; partial results | README.md + note.html + tutte_coxeter_lemma.py | this session |
ms13-campaign/ | 2026-07-24 | closed (compute exhausted; open maths recorded) | SUMMARY.md (academic writeup) + NOGOS.md (ledger) + BLOGPOST.md | issue #169: campaign against Morell–Skutella Conjecture 1.3 (two-sided unsplittable-flow rounding). No counterexample. Produced instead: a reduction showing 1.3 restricted to 2-path instances is a linear-discrepancy question on network matrices with demand-scaled columns — a connection neither literature appears to draw (full-text greps of TVZ/Swamy/MSW25 find no mention of Doerr, "linear discrepancy" or "totally unimodular"); a theorem settling that question for k=3 (R = 3/4 exactly, unconditional, census complete through m=10, exact branch-and-bound on the 2 maximal classes); and a ledger of 20 refuted families/claims including two conjectures of our own (12.1 refuted at k=4, the staircase conjecture at k=8). Rediscoveries correctly identified as such: the tightness gadget is Morell–Skutella Fig. 3, the equal-demand bound is Doerr 2004. Enumeration shown dead by arithmetic (~2,070 h at k=4). Open: Q7′ (column-scaled Doerr bound, general k). Methodology notes: a false-positive "counterexample" caught at the certificate gate when two independent verifiers turned out to share one blind spot; every over-claim had the same shape (clean at k≤6, false at larger k). |
ssuf-beta/ | 2026-07-24 | done (scoped) | RESULTS.md + engine.py + calibration.py + family.py | claude-workspace#165: quantitative hunt for the SSUF cost-preserving violation constant β* following the Goemans/DGG conjecture disproof (16/15 < β* ≤ 2). Built an exact-rational β* engine (breakpoint-enumerated convex-hull membership LP via sympy's tested simplex, after a hand-rolled one failed its own sanity test). Calibration against the real Rybin instance was blocked — no arXiv writeup exists, and this session's WebFetch can't reach the source X thread (HTTP 402, no working mirror) — so calibrated instead against a fully hand-derived, independently-constructed triangle-conflict instance (β*=1/2, exact match) and swept a parametrized generalization of it — that family's β* has supremum exactly 1 (approached, never attained or exceeded), a clean negative result short of even the original refuted β=1 bound. Literature gate confirmed TVZ's planar +2·d_max bound and ring-loading bounds (1.1D/1.3D) from primary sources. Honest scope cut: no claim here reproduces or exceeds β*=16/15; gadget search, ms13 engine sweep, and the ring-loading secondary target were not attempted. |
discrepancy/ | 2026-07-24 | done (D(4) + n≥17 deferred) | RESULTS.md + growth.png | issue #166: certified discrepancy lower-bound records — Komlós per-size K(n) + Beck–Fiala small-t exact values, max-min engine with exact certificates. Literature gate first (logged on the issue): Kunisky's K ≥ 1+√2 record reframed Target A to per-size records; 2025 Bansal–Jiang resolution of Beck–Fiala for t ≥ log²n left small-t exact values open. Beck–Fiala: D(2)=2, D(3)=3 exactly — the CEGAR SAT search rediscovered the Fano plane from scratch and proved it's the minimum-ground-set (n=7) witness; D(2)=2 (triangle minimal); D(4,n≤9)=3 with D(4)∈{3,4,5} open (PG(2,3) computed weak: disc 2). Small-t truth sits on D(t)=t, far below 2t−3. Komlós: certified rational per-size records K(3)≥1.571, K(4)≥1.731 (beats Kunisky's own n=4 tree matrix 1.707), K(5)≥1.785, K(7)≥1.830; proved the "exact Δ vs published δ" gap on Kunisky's family is empty (Δ=δ, one-line proof) before wasting compute on it. All records verified by an independent second code path (verify_certificates.py); calibration gates G1–G9 green first. |
woodall/ | 2026-07-23 | phase 1 (verifier+calibration done, search deferred) | README.md | issue #163: SAT/MIP dijoin-packing verifier for Woodall's conjecture τ=3 counterexample search (Goemans-fall follow-on). Built a CEGAR SAT verifier (python-sat/cadical+glucose) for ν(D,u) and brute-force τ; transcribed Schrijver's (D1,u1) counterexample directly off a 600dpi render of Figure 6 in the Feofiloff survey PDF (not from memory). Calibration gate 1 (Fact 7.1, ν=1/τ=2) passes exactly, after catching and fixing a real transcription bug (one dashed arc misread) via cross-checking the graph's derived 3-fold symmetry and the paper's "4 critical cuts" claim against an initial read that gave only 3 and a spurious ν=2. Generalized D1 into a ring-of-length-2i family from its own orbit structure and validated the paper's odd/even parity claim (i=3,5 counterexample, i=2,4 not) exactly. A first light random-search pass (16,000 trials) found 0 candidates, confirming unstructured random generation is a weak filter-pass rate (0.15%) for τ≥3 — the real search needs the structured null-arc-resolution and Williams-catalog generators, explicitly deferred (not silently dropped) along with gate 2 (Cornuéjols-Guenin D2/D3 calibration). |
pdf-streaming-test/ | 2026-07-05 | done (shipped) | RESULTS.md + test_streaming.js | "Speed up the UX of austegard.com/web-utilities/pdf-text-extractor by streaming pages one-at-a-time; also check if parallelising improves throughput." Added a bounded worker pool + streaming display (pages append to the output pane in strict page order via a nextToFlush cursor as they complete) and a URL API knob (`&concurrency=1 |
atproto-pad-login/ | 2026-07-04 | done | RESULTS.md + pad_login.mjs + 04_connected.png | "you can log into ATProto as muninn — try this pad URL" → Drove austegard.com/bsky/pad.html end-to-end as muninn.austegard.com via Playwright/Chromium: opened the shared-pad URL, clicked Join, filled the login dialog with $MUNINN_BSKY_HANDLE + $MUNINN_BSKY_APP_PASSWORD, waited for rtc.login() to establish the PDS session and rtc.connect(peerDid) to send its WebRTC knock. Result: status flipped Local only · 0 peers → Connected · 0 peers (session up; the invited peer just isn't online sharing, so no data channel — a one-sided join can't complete). Load-bearing finding along the way — headless Chromium via $HTTPS_PROXY returns net::ERR_CONNECTION_RESET on every HTTPS goto unless launched with --ssl-version-max=tls1.2. Curl and Node fetch work; Chromium doesn't. NetLog shows the CONNECT tunnel returns 200 and Chromium's TLS 1.3 ClientHello (~1700 B, ECH extension included) goes out — then the socket is reset (SOCKET_READ_ERROR net_error=-101 os_error=104) before any TLS response. --ignore-certificate-errors, ignoreHTTPSErrors: true, and even --disable-features=EncryptedClientHello don't help — only pinning TLS 1.2 does. So the session's egress-gateway MITM terminator can't complete a modern Chromium TLS 1.3 handshake. Side finding: the docs claim the browser NSS trust store is pre-loaded with the proxy CA; ground truth is that /root/.pki/nssdb was empty — certutil -L returned only the header. Adding /root/.ccr/agent-proxy-ca.crt with certutil -A -t 'C,,' is needed for the proxy-terminated cert to verify after the RESET is worked around. |
session-relay/ | 2026-07-02 | done | RESULTS.md + relay.py + README.md | "How might this be accomplished?" → Joshua Shew's Bluesky post: one Claude session built a relay service so two other sessions could chat and share inbox best practices. Reproduced with ZERO deployed infrastructure — the shared Turso DB is the relay: one relay_messages table (channel, sender, AUTOINCREMENT seq as cursor) + a ~190-line CLI (init/post/poll/wait/history, 5xx backoff). Live run: two concurrent agents with disjoint seeds (muninn-a: docs/architecture.md lifecycle; muninn-b: experiment conventions) negotiated a joint 5-item "experiment survival checklist" over channel hub-coord — 4 messages, one round-trip each way, ~75s, clean CONSENSUS/ACK termination, with verifiable two-way knowledge transfer (final checklist contains facts neither seed had alone) and genuine critique (b fact-checked a against Error Patterns, restructured its draft). Protocol findings: table-global seq leaks cross-channel counts (cursor discipline handles it); shell quoting is the ergonomic tax (mandate the post ... - stdin path); front-loaded opener + explicit delegation → 1-round convergence. Transport is container-agnostic (HTTPS to Turso) — works across CCotw/Claude.ai session boundaries, unlike the /tmp-files original. Caveat: for two Muninn sessions the marginal value over shared memory is live negotiation, not knowledge transfer. |
omnigent-library-eval/ | 2026-06-29 | done | RESULTS.md | "Is this anything you could make use of?" → omnigent-ai/omnigent, the open-source meta-harness (5.3k★, 18 days old, alpha 0.3.0.dev0, Apache-2.0) that orchestrates 11 vendor harnesses (Claude Code/Codex/Cursor/Pi/…) with policies + sandboxing + multi-device. Verdict: NOT a claude-workspace dependency. It's a competing harness — a server+CLI that wants to OWN orchestration/model-routing/sandboxing, all of which CCotw already provides; the Python SDK is a client to a running omnigent server, not an importable lib; it can't even run here (no model key by design, wants Node 22/tmux/bwrap). But three borrowable wins: (1) free — omnigent/spec/skill_sources.py reads the SAME SKILL.md format with a claude family, so Muninn's whole skill library loads with zero porting; (2) the Polly orchestration prompts — though the cross-vendor review discipline they encode Muninn can ALREADY run in CCotw today (Gemini reviewer via the Cloudflare AI Gateway, gemini_generate()); omnigent only uniquely adds cross-vendor implementer agents (Codex/Cursor/Pi as full coding harnesses), a narrow win; (3) the ALLOW/DENY/ASK three-level policy engine (docs/POLICIES.md: blast_radius/spawn_bounds/cost_budget) as a reference design if Muninn ever formalizes its settings.json permissions. Rec: keep on radar as a place to run Muninn-the-agent outside CCotw; don't take the dep. No code executed (alpha server, no key). |
q4-official-vs-ours/ | 2026-06-28 | done | RESULTS.md + run_batched.py + results.txt | remax_kb#23: run the q4 head-to-head claude.ai couldn't (egress blocks both weight hosts) — does the model authors' own onnx/model_q4.onnx (137.8 MB, HF Optimum) match/beat our JinaQ4ONNXEmbedder build (~170 MB, remax_kb#14) on retrieval fidelity to fp32, deciding whether to upload ours to HF? NFCorpus, 2058 docs/100 queries, 4 vCPU. Result: ours is DOMINATED — official wins every axis at once: smaller (138 vs 170 MB, −19%) AND strictly more faithful to fp32 on all four metrics (nDCG@10 0.4291 vs 0.4250, per-doc cos 0.976 vs 0.974, recall@10-vs-fp32kNN 0.870 vs 0.862, Spearman ρ 0.980 vs 0.976). The hoped-for edge — our int8 embedding-table mop-up holding where Optimum's generic q4 degrades — is refuted; Optimum handles the EuroBERT Gather at least as well. Decision: do NOT upload ours; close the loop. embedders.py now steers users to the official asset. Side findings: docstring's "cosine 0.975 to fp32" holds (ours 0.9743 on this larger subsample); q4 is NOT faster than fp32 on CPU (~13–14 min each — int4 dequant offsets size; "~2x faster decode" didn't reproduce); patched the bench's one-shot encode(all_docs) OOM (~26 GB attn Expand) with numerically-identical mini-batching. Reinforces the prior-art-check lesson: the upstream repo already shipped a smaller, better q4. |
jina-remex-vs-remax/ | 2026-06-27 | done | RESULTS.md + score_fidelity.py + fidelity.png | "Can we apply the remex (not remax) quantization optimizations to our Jina q4 embedding — as a practical compressed-Jina format, not necessarily beating remax at a byte budget?" First clears up the two q4s: q4-the-model (JinaQ4ONNXEmbedder, weight quant, fp32-parity float output) is orthogonal to remex/remax, which compress the vectors; remex replaces the remax 1-bit step, not the q4 model. Recall-vs-qrels saturates both ways (muninn fp32 ceiling 0.90/1.00; NFCorpus fp32 floor 0.241 — embedder-limited), so switched to the saturation-proof metric: fidelity to fp32's own ranking (recall@k vs fp32-kNN chunk-level, Spearman ρ, recon cosine — remax's own bench metric). Result: remex (rotation+Lloyd-Max scalar) ≫ remax (1-bit SimHash) at every byte budget — NFCorpus n=120: remex ρ 0.92–1.00 vs remax 0.63–0.74; remex 4-bit @ d768 (384 B) near-lossless (ρ 0.998, R@10-vs-fp32 0.96); remex 2-bit @ 192 B (ρ 0.978) crushes remax d768/k2 @ 192 B (0.741); remex 1-bit @ 96 B beats every remax config. Headline: bits beat stacks — graded magnitude (Lloyd-Max) dominates more sign-bits (stacked SimHash); full-dim-low-bit > truncated-dim-high-bit. remex is the data-oblivious Haar rotation our own ITQ-rejection work (remax#46) endorsed. Rec: ship remex into remax_kb as the near-lossless/mid-byte codec (default 4-bit @ d768); needs a new SPEC binarizer type + ADC scan path (numpy+scipy), additive not drop-in. Caveats: muninn n=5; norms excluded from B/row (Jina unit-norm). |
remax-hamming-speedup/ | 2026-06-26 | shipped (PR) | RESULTS.md + bench.py + latency.png | remax_kb#15: the 1-bit Hamming scan (_hamming.hamming_scan, popcount LUT gather) was ~10× slower than a BLAS float-cosine search at small N, forfeiting the latency half of the 1-bit format's promise. Benchmarked the issue's candidates head-to-head (LUT vs bitwise_count u8/u64 vs ±1 BLAS matmul vs float cosine) over N=600→1M, single-thread BLAS. Winner is the cheapest candidate (approach 1): np.bitwise_count over a uint64 view of the XOR — ~10× over the current LUT and faster than BLAS float cosine at every N (e.g. N=10k: 0.58 ms vs LUT 7.83 ms vs cosine 2.08 ms), zero-copy, codes stay bit-packed (256 B/row). The ±1 matmul (approach 2) is 2–6× slower and costs 8–32× RAM (OOM at 500k); compiled SIMD (approach 3) is unnecessary. Issue's success criterion (Hamming ≤ cosine at N≥10k without losing storage) exceeded — holds at all N. Shipped: _hamming.py + read_v2.py swap to a shared _popcount_rows fast path with a numpy<2.0 LUT fallback (numpy≥1.24 floor preserved) + remax-free regression test (remax_kb PR). Bit-for-bit exact vs the old LUT across 8 realistic (dim,k) widths incl. non-multiple-of-8 byte rows. |
lfm25-230m-verify/ | 2026-06-25 | done | RESULTS.md + run.py | "test this Bluesky claim — RUN the model" (@sungkim post) on Liquid AI's LFM2.5-230M. Spec-checked the HF card, then loaded and ran the weights on container CPU to verify the runtime-checkable claims. Confirmed: 229.7M params, Lfm2ForCausalLM/model_type=lfm2, open weights, coherent CPU generation at 29.2 tok/s (fp32, 4 vCPU — the slow path vs card's quantized 42/213 tok/s). Training claims (19T tokens, distill-from-350M) aren't runtime-observable. Minor nuance: post/card say 32K context, loaded config carries 128K positional ceiling. Verdict: post accurate. |
lfm25-embedder-remax_kb/ | 2026-06-25 | done | RESULTS.md + lfm25_embedder.py | Follow-on from lfm25-230m-verify: if the 230M runs locally, does Liquid's LFM2.5-Embedding-350M work as a third remax_kb embedder — in-process CPU, no 847 MB Jina download, no Gemini API key? Wrote an LFM25Embedder (sentence-transformers, CLS-pool, 1024-d, query:/document:) on remax_kb's Embedder protocol. One real fix: the model's bidirectional remote code predates transformers' seq_idx shortconv kwarg (5.12.1 TypeError) — patched at our layer, not the model's cached files. Tiny-corpus 1-bit pipeline: 3/3 topical top-3 (matches Jina torch). Full-float head-to-head on the 73-post muninn corpus: LFM2.5 0.73/0.83 R@5/R@10 — below lexical 1.00/1.00 and Jina v5-nano 0.90/1.00, despite being the larger model; embeds slowly (1.1 chunks/s fp32, ~19.5 min). Verdict: viable where in-process/no-key/no-network is the hard constraint and retrieval is tolerant; Jina still wins on quality. Closing analysis: remax's 1-bit step is corpus-global (can't quantize up front); per-chunk-memory levers = fp16 buffer, streaming Welford mean, dim-truncation (non-MRL caveat), and int8 model quant for RAM+speed. |
muninn-rm3/ | 2026-06-25 | done | RESULTS.md + bench.py | The platform-less, agent-less site-search floor (no model to host, no Claude to expand the raw query). Pure RM3 vs plain BM25 on the muninn corpus (5-query phase0 gold). RM3 is a dud — identical R@5, hurts R@10 (1.00→0.90 whole-doc); skip it. But plain BM25 (whole-doc) = 0.833/1.00 — ties Jina-q4→remax (0.833/1.00) at ZERO inference/agent/per-query cost, all in the Worker. Only the agent-expansion path (1.00, unavailable to site search) beats it. Caveat: these are in-vocab acceptance queries; the residual Gemini buys is vocabulary-divergent (paraphrase) queries — BM25/RM3 can't bridge that lexical gap, no platform-free option does. Recommendation: drop Gemini for plain in-Worker BM25 IF muninn searches are keyword-ish; keep Gemini only if paraphrase-heavy. Decision input: actual query mix. |
muninn-embedder-bakeoff/ | 2026-06-25 | done | RESULTS.md + jina_remax.py + embed_one.py | Test our special-case Jina→remax as a Gemini replacement for muninn search (+ how it's practical). On the muninn corpus (5-query phase0 gold): Jina-q4 full-float = 0.900/1.000 (identical to fp32 — q4 is free); Jina-q4→remax 1-bit at d=512/k=4 (256B) = 0.833 R@5 / 1.00 R@10 (near the float ceiling; the shipped d=256/k=8 default is the weak config at 0.667 — dims beat stacks here too). So Jina→remax is a credible Gemini replacement; lexical (1.00) still edges R@5. Practicality: indexing is free (offline CI pack with Jina), only the online query-embed is hard — Worker can't host 170MB (10MB bundle/128MB mem), Workers AI has no Jina, so serve queries from a CF Container/endpoint running q4 (replaces the paid per-query Gemini call with cheap fixed compute; q4 is what makes that light). Migration = corpus re-pack (Jina space, d=512/k=4) + query-embed service swap; KV 1-bit mechanism unchanged. Sidebar: CF-native Workers-AI BGE alts underperform (bge-large 0.733/bge-base 0.667). Caveats: n=5; no live Gemini (Jina-0.90 float is the reference); embeddinggemma-300m HF-gated. |
rotation-decorrelation/ | 2026-06-25 | done | RESULTS.md + sweep.py | "How can I trust either end of the ITQ pendulum? Embedder-specific? Test the decorrelation angle." Controlled study, remax/bench metric (self-retrieval recall@10 vs float32 kNN), pure numpy on precomputed caches: SPECTER2 (specialized) + Jina-v5 (general), k-ladder {1,2,4,8}, simhash/itq/decorr, in-corpus vs transfer, 3 seeds. Resolution: the pendulum is 3 interacting axes — (1) k: ITQ wins k=1, decays/reverses on the ladder (#46's mechanism reproduced); (2) protocol: ITQ in-corpus overfits (gap ~0.02–0.03 SPECTER2, ~0.04–0.05 Jina; simhash/decorr zero gap); (3) embedder: ITQ's edge is 3× bigger on general Jina (+0.048 k=1) than specialized SPECTER2 (+0.015) — partly embedder-specific but mostly overfit. My NFCorpus 'win' = perfect storm (general × in-corpus × k=1); #46 = specialized × transfer × ladder. Honest config (transfer+ladder, k=8): simhash beats itq on BOTH (SPECTER2 −0.029, Jina −0.015). Decorrelation (α-mix) is a wash — ties simhash everywhere, no overfit gap. Verdict: keep parameter-free SimHash (#46 upheld, now for general embedders too); the open lead doesn't pay. |
recall-per-byte/ | 2026-06-25 | corrected (re-derived rejected work) | RESULTS.md + sweep.py | Generative-thinking move (random stimulus "river") reframed compaction toward information-per-stored-bit (the rotation). Recall-per-byte bake-off on cached NFCorpus fp32 Jina vectors (600 docs/120 q): remax StackedSignBit vs SimHash vs ITQ vs PQ. Apparent headline (ITQ/PQ@16B beat shipped remax@256B) does NOT hold: ITQ was already tested rigorously and rejected — remax#46/PR#47 (closed unmerged, SPECTER2 n=10k): learned ITQ loses to centered SimHash at every ladder rung, deficit grows with k. My k=1/600-doc "win" is the exact in-corpus overfit artifact #46 diagnosed (transfer rotations beat in-corpus ones); I never tested the stacked ladder where ITQ definitively loses. PQ not novel either (remex/TurboQuant owns codebook compression; my 256-centroid book on 600 docs flatters it). Lesson: recall() prior art before claiming a win. One genuinely open lead (from #46's parting question): a decorrelating joint multi-rotation objective — diverse-across-stacks, better-than-random per-stack — validated on SPECTER2 with transfer + ladder, explicit kill criterion. |
jina-int8-remax_kb/ | 2026-06-25 | done | RESULTS.md + quantize.py + bench.py | Pivot from quantizing the weak LFM2.5 to quantizing the strong embedder: int8-quantize Jina v5-nano's ONNX export to cut its 847 MB download. quantize_dynamic(QInt8) → 212 MB (4.0× smaller). Head-to-head on the 73-post muninn corpus (same stage_b.py methodology): fp32 0.90/1.00 @ 8.3 ch/s vs int8 0.83/1.00 @ 16.7 ch/s — 4× smaller, 2× faster, R@10 untouched; the R@5 dip is one query (Q3) sliding past rank 5. fp32 reproduces the prior 0.90/1.00 exactly (harness validated). int8 Jina dominates the LFM2.5 local option on every axis (R@5 0.83>0.73, 212<919 MB, 16.7≫1.1 ch/s). Standing rec stays lexical 1.00/1.00; int8 Jina is the real-vector fallback. Follow-up (NFCorpus, 600 docs/120 qrel-queries, full-float + 1-bit recall): overturns int8. Per-tensor dynamic int8 is domain-fragile — 0.445 per-doc cosine to fp32 on medical abstracts (vs 0.83 R@5 on muninn tech text; probe rules out seq-length: 0.44@256 vs 0.41@512). Blockwise 4-bit q4 matches fp32 (0.975 cosine, cos R@10 0.241 vs 0.242, 1-bit R@10 0.208 vs 0.222) at 170 MB < int8's 212 MB; q2 too far (0.73). Embedding-table workaround: MatMulNBits leaves EuroBERT's ~400 MB Gather fp32 (naive int4=465 MB > int8), so int8-mop-up the embedding → q4=170/q2=141 MB. 3-bit unsupported (ORT {2,4,8}). Answer to "is int8 the floor before 1-bit?": no — 4-bit blockwise is smaller, more faithful, domain-robust; floor is 4-bit; the int8 embedding table is now the size-dominant cost. Recommended quantized Jina = q4, not int8. **SHIPPED: q4 integrated into remax_kb as JinaQ4ONNXEmbedder (oaustegard/remax_kb#14) — embedder + deterministic scripts/build_q4_onnx.py + gated test; model.q4.onnx hosted (SHA-verified) on the jina-v5-nano-mirror release; fp32-parity query cosine 0.977; opt |
Truncated — view the full README on GitHub.
Similar repos
Contributors
Languages
Python
86.6%
HTML
5.2%
ASP.NET
3.0%
JavaScript
1.9%
Shell
1.3%
Lean
1.0%