Kisoku 1.6B evaluation results

Raw per-sample outputs of lm-evaluation-harness 0.4.13 for every model in the Kisoku 1.6B technical report, published so the report's tables can be checked from the source files rather than taken on trust. Each run folder holds the harness results_*.json (aggregate metrics, config, task versions) and one samples_<task>_*.jsonl per task (every prompt, model output and score).

• Report: https://github.com/0arch-io/kisoku/blob/main/report/kisoku-report-draft.md
• Model: https://huggingface.co/0arch-io/kisoku-1.6b

Layout

results/<run-name>/<model>/results_<timestamp>.json
results/<run-name>/<model>/samples_<task>_<timestamp>.jsonl
contamination/    contamination audit: scripts, patterns, hits, summaries, per-dataset outputs (out/)
logs/<batch>/     harness stream logs and the run scripts for each evaluation batch

The run folder name is the original name used when the evaluation was launched (model, suite, and for RULER the context length, for example kisoku-longC-core or qwen3-1.7b-yarn2-ruler-65536). Runs prefixed with _ are diagnostics, probes or alternate-precision checks. Suffix -clean marks runs on the decontaminated task variants. The same run reached us in more than one archive in several cases; the copies were byte-identical, so one copy is kept.

Runs

Reading a results file

Open results_*.json and read the results key: one entry per task, with metric names of the form metric,filter.

• PIQA, ARC (easy and challenge) and HellaSwag: use acc_norm,none.
• GSM8K: use exact_match,strict-match.
• HumanEval: use pass@1.
• MMLU, TriviaQA, BBH and DROP: use the primary metric of each task as reported in the harness output.
• RULER: the group average stored in results.json is wrong at context lengths other than 4096. Compute the score as the mean of the 13 individual RULER task values at that length (each stored under the key <length>,none, for example 65536,none), ignoring the group entry. Entries with a negative value or fewer than 13 tasks mean the length was not fully evaluated.

The n-samples key gives the number of examples scored, and config.model_args identifies the model and dtype.

Scrubbing note

Absolute machine paths in the files were replaced with <local> (for example pretrained=<local>/kisoku-1.6b-base-longC-1299), the storage bucket name with <bucket>, and host addresses and login names were removed. The model folder name is kept so each model stays identifiable. Metric values and sample contents were not modified. Benchmark text and model outputs that happen to contain example paths or private-range IP addresses (for example router setup answers in HellaSwag) are left as they are.

Links

• Technical report: https://github.com/0arch-io/kisoku/blob/main/report/kisoku-report-draft.md
• Model: https://huggingface.co/0arch-io/kisoku-1.6b

heldout-chat/

The held-out conversation test from report section 7B. scripts.jsonl holds the 297 simulated-user conversation scripts. rollout-<model>.jsonl holds each model's replies to every script (Kisoku chat passes 5, 6, 8 and 9, and Llama 3.2 1B Instruct and Llama 3.1 8B Instruct as references, each given a system prompt saying it is Kisoku). judge-<model>.jsonl holds the automatic judge's verdict per turn (ok true or false, with a one-sentence problem when false). The acceptable-turn percentages in the report are computed from these. The on-policy correction data built from these rollouts is training data and is not released.