This is a preview, not a finished assistant. It is the base model 0arch-io/kisoku-1.6b after nine supervised fine-tuning passes. This repo is pass 9 (the pass labelled sft009 in my run names). It loses the thread in longer conversations, is behind Llama 3.2 1B Instruct on a multi-turn test, and states wrong answers confidently. Read "Known failures" before using it.
All numbers come from the technical report draft (draft 3, section 7B and 7) and were measured by me.
Chat fine-tuning ran from 2026-10-04 to 2026-10-06 on the TPU v4-32, nine supervised passes of about 70 to 95 minutes each (about 131K tokens per step). Every pass started from the final long-context base checkpoint (Phase C, step 1299) with a fresh optimizer. No pass continued from an earlier chat model. One DPO experiment was run and not shipped.
Training data (every example containing a benchmark test item was dropped by verbatim matching, and a rescan of the final shards found zero hits):
Thinking mode was trained in pass 3 and removed. "Hold your ground" examples were removed (the model could not tell its right answers from wrong ones).
Base-style tests, no chat template, same harness as the base card (lm-evaluation-harness 0.4.13, bfloat16, RTX 4090). The first base column is the stage 3 checkpoint, before long-context training; the second is the released base, which the chat model was tuned from.
| Test | Base (stage 3) | Base (released) | Chat, pass 9 |
|---|---|---|---|
| GSM8K | 15.3 | 15.0 | 20.6 |
| HumanEval | 13.4 | 14.6 | 14.0 |
| ARC-Easy | 65.4 | 64.4 | 66.0 |
| ARC-Challenge | 39.8 | 40.1 | 42.2 |
| WinoGrande | 57.3 | 56.8 | 57.2 |
| PIQA | 73.9 | 73.0 | 73.0 |
| HellaSwag | 57.7 | 57.8 | 57.2 |
| TriviaQA | 22.8 | 23.1 | 20.8 |
| MMLU | 33.0 | 34.2 | 29.8 |
| BBH | 29.2 | 28.6 | 26.3 |
Against the released base, math went up by 5 points, MMLU down by 4, BBH by 2, and the rest moved by about a point or less. No confidence intervals.
| Model | 4K | 32K | 64K |
|---|---|---|---|
| Kisoku chat, pass 9 | 76.5 | 60.5 | 54.3 |
| Kisoku base, final | 76.1 | 58.8 | 56.7 |
| Llama 3.2 1B (base) | 73.5 | 56.7 | 49.2 |
Single runs of 650 samples, no confidence interval. The chat model keeps most of the base long-context ability. The Phase C mix included synthetic tasks resembling RULER, so see the base card's caveat. The chat model was run at these three lengths only.
After pass 5 failed a real nine-turn conversation, I built a test that is not a replay. A teacher model wrote 2,396 conversation scripts of up to nine user turns that make sense whatever the assistant answers (small talk, a casual build request, "make it shorter", "that's not what I asked", a change of topic and a return). The chat model answers every turn itself. The teacher then reads the transcript once and marks each assistant turn acceptable or not. One script in eight is held out by a hash of its id (297 scripts, about 2,530 turns) and never trained on.
| Model on the same 297 held-out scripts | Turns acceptable | Bad first turns |
|---|---|---|
| Llama 3.1 8B Instruct | 59.5% | 10% |
| Llama 3.2 1B Instruct | 56.7% | 11% |
| Kisoku chat, pass 6 (best of mine) | 50.3% | 8% |
| Kisoku chat, pass 9 (this model) | 48.9% | 10% |
The Llama models were each given a system prompt saying they are Kisoku (with Kisoku's facts and style) so the judge would not penalize their own names. They still lost some turns by calling themselves "a large language model", so their true figures are a little higher.
Kisoku chat is 6 to 8 points behind Llama 3.2 1B Instruct on multi-turn conversation. Llama 3.2 1B was distilled from the 8B and 70B models and post-trained by a large team. This model is nine passes by one person over three days on a base trained on about a sixth of the data. The judge is strict for everyone (the 8B model clears only 60%), and almost no conversation gets through all nine turns with every turn acceptable (0 to 2 of 297 for every model). I have not measured how much the judge's marks vary between two readings of the same transcript. Passes 6 to 9 (50.3, 49.5, 49.5, 48.9) are a tie.
What moved the number: training on corrections of the model's own bad turns took the held-out figure from 37.6% (pass 5) to 50.3% (pass 6). A second round gave nothing. DPO gave nothing.
From "What remains in the released chat model" and the manual-chat findings:
The template is Llama 3 style and ships in chat_template.jinja and tokenizer_config.json (source: training/finalize_hf.py and the training template in sft/build_sft_v2.py). Training used one <|begin_of_text|> at the start of the conversation only, then each message as <|start_header_id|>ROLE<|end_header_id|>\n\n + content + <|eot_id|>, and the assistant turn is generated after <|start_header_id|>assistant<|end_header_id|>\n\n. The end-of-turn token is <|eot_id|> (128009).
Roles: system, user, assistant, and tool for tool results. The template has no default system prompt. In training some examples had a system prompt like "You are Kisoku, a helpful AI assistant created by 0ARCH." and many had none. No default system prompt is required; the model was trained both with and without one.
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
repo = "0arch-io/kisoku-1.6b-chat"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype=torch.bfloat16, device_map="auto")
messages = [{"role": "user", "content": "hey, can you help me write a short bio?"}]
ids = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=300, do_sample=True, temperature=0.7, top_p=0.9)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))
generation_config.json sets sampling on with temperature 0.7 and top-p 0.9 and stops on 128009 and 128001. It sets no repetition penalty, and you should not add one.
Tool use (trained, lightly tested): put the tool list as JSON inside <tools></tools> in the system message, with the instruction that calls are a JSON object inside <tool_call></tool_call> and results come back inside <tool_response></tool_response> in a tool message. The exact system prompt text used in training is TOOL_SYSTEM in sft/build_sft_v2.py in the code release.
The config follows the base model: trained to 64K, YaRN factor 2 for about 128K. See the base card for the exact config and how to turn it off. This repo's config.json carries the same YaRN x2 setting, with config.plain.json beside it.
Builds for llama.cpp and Ollama are in 0arch-io/kisoku-1.6b-gguf. F16, Q8_0 and Q4_K_M are there for both the base and this chat model.
This card's failure list above is the main one. Also: English-focused, one evaluation setup (one harness version, bfloat16, my prompt settings), the judge in the conversation test is a language model that I did not calibrate, and the model has had no safety or red-team evaluation. Do not use it for anything where a wrong answer matters.
Licence: Apache 2.0. The tokenizer is the Llama 3 vocabulary; see the base card's licence note. Teacher-written data came from DeepSeek-V4.1-Flash.
@misc{rodriguez2026kisoku,
title = {Kisoku 1.6B: A Solo, From-Scratch Pretrain on a Free TPU Grant},
author = {Rodriguez, Joseph},
year = {2026},
howpublished = {\url{https://huggingface.co/0arch-io/kisoku-1.6b-chat}}
}
This is a preview, not a finished assistant. It is the base model 0arch-io/kisoku-1.6b after nine supervised fine-tuning passes. This repo is pass 9 (the pass labelled sft009 in my run names). It loses the thread in longer conversations, is behind Llama 3.2 1B Instruct on a multi-turn test, and states wrong answers confidently. Read "Known failures" before using it.
All numbers come from the technical report draft (draft 3, section 7B and 7) and were measured by me.
Chat fine-tuning ran from 2026-10-04 to 2026-10-06 on the TPU v4-32, nine supervised passes of about 70 to 95 minutes each (about 131K tokens per step). Every pass started from the final long-context base checkpoint (Phase C, step 1299) with a fresh optimizer. No pass continued from an earlier chat model. One DPO experiment was run and not shipped.
Training data (every example containing a benchmark test item was dropped by verbatim matching, and a rescan of the final shards found zero hits):
Thinking mode was trained in pass 3 and removed. "Hold your ground" examples were removed (the model could not tell its right answers from wrong ones).
Base-style tests, no chat template, same harness as the base card (lm-evaluation-harness 0.4.13, bfloat16, RTX 4090). The first base column is the stage 3 checkpoint, before long-context training; the second is the released base, which the chat model was tuned from.
| Test | Base (stage 3) | Base (released) | Chat, pass 9 |
|---|---|---|---|
| GSM8K | 15.3 | 15.0 | 20.6 |
| HumanEval | 13.4 | 14.6 | 14.0 |
| ARC-Easy | 65.4 | 64.4 | 66.0 |
| ARC-Challenge | 39.8 | 40.1 | 42.2 |
| WinoGrande | 57.3 | 56.8 | 57.2 |
| PIQA | 73.9 | 73.0 | 73.0 |
| HellaSwag | 57.7 | 57.8 | 57.2 |
| TriviaQA | 22.8 | 23.1 | 20.8 |
| MMLU | 33.0 | 34.2 | 29.8 |
| BBH | 29.2 | 28.6 | 26.3 |
Against the released base, math went up by 5 points, MMLU down by 4, BBH by 2, and the rest moved by about a point or less. No confidence intervals.
| Model | 4K | 32K | 64K |
|---|---|---|---|
| Kisoku chat, pass 9 | 76.5 | 60.5 | 54.3 |
| Kisoku base, final | 76.1 | 58.8 | 56.7 |
| Llama 3.2 1B (base) | 73.5 | 56.7 | 49.2 |
Single runs of 650 samples, no confidence interval. The chat model keeps most of the base long-context ability. The Phase C mix included synthetic tasks resembling RULER, so see the base card's caveat. The chat model was run at these three lengths only.
After pass 5 failed a real nine-turn conversation, I built a test that is not a replay. A teacher model wrote 2,396 conversation scripts of up to nine user turns that make sense whatever the assistant answers (small talk, a casual build request, "make it shorter", "that's not what I asked", a change of topic and a return). The chat model answers every turn itself. The teacher then reads the transcript once and marks each assistant turn acceptable or not. One script in eight is held out by a hash of its id (297 scripts, about 2,530 turns) and never trained on.
| Model on the same 297 held-out scripts | Turns acceptable | Bad first turns |
|---|---|---|
| Llama 3.1 8B Instruct | 59.5% | 10% |
| Llama 3.2 1B Instruct | 56.7% | 11% |
| Kisoku chat, pass 6 (best of mine) | 50.3% | 8% |
| Kisoku chat, pass 9 (this model) | 48.9% | 10% |
The Llama models were each given a system prompt saying they are Kisoku (with Kisoku's facts and style) so the judge would not penalize their own names. They still lost some turns by calling themselves "a large language model", so their true figures are a little higher.
Kisoku chat is 6 to 8 points behind Llama 3.2 1B Instruct on multi-turn conversation. Llama 3.2 1B was distilled from the 8B and 70B models and post-trained by a large team. This model is nine passes by one person over three days on a base trained on about a sixth of the data. The judge is strict for everyone (the 8B model clears only 60%), and almost no conversation gets through all nine turns with every turn acceptable (0 to 2 of 297 for every model). I have not measured how much the judge's marks vary between two readings of the same transcript. Passes 6 to 9 (50.3, 49.5, 49.5, 48.9) are a tie.
What moved the number: training on corrections of the model's own bad turns took the held-out figure from 37.6% (pass 5) to 50.3% (pass 6). A second round gave nothing. DPO gave nothing.
From "What remains in the released chat model" and the manual-chat findings:
The template is Llama 3 style and ships in chat_template.jinja and tokenizer_config.json (source: training/finalize_hf.py and the training template in sft/build_sft_v2.py). Training used one <|begin_of_text|> at the start of the conversation only, then each message as <|start_header_id|>ROLE<|end_header_id|>\n\n + content + <|eot_id|>, and the assistant turn is generated after <|start_header_id|>assistant<|end_header_id|>\n\n. The end-of-turn token is <|eot_id|> (128009).
Roles: system, user, assistant, and tool for tool results. The template has no default system prompt. In training some examples had a system prompt like "You are Kisoku, a helpful AI assistant created by 0ARCH." and many had none. No default system prompt is required; the model was trained both with and without one.
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
repo = "0arch-io/kisoku-1.6b-chat"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype=torch.bfloat16, device_map="auto")
messages = [{"role": "user", "content": "hey, can you help me write a short bio?"}]
ids = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=300, do_sample=True, temperature=0.7, top_p=0.9)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))
generation_config.json sets sampling on with temperature 0.7 and top-p 0.9 and stops on 128009 and 128001. It sets no repetition penalty, and you should not add one.
Tool use (trained, lightly tested): put the tool list as JSON inside <tools></tools> in the system message, with the instruction that calls are a JSON object inside <tool_call></tool_call> and results come back inside <tool_response></tool_response> in a tool message. The exact system prompt text used in training is TOOL_SYSTEM in sft/build_sft_v2.py in the code release.
The config follows the base model: trained to 64K, YaRN factor 2 for about 128K. See the base card for the exact config and how to turn it off. This repo's config.json carries the same YaRN x2 setting, with config.plain.json beside it.
Builds for llama.cpp and Ollama are in 0arch-io/kisoku-1.6b-gguf. F16, Q8_0 and Q4_K_M are there for both the base and this chat model.
This card's failure list above is the main one. Also: English-focused, one evaluation setup (one harness version, bfloat16, my prompt settings), the judge in the conversation test is a language model that I did not calibrate, and the model has had no safety or red-team evaluation. Do not use it for anything where a wrong answer matters.
Licence: Apache 2.0. The tokenizer is the Llama 3 vocabulary; see the base card's licence note. Teacher-written data came from DeepSeek-V4.1-Flash.
@misc{rodriguez2026kisoku,
title = {Kisoku 1.6B: A Solo, From-Scratch Pretrain on a Free TPU Grant},
author = {Rodriguez, Joseph},
year = {2026},
howpublished = {\url{https://huggingface.co/0arch-io/kisoku-1.6b-chat}}
}