halogen-flash-server

halogen™ is the fastest way to run Qwen3.8-Flash-Next on AMD Strix Halo, and it does not get there by spending fewer bits.

Every kernel is written for this one GPU and this one model family. No general-purpose runtime, no portability layer, no fallback path. That is why it can do things a general engine cannot, and why it runs on exactly one piece of silicon.

On a 32K prompt with a 256-token answer, against the fastest numbers anyone else has published for this model on this hardware:

Roughly 4x faster end to end than the best of them. Prefill is where that is won, and on any prompt with real context prefill is most of the wall clock. The one runtime carrying more bits than we do is the slowest of the three, and the fastest of them runs at 3.71 bpw, two thirds of our precision.

Our two cells are the rows published under Measured, which is also where the conditions are: 32,768 tokens at 1,424 tok/s, then 256 tokens at the served speculative rate of 41.7 tok/s. Read those conditions before comparing, particularly the power envelope. The competitor rows are their own published figures on their own machines, and Against the alternatives says what differs.

Bits per weight is measured from the checkpoint's own tensor table rather than quoted from a format name. It is 5.53 bpw across all 179.55B parameters, or 4.55 bpw across the trunk and experts with the FP8 n-gram lookup table set aside. docs/QUANT.md gives the breakdown by tensor family and says how the figure is derived, so it can be checked with arithmetic rather than taken on trust.

On the decode column, which is the soft one. Those are the published figures at this depth, and for two of the three we cannot tell whether speculative decoding was on. EngramHalo's 14.3 s is explicitly its non-speculative number; its speculative rate at 32K is not published, and interpolating its own curve suggests something nearer 9 s. Hand every competitor its best plausible speculative decode and the totals still land around 110 s against our 31.1 s. The prefill column is the one carrying the claim, and it has no such ambiguity.

At temperature 0, output is byte-identical to serial greedy decode. Speculation here is a pure speed optimization, verified on every release, not a quality trade. Since 0.6.0 there are two draft sources, the model's own draft head and the request's own text (prompt lookup), and the guarantee covers both.

Since 0.7.0 the engine also opens a llama.cpp GGUF of this model directly: point it at the file you already have (unsloth's UD-IQ4_XS, say) and it runs on these kernels, with the same speculation and the same identity guarantee. Same file, faster runtime, no conversion step. See Bring your own GGUF for which files, and for the numbers.

---

Contents

• Quickstart, then Using it: sampling, images, token budgets, Codex and the Responses API, from an agent harness
• Give it a machine of its own: what this server holds, and what that leaves for anything else
• Measured: prefill and decode, against the alternatives, and end to end over HTTP
• Quality: what is measured, and what is not
• Precision: what you get, and how to trade it
• Bring your own GGUF: run a llama.cpp file of this model on this engine, and what that costs and buys
• Configuration: every setting worth knowing, plus cache modes, context and memory and 1M context, attention budget, composable context
• Troubleshooting: will not start, starts but crawls, the host settings we measured on
• What this release is not, and the license
• AGENTS.md: the short form of all of the above for an AI agent deploying, driving or debugging this server

---

Quickstart

podman run --rm -p 8731:8731 \
  --device /dev/kfd --device /dev/dri --group-add keep-groups \
  --ipc=host --ulimit memlock=-1:-1 \
  -e HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next \
  -v ~/halogen-models:/models \
  ghcr.io/peonist-ai/halogen-flash-server:0.11.8

That is the whole thing. It fetches the weights on first start (118 GiB, so give it a while; the transfer resumes if interrupted) and serves an OpenAI-compatible endpoint on :8731, reachable from your network.

Note the models volume is read-write here, with no :ro, because it is being downloaded into. Nothing is fetched on later starts, with one exception: a start with HALOGEN_DOWNLOAD set and the volume writable re-fetches the 2.4 GiB quality sidecar when the one on disk predates the image (0.6.0 changed that file; the 115 GiB checkpoint is never re-fetched). With HALOGEN_DOWNLOAD unset the container opens no outbound connections at all, and says at startup if the sidecar is the older one.

If you would rather fetch the weights yourself:

hf download peonist-ai/halogen-qwen3.8-flash-next --local-dir ~/halogen-models

podman run --rm -p 8731:8731 \
  --device /dev/kfd --device /dev/dri --group-add keep-groups \
  --ipc=host --ulimit memlock=-1:-1 \
  -v ~/halogen-models:/models:ro \
  ghcr.io/peonist-ai/halogen-flash-server:0.11.8

The weights repo carries the tokenizer, so one -v is all either form needs. On Docker rather than Podman, replace --group-add keep-groups with --group-add video --group-add render: keep-groups is a Podman extension.

If you split the engine and the API into two containers (the shipped docker-compose.yml does), run both from the same image tag. The API renders the prompt and the engine runs it, and what one release can do the other may not know how to ask for: an API from before 0.5.0 in front of a newer engine sends an image as a placeholder with no pixels behind it, and the model describes a picture it never received. Since 0.5.8 the engine refuses that, each container prints its version on its first log line, the API warns at startup when the engine's differs, and /health reports both under version. (The text <|image_pad|> written in a message is not a placeholder and, since 0.6.2, is served as text; see #39.)

---

Using it

The server speaks the OpenAI API at /v1, and /health is the authoritative account of what the build you are running supports: the sampling fields, whether images are accepted, the token budget aliases and the current default, and the tool-call wire format. What follows is the part worth reading first.

Sampling

temperature, top_p, top_k, min_p, seed, presence_penalty, frequency_penalty, logit_bias and logprobs are supported. temperature absent or 0 is greedy decode. Above 0, the request samples from the filtered distribution on the same drafter it would otherwise get, so speculation stays on. A seed reproduces a request on the same server configuration. top_logprobs, logprobs with stream: true and n > 1 are not implemented and are refused with a 400, as is any value outside its defined range, rather than clamped. /health lists what the running build supports.

Server-side defaults, and the model card's settings. This image decodes greedy unless a request says otherwise, because greedy is what every byte-identical guarantee below is made on. The model's authors recommend sampling: the card's thinking-mode settings are temperature=1.0, top_p=0.95, top_k=20, and every benchmark in it was run at that point. Most agent clients send no sampling fields at all, so the server can supply them:

-e HALOGEN_TEMPERATURE=1.0 -e HALOGEN_TOP_P=0.95 -e HALOGEN_TOP_K=20

HALOGEN_TEMPERATURE, HALOGEN_TOP_P, HALOGEN_TOP_K, HALOGEN_MIN_P, HALOGEN_PRESENCE_PENALTY and HALOGEN_FREQUENCY_PENALTY each set the value a request gets when it omits that field. The rule is one sentence: a field the request sends always wins, a default fills only a field the request omits, and a request that sends temperature: 0 decodes greedy and takes none of the sampling defaults. With a temperature default set, a request that sends no temperature is sampled, so its output differs run to run unless it sends a seed, and the byte-identical claims below apply only to requests that send temperature: 0. That is why the image does not set these itself: it is your call which default you want, and this is the switch. /health reports what is set under server_defaults, and a value outside its range refuses to start, before the model loads, naming the variable. (The card's non-thinking settings are temperature=0.7, top_p=0.80, top_k=20, presence_penalty=1.5; they apply when a request disables thinking, which these defaults cannot tell apart, so send them from the client in that case.)

Images

The server reads images, and it is off until you turn it on. Point HALOGEN_VISION_TOWER at the vision sidecar that ships beside the weights, or set it to 1 to look for the file next to the checkpoint:

-e HALOGEN_VISION_TOWER=1

With no tower the image path is absent rather than disabled, so a text-only deployment behaves exactly as it did before this release. /health reports whether images are accepted and, when they are not, why; an image sent to a server without a tower is a 400 naming the flag.

Both /v1/chat/completions and /v1/responses take an image content part in the usual OpenAI shape, carrying a data: URL or bare base64:

{"role": "user", "content": [
  {"type": "text", "text": "What does the error message say?"},
  {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}
]}

An http(s) URL is refused deliberately: fetching one would make the server issue outbound requests to wherever a client pointed it. Several images in one conversation are attributed correctly, including an earlier one referred to after a later one has arrived.

What to expect from it. Text at 12 pt and above is read exactly at every supported resolution. Below that it degrades gradually rather than failing: across a battery of several hundred readings every miss was the right field with one to three characters wrong, and none read a different field or invented a value. Two things are worth knowing when you choose what to send. A bigger frame is not better for the same text, because past a point it adds empty area and not detail. And a densely filled page is harder than a sparse one at the same point size, which is a matter of finding the right row rather than resolving it.

What it costs. One image adds roughly 5.5, 11.8 or 25.3 seconds at 1280x800, 1920x1080 or 2560x1440. HALOGEN_VISION_MAX_PIXELS (default 2560x1440) is the size an image is scaled down to fit, preserving aspect ratio; larger images are downscaled rather than refused, and nothing is refused until four times that. 3840x2160 costs about 105 seconds and reads no better than 1440p, which is why the default sits where it does. There is no fixed aspect ratio anywhere in the path: tall, wide and square crops all work, and a crop under 256x256 is scaled up, which helps small text rather than hurting it. A 1920x1080 frame occupies about 2,040 tokens of the context.

Token budgets, and why an empty answer means you ran out

The token budget covers thinking, not just the answer. This model reasons before it replies and those tokens count against the budget. Before 0.11.0 a budget that ran out mid-thought did not shorten the answer, it removed it: the reply came back with finish_reason: "length", an empty content, and the partial reasoning in reasoning_content, which most OpenAI clients do not display. Since 0.11.0 the server closes the think block with room left for the answer (the answer room, below), so that shape needs a request that asks for it (HALOGEN_THINKING_ANSWER_ROOM=0).

The default is 8192, which finished every ordinary prompt we measured with room to spare. Send more when you want more, up to HALOGEN_MAX_TOKENS_CAP (65536 by default); above the cap you get a 400 rather than a silent truncation, so ask for what you need and the server will tell you if it is too much. Hard reasoning problems can genuinely exceed 8192: pass a larger budget, or "reasoning_effort": "low" to make the model think less. Accepted efforts are minimal, low, medium, high and xhigh; the model's own default is xhigh. The chat template itself knows three levels, so the five names fold onto them: minimal and low are low, high and xhigh are xhigh, and the startup line and /health report the level the template will see (#71 asked why high printed as xhigh). medium is the one step down from the default. "reasoning_effort": "none" turns thinking off for that request (the same as chat_template_kwargs: {"enable_thinking": false}, and reasoning: {"effort": "none"} on /v1/responses).

A thinking budget, since 0.8.1. "max_thinking_tokens": N on a request (or HALOGEN_MAX_THINKING_TOKENS as the server default; the request wins) bounds the think block: if the model has not closed it after N generated tokens, the server closes it (Qwen's own budget sentence, then </think>) and the answer follows in the same stream, on the same state, with no second request. The max_tokens budget still covers both. This exists because greedy decoding at 100k+ of context can loop inside the block and spend the whole budget there (issue #56: 32,000 tokens of reasoning and an empty answer); the model card's sampling settings above are the cure, and the budget bounds the damage when a client sends none. Unset, nothing changes.

The answer room, since 0.11.0. Thinking no longer consumes the whole budget. When a request sends no thinking budget of its own, the server closes the think block once max(1024, 15% of max_tokens) tokens of the budget remain (the same close as max_thinking_tokens), so a capped request ends with an answer rather than finish_reason: "length" and an empty content. This matters because agent harnesses send no thinking control at all to an OpenAI-compatible server unless configured to, so the model's own xhigh runs under whatever cap the harness set for the answer: a compaction summary capped at 13,000 tokens that the model thinks past is a compaction that fails, and the harness retries it. A request's own budget still wins when it is smaller. HALOGEN_THINKING_ANSWER_ROOM sets the room in tokens; 0 restores the 0.10.x behaviour; /health reports it as thinking_answer_room. Only a request whose thinking would have run past the line is affected; every other reply is untouched.

Your harness's own name for the thinking controls works, since 0.11.0. Besides reasoning_effort, enable_thinking, chat_template_kwargs and max_thinking_tokens, the chat route reads thinking_budget_tokens, thinking_budget and thinking_token_budget (the three names Pi's compat.thinkingTokenBudgetField can send), the reasoning object ({"enabled": false}, {"effort": "low"}, {"max_tokens": 4096}: the OpenRouter shape, which hermes-agent and aider send) and the thinking object ({"type": "disabled"}, {"type": "enabled", "budget_tokens": 4096}: the Anthropic shape, which aider's --thinking-tokens sends to every non-OpenRouter model). Two names with two different values is a 400, as with the token budget. /health lists them under supported.

Any of three field names works, and they mean the same thing here: max_completion_tokens (current OpenAI Chat Completions), max_output_tokens (OpenAI Responses), or max_tokens (deprecated upstream, still widely sent). Send one, or send several as long as they agree; two different values is a 400 rather than a guess about which you meant. /health lists all three under token_budget_aliases and reports the current default as max_tokens_default. HALOGEN_MAX_TOKENS_DEFAULT moves that default for every route. The card's advice is not to cap the budget at all; here a request reserves its prompt plus its budget in the KV pool when it is admitted, so a large default costs concurrency (four slots at 65,536 is a whole 262,144-position pool before a single prompt token). -e HALOGEN_MAX_TOKENS_DEFAULT=16384 is the step that clears an ordinary agentic turn's reasoning without that cost. HALOGEN_REASONING_EFFORT moves the effort a request gets when it names none, and the card's own guidance is to leave it at xhigh: lower effort on multi-turn agentic tasks "can lead to insufficient analysis, more failures, and repeated retries." A request that sends either field still wins.

{
  "model": "halogen-qwen3.8-flash-next",
  "messages": [{"role": "user", "content": "..."}],
  "max_completion_tokens": 16384,
  "reasoning_effort": "low"
}

If a reply looks empty or cut off, read finish_reason first: "stop" means you have the whole answer, "length" means you ran out of budget.

Codex and the Responses API

The server also speaks the OpenAI Responses API at POST /v1/responses, so clients that dropped Chat Completions can use it directly. The OpenAI Codex CLI is the reason it exists: point it at this server and it works, including tool calls.

# ~/.codex/config.toml
model = "halogen-qwen3.8-flash-next"
model_provider = "halogen"

[model_providers.halogen]
name = "halogen"
base_url = "http://<your-server>:8731/v1"
wire_api = "responses"
requires_openai_auth = false

Streaming and non-streaming both work, function_call and function_call_output round trip, and tools entries that are not functions (web_search, and the namespace wrapper, whose nested functions are used) are ignored rather than rejected. instructions and any developer turns are folded into the system prompt.

Reasoning is returned (since 0.7.0; #44). The model's thinking goes out as a reasoning output item ahead of the message, its text as a reasoning_text content part streamed in response.reasoning_text.delta events, and usage.output_tokens_details.reasoning_tokens says how much of the output it was (both routes carry that count). When the request asks for a reasoning summary, as Codex does (reasoning: {"summary": "auto"}), the same text is sent again as the item's summary_text, with the response.reasoning_summary_* events: there is no separate summarizer, the summary is the reasoning. Codex renders summaries by default and shows raw reasoning content only with show_raw_agent_reasoning = true in its config, so with that setting on you will see the text twice. No encrypted_content is sent, and a reasoning item echoed back in a later turn is dropped, as before. There is no response store, so previous_response_id, retrieving a response by id, and cancelling one are not available; send the history with each request, which is what Codex does.

Statistics for llama-swap (since 0.7.0; #45): every response, on both routes, carries a timings object in llama-server's shape (prompt_n, predicted_n, prompt_ms, predicted_ms, prompt_per_second, predicted_per_second, cache_n, draft_n, draft_n_accepted), on the non-streamed body and on a stream's last frames, so llama-swap's activity page shows prefill and decode rates and the draft count. draft_n counts the draft head's proposals and the prompt-lookup chains' together; the numbers are the engine's own per-request line, copied.

Prometheus (since 0.8.0): GET /metrics answers in llama-server's metric names, so a dashboard built for llama.cpp (or for llama-swap's upstream) reads this server unchanged: llamacpp:prompt_tokens_total, llamacpp:tokens_predicted_total and their _seconds_total counters (the engine's own per-request numbers, summed; prompt_n is the processed count), the prompt_tokens_seconds / predicted_tokens_seconds gauges over the requests since the last scrape, requests_processing, requests_deferred, kv_cache_tokens and kv_cache_usage_ratio (since 0.11.5 the engine's own occupancy: the positions its pool holds in every region, busy and held, over the pool, as of the last completed request; before 0.11.5 they counted what the requests holding a front-end slot had asked for, admitted or not, which read 913k against a 655k pool in issue #74). Beside them, halogen:requests_total, halogen:prompt_tokens_cached_total, halogen:draft_tokens_total, halogen:draft_tokens_accepted_total, halogen:structured_requests_total, and since 0.11.5 halogen:kv_pool_positions and halogen:kv_pool_reserved_tokens (the front end's reservation, the old meaning). Always on, no flag, no engine round trip.

Cache and pool occupancy in the log (since 0.11.5; #73): every request's serve_api: line carries the cached share of its prompt, the prefill rate over the tokens actually processed, and the pool's occupancy as the engine reports it: prompt 59498 (58013 cached, 97.5%), prefill 2.29s = 648 t/s | ... | pool 412224/655360 63%. GET /cache adds token_hit_rate (prompt tokens the cache covered over every prompt token seen since the server started; hit_rate counts requests) and, under pool, positions, used, usage_ratio, busy_regions (a request decoding), held_regions (a warm conversation between turns, not a full pool), room_clamped and moved.

Structured output (since 0.8.0; #14, #43). response_format: {"type": "json_schema", "json_schema": {"name": ..., "schema": {...}}} and {"type": "json_object"} on /v1/chat/completions and /v1/completions, text: {"format": {"type": "json_schema", "name": ..., "schema": {...}}} and {"type": "json_object"} on /v1/responses (the shape Codex sends; its approvals reviewer sends one on every auto-reviewed tool call, which is what #43 hit). The engine enforces the schema while it decodes: every token is chosen from the tokens the schema allows next, so the reply parses and validates by construction, with no retry and no repair. It is greedy decoding under a mask, nothing else changes: the same request without the schema is bitwise what it was, and a constrained request is identical whether it decoded serially, with the draft head, with prompt lookup, or beside other requests. It costs nothing measurable: the schema is compiled once (a millisecond) and every state's mask lives on the GPU, so the decode loop reads a pointer.

What is enforced: type (object, array, string, number, integer, boolean, null, and ["string", "null"]), properties with keys emitted in schema order, required (an optional key may be skipped), additionalProperties (false, true, or a schema: extra keys only after the listed ones), items, minItems, maxItems, minLength, maxLength, enum, const, anyOf (oneOf is treated as anyOf), $ref and $defs including recursive ones. Accepted and not enforced: minimum, maximum, exclusiveMinimum, exclusiveMaximum, multipleOf, and the annotation keywords. Refused with a 400 naming the keyword: pattern, format, allOf, not, if/then/else, patternProperties, dependentRequired, dependentSchemas, uniqueItems, contains, propertyNames, unevaluated*, minProperties, maxProperties, prefixItems. /health lists all three under structured_output.

The reasoning block is not constrained: with thinking on the model thinks freely and the JSON starts after </think>. A request with tools may open a tool call instead of the JSON (the schema binds the final text, not a call), which is how the Codex reviewer's own read-only tool checks keep working. Whitespace between tokens is allowed, so a pretty-printed reply is fine, but at most two whitespace-only tokens in a row (a forbidden preference otherwise falls to a newline, and then another). Only the end-of-turn token is legal once the value is complete, so the reply is exactly the JSON. Temperature must be 0 (or omitted): a sampled request with a schema is a 400 for now, and so is a schema with an image. HALOGEN_GRAMMAR=0 turns the feature off (the 400 of 0.7.0 comes back).

Verified against the Codex CLI driving real tasks end to end, and separately against the official openai Python SDK, which parses every event into its own typed models.

---

From an agent harness

Every coding-agent harness we read (Pi, opencode, Codex CLI, hermes-agent, Cline, Roo Code, aider, oh-my-pi) follows the same sensible rule against an OpenAI-compatible server it was not written for: send nothing the server was not declared to accept. So none of them sends a thinking control unless you configure one, and most cannot express "thinking off" at all against a custom endpoint. Two consequences: the server's defaults are what your harness runs at, and a compaction (which six of the seven build as a new conversation, a cold prefill of the whole history under the harness's own output cap) runs at the model's xhigh effort. The answer room keeps that from failing; these are the switches if you want it faster:

The server variables are HALOGEN_REASONING_EFFORT (the effort a request gets when it names none; the card's advice is to leave it at xhigh for agentic work), HALOGEN_ENABLE_THINKING=0 (thinking off unless a request turns it on) and HALOGEN_MAX_THINKING_TOKENS (a budget for requests that send none). A request that names any of these wins over the variable.

What the log says during a long turn, since 0.11.0. The container log prints flash_serve: req N prefill P/T tokens, S s at every 32,768-token chunk of a long prompt and every 20 s inside one, and req N generated K tokens, S s every 30 s of a long answer, so a 160k-token compaction that takes minutes is visible while it runs rather than only in the serve_api: line at its end. /health reports busy_for_s while a request is in flight.

Give it a machine of its own

This server holds most of the host once it is loaded: the weights stay resident and the KV pool is reserved up front. On a 128 GB machine that leaves roughly twelve gigabytes free, and very little of it in the large contiguous pieces that another big process needs in order to start or to grow.

If you run application containers, a database, or another model on the same machine, they compete for what is left. When it runs out, allocations do not fail cleanly: the kernel goes looking for contiguous memory it cannot find, and whatever asked for it, including this server, can stop for minutes at a time at 100% of one core with no disk activity and no output. It is not a crash, it needs no restart, and it looks exactly like a hang.

The startup line says how much room is left, and a second line says why your own tools will disagree:

startup [   4.9 s] host memory left for everything else: 465 contiguous 2 MiB
                   blocks (12.4 GiB total, most of it not contiguous)
startup [   4.9 s] free(1) and MemAvailable will report about 80.4 GiB
                   instead: the kernel counts this server's locked weights as
                   reclaimable file cache, and they cannot be reclaimed

Believe the first line. free, MemAvailable, and every monitoring tool that reads them, count this server's locked weights as reclaimable page cache, so they overstate the memory available on this host by the size of the model, about 68 GiB. No kernel field reports the difference (Mlocked and Unevictable both stay at zero across the load), which is why the server has to print the correction itself.

A few hundred blocks is normal for this server and is fine on a host of its own. If that number is small and you have other work on the machine, expect the above. Options, in the order worth trying:

• Give it its own machine. This is the honest answer for a server that holds this much of one.
• Lower HALOGEN_KV_POOL_POSITIONS. Fewer conversations stay resident at once; each one's speed and its answers are unchanged.
• HALOGEN_FLASH_PIN_TRUNK=0 gives a great deal of memory back and costs several times the decode speed. It is a last resort, not a tuning option.

Compacting memory afterwards does not help, because the memory this server holds cannot be moved. If you need to reclaim it, stop the server.

---

Measured

Conditions, because they change the numbers: AMD Ryzen AI Max+ 395 (Radeon 8060S, gfx1151), 128 GB unified memory, ROCm 7.14.0, and about 85 W of sustained package power, sampled from sysfs during a 32,768-token prefill alongside a 2,229 MHz median clock against the part's 2,900 MHz top state.

The IOMMU is off on the reference machine, and it is worth 13 to 16 percent of prefill. Prefill is compute-bound, and on this hardware an enabled IOMMU is a power-budget tax rather than a memory-path one: with iommu=pt we measured the SoC drawing more power (122 to 127 W against 108 to 118) for lower shader clocks (2,357 to 2,409 MHz against 2,549 to 2,713) at the same temperature, and prefill fell from 460 to 385 tok/s at 2,048 tokens while every bandwidth-bound number held exactly. Bisected on one kernel, so it is the IOMMU and not the kernel version. We have not measured the IOMMU in translated mode, only off against passthrough, and this is one machine.

The full kernel command line this was measured on is published under The host settings these numbers were measured on, because numbers you cannot reproduce are not much use.

Match the power envelope before comparing decode numbers. It is the condition most easily left out and it moves these rows: an independent tester running a 70 W-limited handheld measured 11 to 12 percent under both the serial and the drafted figure below, consistently on both, which is the signature of a lower envelope rather than a disagreement about the engine. Prefill reproduced on that same machine.

The shipped checkpoint and its quality sidecar, in the image's default configuration: full 262,144 context, prompt cache on, tuned GEMM plan loaded. Prefill is a cold single-call prefill of real text; decode is greedy at temperature 0. Prefill is measured by the engine's own prefill bench; a served request with the default speculative drafter pays about 2-3% more time-to-first-token, because the draft head prefills too. The prefill rows are 0.5.3's measurements. The control was this same binary with the previous release's ordering step selected, so the two arms differ in one thing and nothing else; it ran in the same session, on the plan this image bakes, and it reproduced the rows it replaces to within 1.4%. The serial decode rows are 0.2.0's and have not moved since: the releases between them changed the scheduler, the memory layout and one host-side sort, not the decode kernels. 0.6.0 moves the speculative rows twice, and both moves are draft-side: the sidecar now carries the draft head's own projections at 8 bits (its proposals are accepted more often), and the request's own text is a second draft source (the rows below the served one).

The 0.6.0 rows are agent turns, not prose. Each is the mean over six prompts: a real coding-agent conversation (SWE-agent trajectories over real repositories, driven by another model) or a function-calling dialogue, cut at the start of an assistant turn, ~1,000–1,800 tokens of context, 400 tokens generated, greedy. Prompt lookup drafts from the request's own text: when the last three tokens of the answer already occur earlier in the conversation, the three that followed are proposed as a chain and verified in one step, with the draft head's own proposal opening the chain. On a coding turn about half the generated tokens are such copies (tool-call arguments, paths, code quoting the file being edited), thinking on or off, which is why the gain over the head alone is 13–15% there, 6–9% on function-calling turns, and within noise on prose and code text (the head already takes what there is). Serial on the same prompts is 36.8 tok/s, so a coding-agent turn decodes at about 1.5x serial. Every one of those runs produced the serial run's tokens exactly. Like the draft head, prompt lookup runs while the request is the only one generating; with several conversations generating at once the scheduler batches them instead (the concurrency table below is unchanged by it).

Decode barely moves with depth. Serial gives up about 7% going from 1,500 to 32,768 tokens of context, a 22x increase. The 32,768 served figure is the one to compare against other runtimes' depth curves, and it is measured through the full HTTP stack rather than on a raw token fixture, which is the harder condition.

Two levers move these and both are one environment variable:

• A tuned GEMM plan ships in the image and is on by default. The matrix library exposes many kernels per shape, and the image carries choices measured on this hardware rather than picking at runtime (HALOGEN_MATMUL_TUNING_FILE). It costs nothing in quality: paired perplexity over 32,767 positions differs by 0.0006 nats, a confidence interval spanning zero. It is also deterministic, since every process reads the same decisions, so the same prompt keeps giving the same answer.

• The prompt cache is ON by default, which is what makes the native context usable in practice. A session whose prompt grows, whether an agent, a chat, or a document you keep asking about, does not re-read its shared prefix. Only the tokens you actually added get processed:

  	first turn	every turn after
  100,000-token conversation	~88 s	~2 s
  10,000-token conversation	~9 s	~1.4 s

  The follow-up cost is flat. It does not grow as the conversation does, because it depends on how much you added, not on how much is already there. Measured over a 20-turn session growing to 108,000 tokens, every turn after the first landed between 2.0 and 2.3 s. See Choosing a cache mode for when to change it.

Against the alternatives

Three other runtimes publish figures for this model on this hardware. All are llama.cpp derivatives or forks of one.

The shape matters more than the ratio. Every one of them decays hard with depth. Ours does not: 1,246 at 8K, 1,424 at 32K, 1,358 at 131K. Their own documentation puts it plainly enough. A 156K prompt takes EngramHalo about twelve minutes. We prefill 131K in 96 seconds.

Decode is the closer row. Against the fastest of them we are roughly 1.2x on code and 1.7x on prose at short context, and the comparison at depth is muddied by their speculative numbers mostly not being published.

These are published figures, not a head-to-head we ran. Every number in the competitor columns is from their own model card or repository, on their machine, at their quantization and their settings. We have not run their builds. Their conditions differ from ours in ways that matter: EngramHalo measures on a 96 GB machine rather than 128 GB, runs a q8_0 KV cache, and quantizes the n-gram lookup table harder than we do, to 26.8 GiB against our 47.7 GiB. Keeping that table on disk is not one of the differences: we do the same, by default and with no way to turn it off. Treat the prefill gap as real and the decode rows as indicative.

Served throughput, end to end over HTTP

The prefill numbers above are the engine's own prefill bench. Through the full stack of chat template, tokenizer, HTTP and SSE, the image's own sweep mode measures 812 tok/s at pp2048 and 1,041 at pp8192, and bench over ten real prompt shapes measures 45.3 tok/s mean with speculation on the 0.6.0 image with its sidecar (min 39.5 on chat, max 49.4 on procedural text; 1.63 tokens committed per round; the 0.3.0 image read 43.6 on the same instrument, and the difference is the draft head's 8-bit projections: these short prompts give prompt lookup one to eight rounds a case). Acceptance depends on how predictable the text is, so quote the mean with the prompt set named, never a single shape.

That run also re-checks the identity property on live traffic: every drafter produced byte-identical output on every case.

Reproduce the numbers with the benchmarks baked into the image:

podman run ... ghcr.io/peonist-ai/halogen-flash-server:0.11.8 bench serial,mtp 256 low 3
podman run ... ghcr.io/peonist-ai/halogen-flash-server:0.11.8 sweep -p 8192,32768 -n 128

---

Quality: what is measured, and what is not

Speed claims are cheap. These are the checks behind them.

Token-for-token against transformers. Six real prompts, 32 greedy steps each, teacher-forced against goldens dumped from HuggingFace transformers running the original BF16 weights: 182 of 192 steps identical, two of the six prompts perfect. That figure is END-TO-END. It includes everything 4-bit quantization costs, not only the engine. The engine's own share is measured separately, against a reference run on the same dequantized weights, and is the smaller half.

Perplexity at corpus scale. Three 32k-token corpora, scored per position and compared paired between arms. Measuring each tensor family against its own BF16 ceiling located nearly all of the non-expert quantization cost in twelve o_proj tensors; at the shipped precision those twelve measure as a statistical tie with that ceiling. The rest of the trunk still has a little left in it, and the experts have not been probed this way at all.

Long context, the 10 to 32k band. A needle-in-a-haystack battery: a synthetic fact is spliced into filler at a known token position, the document continues into a sentence whose next words are that fact, greedy decode, exact string match. Three needles x five insertion positions x two filler corpora x five depths from 1,024 to 32,768 tokens.

The two misses confabulate a plausible-looking code rather than trailing off. The test can fail, and does. Since 0.9.1 those two misses are known to be the attention budget's: at HALOGEN_INDEXER_BUDGET=4096 both retrieve (Attention budget). The 1,024 depth is the control: below the attention selection budget the sparse path is not engaged, so it exercises the same dense attention the fixture gate already covers. Every depth above it runs block selection live, which no short fixture can reach.

This is the first quality measurement this project has in the band its prefill numbers are about. It is a retrieval test and not a general one: it says the model finds a fact it was given, not that its reasoning holds at depth.

Identity properties, gated on every build. The first two hold whatever your configuration; the third depends on one setting.

• At temperature 0, speculative decoding emits byte-identical tokens to serial greedy decode. The draft head only proposes; a token is emitted only if the full model would have produced it. It is speed with no quality cost. When sampling, the accept/reject rule emits exactly the requested distribution; a seed reproduces a request on the same drafter.
• A request batched alongside others emits byte-identical tokens to the same request run alone.
• A prompt-cache hit answers byte-identically to a cold run of the same prompt, under HALOGEN_PROMPT_CACHE=1, which is the setting to choose when you need that guarantee. The default cache mode trades it for speed at every prompt length; Choosing a cache mode has the numbers on what that trade actually costs.

What is not measured. We have never run the model at BF16. It does not fit in 124 GB, which is the whole reason this engine exists, so every quality number is against either a dequantized-weight reference or our own arms, never against the full-precision model at scale. Quality comparisons against other runtimes are not possible: their instruments differ from ours and neither of us has the BF16 baseline.

---

Precision: what you get, and how to trade it

You are running the quality build by default. There is nothing to enable.

The checkpoint ships as two files, and the engine picks the second one up on its own when it sits beside the first:

qwen38-flash-next-w4b.hgn              115.55 GiB   the checkpoint
qwen38-flash-next-w4b.overlay.hgn        2.31 GiB   the quality sidecar

One hf download gets both, so this is a fact about the files rather than a step you have to take. The server says which precision it loaded at startup, and warns if the sidecar is missing rather than quietly serving something worse.

The sidecar is a patch overlay: 723 tensors re-quantized against measured activation statistics, plus twelve o_proj tensors promoted to 8 bits, read in place of the base file's copies. It costs 0.09 GB net, because it is not adding weight, it is spending the same bits better. Measuring each tensor family against its own BF16 ceiling put nearly all of the non-expert quantization cost in those twelve tensors, 106 MB of a 115 GiB file. At 8 bits they measure as a statistical tie with that ceiling.

Which tensor families are stored at which precision is written out in docs/QUANT.md, along with what the sidecar changes and what has not been measured. Bits per weight there is computed from the tensor shapes in the checkpoint rather than quoted from a format name, so a format whose real cost differs from its nominal one shows the difference.

To trade quality for speed, point HALOGEN_CK_OVERLAY at the speed arm:

-e HALOGEN_CK_OVERLAY=/models/qwen38-flash-next-w4b.overlay-speed.hgn

That is the same re-quantization without the 8-bit promotion. It buys back about 2% of serial decode and gives up the calibration those twelve tensors carry. Setting it to none runs the bare 4-bit checkpoint, which costs about 6-9% perplexity and is the measurement control rather than a serving configuration.

4-bit weights are a correctness precondition, not an optimization: 125B parameters plus a 51B-parameter n-gram embedding table is 335 GiB at BF16 and 173 GiB at FP8, against 124 GB of unified memory.

---

Bring your own GGUF

Since 0.7.0 HALOGEN_CHECKPOINT may name a llama.cpp GGUF of this model instead of the engine's own checkpoint. The engine reads the file itself: at startup it repacks every tensor but the lookup table into the layouts its kernels read, losslessly (the file's own quantized values, moved, not requantized), reads the lookup table from the GGUF in place, and takes the draft head from a 1.4 GiB file of its own, because a GGUF carries no draft head this engine can run. Nothing is written to disk unless you ask.

podman run --rm -p 8731:8731 \
  --device /dev/kfd --device /dev/dri --group-add keep-groups \
  --ipc=host --ulimit memlock=-1:-1 \
  -e HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next \
  -e HALOGEN_CHECKPOINT=/models/Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf \
  -v ~/gguf-models:/models \
  ghcr.io/peonist-ai/halogen-flash-server:0.11.8

Name any shard of a split; the siblings are found by name. With HALOGEN_DOWNLOAD set and the volume writable, the first start fetches the draft head (qwen38-flash-next-mtp.hgn) and the tokenizer from the weights repo, 1.4 GiB in all; the GGUF itself is never downloaded by this image. Or put both beside the GGUF yourself and mount the volume read-only. HALOGEN_MTP_HEAD points at the head file if it lives elsewhere.

Which files. The repack is lossless where the format's values are a small set times a per-block scale, which is the whole IQ4_NL / IQ4_XS / IQ3_S / Q4_0 family for the experts, Q8_0 for the dense layers and Q6_K for the output projection; that is unsloth's UD-IQ4_XS build exactly (the file most numbers below were measured on), and any llama-quantize output in those types. Since 0.11.6 the engine reads the K-quant blocks Q4_K, Q5_K and Q5_1 the same way, as their exact affine planes, which is unsloth's UD-Q4_K_XL build. Only Q4_1, Q5_0, Q2_K, Q3_K and the IQ2/IQ1/F16 families are refused by name at startup, before anything is loaded, because reading them needs kernels for their block layouts rather than a repack, and a lossy fallback would make "the same file" untrue.

The short version: the GGUF costs decode and 4 GiB of RAM, and nothing else. Same prefill, better perplexity, 24 GB less disk; serial decode about 28% slower and coding-agent turns about 22% slower, because its 8-bit dense layers are 2 GB more to read per token. The full comparison:

What it costs and buys, measured on the reference machine with unsloth's UD-IQ4_XS (the same file llama.cpp reads; the engine's own checkpoint with its quality sidecar is the other arm; MTP on in both):

The quality row is the interesting one: unsloth's file keeps the dense layers at 8 bits and crushes the experts to about 3.4 bits, and that beats our calibrated 4-bit dense layers over 4.5-bit experts. The decode row is the price of the same bytes: an 8-bit trunk is 2 GB more per token at 240 GB/s, and no lossless repack avoids it. Prefill is compute-bound and does not care.

unsloth's UD-Q4_K_XL, the K-quant build (0.11.6). Read the same way, its own quantized values moved into the affine planes the kernels take with nothing requantized, and measured against UD-IQ4_XS in the same session on the same machine, MTP on in both. It is 104 GiB on disk and holds about 78 GiB in RAM, because its Q4_K / Q5_K dense rows carry more bits than UD-IQ4_XS's. It is the more accurate of the two: perplexity 0.020 nats lower than UD-IQ4_XS over 32K tokens (0.033 lower than the engine's own checkpoint, which both GGUFs beat), and fixture agreement 31/32 and 32/32 against transformers where UD-IQ4_XS scores 30 and 31. Prefill is the same (1,267 / 1,440 tok/s at 8,192 / 32,768, within 2% of UD-IQ4_XS); serial decode is about 3% slower (25.2 against 26.0 tok/s at short context) and the draft head accepts about as often (49% against 45% on prose). It carries the same 4 GiB of draft head and tokenizer and the same refusals; every speculative stream is byte-identical to serial greedy on it too.

Against llama.cpp on the same bytes, same machine, same session (their strix-halo branch, built and run at their settings on stock ROCm 7.14, so their numbers here are below their own published figures; the ratios are about this file on a stock box, and the decode ratio is the durable one): prefill 1.9x at 8,192 and 2.7x at 32,768; serial decode 1.1x at short context and 1.3x at 32K; with the draft head 1.3 to 1.4x; on coding-agent turns with both drafters 1.9x. The identity guarantee holds on their file: every speculative stream's tokens were byte-identical to serial greedy.

Startup. The repack reads the whole file once, on eight threads (HALOGEN_GGUF_THREADS): 18 s from a cold disk on the reference machine, 9 s with the file in the page cache, and every start pays it, because the repacked weights live in RAM and the page cache is dropped behind them so the file is not held twice. That is no slower than the engine's own checkpoint loads from a cold disk on the same machine (about 30 s for its 118 GiB). HALOGEN_GGUF_CACHE=1 writes the repack out once beside the GGUF (70 GiB; five minutes on the reference drive; =<dir> puts it elsewhere) and later starts take the engine's own path: 1.4 s when the cache file is warm, 16 s cold. It buys 7 s on a warm restart and 2 s on a cold one here, since the cache file is larger than the bytes the repack reads; it is for a host whose restarts are warm and whose disk is dear, not a requirement. The write needs the volume mounted read-write with the room to spare; without either it is skipped with a line in the log and the server starts without it, and a file left half-written by a crash is removed on the next start. A cache is checked against the shards' sizes and modification times on every start and is never used stale: with the flag set it is rebuilt, without it ignored, both said in the log. /health reports checkpoint_format as gguf or gguf-cache.

What does not apply. The quality sidecar is the engine's own checkpoint's and is not loaded over a GGUF trunk (the log says so). The draft head is ours and was fitted to our trunk; on unsloth's it accepts fewer draft tokens on prose (45% against 59%) and the same on code, which is inside the decode numbers above. flash_serve --repack IN.gguf --out OUT.hgn writes the same repack to a file for anyone who wants the artifact.

---

Configuration

Full list in docs/FLAGS.md. The ones that matter:

Choosing a cache mode

When a conversation continues, the server can either re-read the whole conversation from the start or pick up where it left off. HALOGEN_PROMPT_CACHE decides which, and there are three settings.

The follow-up figures are measured over a 20-turn conversation growing from 90,000 to 108,000 tokens, adding about 1,000 tokens a turn.

Use the default (2) for chat and agents, anything where one conversation gets longer. It is the only setting that helps short conversations: at the shipped configuration, mode 1 saves nothing at all until a conversation passes 32,768 tokens, so ordinary chat gets no benefit from it. The default has no such threshold; it starts working on the second turn, whatever the length. The first turn costs a little more, the price of capturing the state mid-pass: about 1% of the prefill with an ordinary system prompt and about 3.5% with a 20,000-token one (measured on a 32k prompt), paid once per new saved point, never on a hit.

Use 1 when you need the same prompt to always give the same answer: evaluation suites, regression tests, A/B comparisons, or anything audited. With 1, an answer served from the cache is byte-for-byte what a cold run would have produced. With the default it usually is, but not always.

Since 0.11.3 that "not always" applies only to a request that actually resumes from the cache and continues past it. A request the cache has nothing for (the first turn, or any prompt whose prefix is not held) answers byte-for-byte what modes 1 and 0 would: the server used to split such a request's forward pass at the point it saved, and now captures the state mid-pass instead (issue #65 was a cold request under the default mode landing on a wrong answer at temperature 1.0 that mode 1 did not produce). An exact repeat of a request also answers byte-for-byte its first answer: the server keeps the state at the end of the last request and restores it whole. /health reports the saved place's alignment as snapshot_align: 64: the place is the last 64-token boundary before the end of the system prompt or of the history, and the next turn re-reads at most 63 tokens.

Worth knowing what "not always" means, because it is smaller than it sounds. The difference only appears where the model was already close to a coin flip between two words. Across a battery of tests: the next word was identical in 9 of 9 single-resume tests, and differed 3 times across 38 resumed turns, every one of those three at a point where the model's top two candidates were within a rounding error of each other. On a 240-question fact-retrieval test the resumed server scored 236 against a cold server's 238, and in chat format specifically both scored 100%.

For proportion: the server already splits long prompts into chunks to fit them in memory, and simply changing where it splits moves the output slightly more than resuming from a cache does. Exact reproducibility across configuration changes was never on offer; mode 1 guarantees it across cache state, which is a narrower and more useful promise than it first appears.

Use 0 for many short unrelated prompts. Nothing is shared between them, so saving state is pure overhead.

One case where 1 genuinely wins on speed: one long shared prefix followed by many different short questions, such as a fixed system prompt or document asked about repeatedly from scratch. Mode 1 checkpoints at a fixed position all of those questions can resume from. The default saves its place at the end of each request and cannot rewind, so it misses. If that is your workload, 1 is both faster and stricter.

Context and memory: one KV pool, several conversations

The server admits the model's full native 262,144-token context per request by default, keeps two full-length conversations resident, and generates for four at once. Two settings that used to be one: HALOGEN_CTX is the most a single request may use, and HALOGEN_KV_POOL_POSITIONS is how many attention positions are resident across all conversations. The four slots share that pool rather than each owning a copy, so a slot adds only about 115 MB of its own state and the pool is what has to fit on the device. Attention state costs about 28 KiB per position including its scratch. Measured on a 128 GB machine:

A request reserves its prompt plus max_tokens positions when it is admitted (the chat default budget is 8,192 tokens, so a 30,000-token conversation reserves about 38,000) and waits in arrival order when the pool cannot hold it yet. Each stream's tokens are byte-identical to the same request run alone. Generation speed follows a conversation's own length, not the pool: a short chat in a 1M-position pool runs at short-chat speed, and three conversations at 250k each generate at about 17 tokens per second apiece.

Check what pool you actually got. The fit at startup budgets MemTotal less the resident weights (67.7 GiB) and HALOGEN_HOST_RESERVE_GIB (20), and a 524,288 pool at HALOGEN_MAX_TOK=32768 needs about 36.7 GiB (14.4 for the pool, 20.6 for the arena and the slots, 1.7 of margin). A 128 GB machine whose MemTotal reads 125 GiB fits it; one that reads 122.7 GiB (a crashkernel reservation of 2 GB is enough) does not, and the pool halves to 262,144 with the line kv pool: 524288 positions need ~36.7 GiB ... LOWERING THE POOL TO 262144 in the startup log. Every request line since 0.11.5 ends with pool N/<positions>, so a grep settles it. On such a machine HALOGEN_MAX_TOK=16384 gives back 8.8 GiB (about 9% of prefill speed) and 524,288 fits; at 8192 even 786,432 does (issue #75).

Sizing for a fan-out harness (a parent that runs subagents in parallel, issue #75). Every live conversation holds prompt + max_tokens between its turns, so the pool must hold their sum: pool >= sum over live conversations of (prompt + max_tokens), with max_tokens the client's, not the server default, when the client sends one. A subagent's prompt grows by the whole of each turn's generation when the harness replays the reasoning into the next request (Pi with reasoning: true does; the Qwen template keeps it), so a child that thinks for 25k tokens a turn under a 32k budget grows by about 26k a turn: a parent at 50k and two children at 80k each start at 82k + 112k + 112k = 306k and pass the default 524,288 by their fifth turn. When they do, a move cannot help and one conversation is forgotten by the least-recently-used rule; the log says which. The levers are the pool (786432 holds that shape for about four more turns, by which point the children are near the 262k context in any case), the client's budget (a harness that uses 2k of 32k can send 8k), and fewer parallel subagents.

One cost the pool does carry. A larger pool leaves less RAM for the model's file cache, so the first prompt after a restart reads its rows of the lookup table from disk. Before 0.6.3 those rows were read one at a time and a 32,000-token prompt took up to twice its usual 25 s right after a fresh start (up to 50 s more with the table fully evicted). Since 0.6.3 the rows are read 64 at a time: on this machine's NVMe drive the same first prompt costs about 1.3 s more than its usual time, and an 8,000-token one under half a second. The log line lookup table: ... took N s on 64 threads reports it whenever it takes 2 s or more. HALOGEN_KV_POOL_POSITIONS=262144 still leaves more of the table cached, and 786432 buys a third resident conversation where the machine has the headroom for it.

Speed by concurrency. Measured at the engine's own protocol on the published image at its defaults (the 8-stream row with HALOGEN_KV_SLOTS=8): 1,500-token prompts of prose, 600 tokens generated each, greedy, rates over the window in which every stream is generating. The one-stream rows are the same measurement, so the rows compare; the reproducible one-stream figure is the built-in bench below.

Re-measured on the 0.6.0 image in one session: 2 streams 56.5 total, 4 streams 77.1, every stream byte-identical to alone. Prompt lookup does not change these rows: like the draft head, it drafts only while a request is the only one generating (see below), and a batched step is already the cheapest way to get one token per stream on this hardware.

Slots are a latency policy, not a memory decision. Raising HALOGEN_KV_SLOTS past four trades what each client sees for admitting more clients at once instead of queueing them; past eight the total stops growing. Four is the default because it keeps per-stream speed where the numbers in this document were measured. Slots cap admission: a client past the count queues rather than diluting the batch, so with five workers on four slots each admitted stream still decodes at the four-stream rate and the fifth waits. A reporter's sweep on a five-worker aider fleet (issue #51) read the per-turn wall at 42 s on four slots against 46 on three and 52 on two, and 14.9 t/s per stream on eight slots against 18.7 on four; for reply-heavy, cache-hostile clients the default is the right setting even when more slots would fit.

Two other things the scheduler does for you. A prompt that arrives while other conversations are generating is read in pieces with a generation step for the others between pieces. The piece is the prefill call size (HALOGEN_MAX_TOK, 32,768 tokens), which is exactly how the same prompt is split when it runs alone, so its answer stays byte-identical: a 131k prompt pauses the others three times for about 28 s each instead of once for 105 s. HALOGEN_MAX_TOK=16384 halves the pause for everyone at about 8% slower prefill. HALOGEN_ADMIT_CHUNK=8192 makes the pause about 8 s and costs the admitted prompt about 5 s on its first token, and that prompt's answer then depends on the load when it arrived, which is the one setting here that gives up the identity property. And the speculative drafter, which is the default, speculates while it is the only conversation generating and joins the batch as soon as another one is active, so it never holds the others back; prompt lookup rides with it and follows the same rule.

The prompt cache keeps sixteen entries (HALOGEN_CACHE_ENTRIES), four per conversation: one at the end of its system prompt, two at ends of its history, and since 0.11.3 one at the end of its last request, which serves an exact repeat of that request without reading anything again (HALOGEN_CACHE_FULL=0 turns that one off). Since 0.8.1 a conversation holds a fixed number of entries: a new turn's history entry replaces an older one rather than adding to the list, so a long tool-calling session cannot push other sessions out (before that, eight tool calls in one session evicted every other conversation, issue #54; the count shows as superseded on /cache). Since 0.11.0 it keeps the two most recently used history entries, not the newest two, because of a client pattern that the newest-only rule broke (issue #61): a harness that sends the whole history plus a side question (oh-my-pi's idle recap does) and then drops that turn from its history continues from where the side turn branched, and with only the newest entry kept that point was gone and the next turn re-prefilled everything after the system prompt. Now the side turn hits the true history entry and stores its own beside it, and the next real turn hits the true one again. Conversations taking turns each resume from their own state, and requests that share a system prompt and ask different things, together or in turn, resume from it as well. More than four deep conversations at once wants HALOGEN_CACHE_ENTRIES raised to four per conversation (about 111 MiB of host RAM each, and the KV rows an entry covers stay reserved while it exists). The server prints the memory budget at startup and warns before the allocator refuses.

When the KV pool has no room for a new request, the server forgets the least recently used conversation's region and says so in the log (kv pool: no room for N positions; forgot the region at ...); the entry the request is about to resume from is never the one forgotten, and a conversation whose only stale entries are dead side turns grows its own region in place rather than displacing another conversation. Before 0.11.0 the eviction order could drop the very entry the request had matched, which read as an unexplained cold prefill (issue #61). When nothing else is left to forget and the request still has no span, because the conversation's own region sits where the reservation cannot reach (the upper half of the pool, with a reservation past half of it), the server moves that conversation's rows into the free span and the turn stays a cache hit (kv pool: ... moved the N rows this request resumes from, region A -> B); when the rows cannot be copied there it forgets them and the turn runs cold. Before 0.11.4 that request waited for room that could never appear, with the health probe answered throughout (issue #68). A request that waits for a busy conversation to retire says so once in the log (kv pool: request N waits for M positions ...), and /cache reports it under pool (waiting_for_room, waiting_s, relocated, cold_resorts). Since 0.11.7 that move is the FIRST thing tried when a region cannot grow, not the last: the conversation's own span counts as free, its rows move into any span that holds them, overlap or not (the copy goes in block-aligned chunks in the safe direction, moved the N rows ... in 12.3 ms), and when held neighbours are what stand in the way they are moved up against the next busy region so the region grows in place (kv pool: ... moved N held regions ... grows in place (no loss), counted as packed). Only after every no-loss step fails is another conversation forgotten, and cold_resorts reads 0 from 0.11.7 on. Before that, a harness fanning out two subagents beside a parent (issue #75) had the parent forgotten on every child re-bind (with both children decoding, the parent's region was the only one the least-recently-used rule could reach) and then the children forgetting their own rows on alternate turns; the same workload fails the same way on 0.11.4, so it was the allocator, not the 0.11.5 change.

A conversation keeps its whole reservation between turns, and since 0.11.5 its next turn uses it. A follow-up whose region cannot grow (another conversation's region sits directly after it) runs in the room the region has left, with max_tokens clamped to that room, when the room is at least the answer room (max(1024, 15%) of the request's max_tokens, twice that when the request thinks). The log says so (kv pool: the region at A this request resumes from cannot grow; the turn runs in the N positions it has left (max_tokens 32768 -> 30521)), the request line ends with max_tokens clamped 32768 -> 30521, and the response's timings carries max_tokens_clamped_from and max_tokens_clamped_to, so a finish_reason of length on such a turn is legible. Since 0.11.7 the clamp comes AFTER the moves above: it runs only when no span anywhere holds the region, so a harness whose turns use their budget keeps the whole budget at the cost of a copy (issue #74's own shape now runs warm with zero evictions and no clamp, its two sessions leapfrogging by a 7 ms copy a turn), and the clamp remains for a pool with nothing left to move. When the room is smaller than the answer room, or the conversation resumes from a region another request is decoding in, the rows are copied to a fresh span, and since 0.11.5 a held region whose longest entry the request resumes from is moved (kv pool: moved the N rows this request resumes from, region A -> B ... the old region is free) rather than copied and left behind, so a stale duplicate never competes with a live conversation for the pool. Since 0.11.8 all of this applies whether or not the client sends reasoning_content back; on 0.11.7 a client that did not was on the older path (issue #75, the second report). Before 0.11.5 two long conversations taking turns behind a shared system prompt forgot each other on every turn (issue #74): the first's growth could not extend or copy, the second's region was the only one to forget, and each turn re-prefilled the whole history at 100 to 160 s.

What the pool leaves the host. The engine prints, once loaded, how much host RAM is left after the weights and the pool (host memory left for everything else), and since 0.11.4 says under about 10 GiB what that means: the lookup table is read from disk through the page cache and never held, so that figure is the cache it gets, and below it the table's rows page in on every long prompt, a prefill takes minutes instead of seconds, and the watchdog can read the stall as a wedge. A reporter's measurements on a 128 GB machine with other services resident (issue #35), the pool the only change:

and on identical warm turns (a ~1,425-token prompt, 1,277 cached) before and after that change:

The machine this document's numbers come from runs the default pool at about 12 GiB left and does not page; the line between the two rows above is where the note fires. HALOGEN_KV_POOL_POSITIONS and HALOGEN_MAX_TOK are the two levers; stopping other resident workloads is the third.

HALOGEN_MAX_TOK (default 32,768, capped at the context) is the widest single prefill call, and it sizes the working memory the server holds beside the pool, which is a good deal more than the GEMM arena alone: the startup line's working memory reads 21.3 GiB at 32,768 and 12.5 GiB at 16,384 on the 0.11.4 image. Halving it gives back 8.8 GiB for about 9% of prefill speed, a long prompt read in more pieces, and the answer byte-identical. That made it the lever on the same reporter's next two boxes (issue #35), both shared with other work and both already auto-lowered to one full conversation of pool, so the pool could not go lower without cutting what one request may use (box A / box B):

Longer prompts are prefilled in pieces. Do not raise it to the native context. That allocation does not fit, and the server will not start.

Prompt cache on disk: resume a conversation after a restart

By default the prompt cache lives in memory: it makes a follow-up turn in the same running server resume where the last one stopped, but a restart loses it, and the next turn re-reads the whole conversation. Point HALOGEN_CACHE_DIR at a directory and the cache also lands on disk, so a conversation survives a restart:

HALOGEN_CACHE_DIR=/cache

Each turn, the server writes only that turn's new attention rows behind the request (nothing on the request's own path waits for it), and a request that is no longer in memory is restored from disk instead of re-read. On the test machine a 32k-token conversation, its server stopped and started, reached its first token in a few seconds against about 40 seconds of cold prefill.

What it stores is about 27 KiB per token of context: 0.9 GB at 32k, 2.7 GB at 100k, 7.2 GB at 262k. HALOGEN_CACHE_DISK_GIB bounds the directory (default 64; the least recently used conversations are removed first; 0 is unbounded). The directory must be on a real filesystem that accepts direct I/O; a tmpfs or an overlay is refused with a message, and the cache stays in memory only.

Two things worth knowing:

• It is exact. A conversation restored from disk continues from the same attention state it was saved with, byte for byte across the restart; it is the same "resume anywhere" cache as in memory, not a re-read.
• The write rate falls as the pool fills. Saving a turn's rows copies them out of device memory, and near the memory ceiling that copy slows (from several GB/s when the pool is a third full to a few hundred MB/s when it is near the top). Because the write is behind the request it does not slow the answer, but a machine that keeps very long conversations (past ~64k tokens) warm across restarts should shrink the resident pool with HALOGEN_KV_POOL_POSITIONS=262144, which keeps the copy at the drive's rate. For ordinary chat and agent sessions at the default pool the write is several GB/s and this does not arise.

Each distinct configuration keeps its own files, keyed to the exact engine build, weights, context, and every setting that changes the saved bytes, so a different build or setting never restores another's state; the others are kept untouched. Stopping the server flushes the last turn before it exits, so give it a moment to stop (the bundled compose sets a 60-second stop grace period).

1M context: opt-in, and a different configuration

The model card extends the native 262,144 to 1M by static YaRN (factor 4), and this server implements it. It is off unless you ask:

HALOGEN_ROPE_YARN=4 HALOGEN_CTX=1048576

Unset, nothing changes. Set, it is a different model configuration, not a cache setting: every position's RoPE is rescaled, short prompts included, and the card advises it only when the context needs it. What it costs, measured on the same machine as the table above:

• Quality at 1k-32k: perplexity +0.4-0.6% on three corpora (most of it above 8k positions); the 240-case retrieval battery reads 236/240 against 238/240 unscaled, with the chat register at 100% at every depth in both.
• Speculative decode at 8k-30k context accepts 5-15 points fewer drafts, about 10% slower than unscaled.
• Above 32k: needle retrieval, chat register, three needles at three positions: 9/9 at 262,144 unscaled, 9/9 at 262,144 scaled, 9/9 at 1,000,000.
• Memory: a 1M KV cache is ~25 GB, which on a 128 GB machine leaves no room for the default prefill arena. Past the native context the server therefore caps HALOGEN_MAX_TOK at 16384 (prefill about 9% slower) and prints it. A 1,000,000-token prompt prefills in 22-24 minutes (~700-770 tok/s) and decodes at 19 tok/s serial, 25 with the default speculative drafter. The conversation then continues at ordinary speed: the prompt cache keeps the attention state in place and saves only its small position-free part, so a follow-up turn at 1,000,000 tokens reached its first token in 0.55 s on the test machine (0.45 s at 262,144), against 22-24 minutes cold. A 262,144-token prompt decodes at ~28. The context must leave room for the generation: a prompt at exactly the context is refused.

Attention budget: opt-in, and a different configuration

The model's sparse attention scores every 4-token block of the context with a small indexer and attends the top 512 blocks (2,048 tokens) per query; the checkpoint's config sets that budget and this server runs it by default. It can be raised at startup, and it is off unless you ask:

HALOGEN_INDEXER_BUDGET=4096

Unset, or set to 2048, nothing changes (byte-identical). Set higher, it is a different model configuration, not a cache setting: every query past the budget attends a superset of what the checkpoint was trained to attend, so the answer to a long prompt is not the same answer. Accepted values are 2048 to 8192, rounded down to a multiple of 16; the effective value is printed at startup and reported by /health as indexer_budget. It is static for the server's life. What it buys and costs, measured on the same machine as the tables above, the default beside each arm in the same session:

At 4096 the two misses of the default's battery (the 16,384 row above, both the same needle) retrieve, and one different case is cut off at the end-of-turn token; perplexity moves within noise, with a structure worth knowing: the hardest predictions get better and the easy ones (verbatim copying) a little worse, most visibly on agentic transcripts. At 8192 every planted fact retrieves, but perplexity is a consistent cost on all three corpora and prefill pays a fifth. Speculative decoding stays byte-identical to serial at every budget. The cost is the attention kernel gathering more keys per query; nothing else in the pass changes.

If your workload is long-context lookup (a fact buried in a large document or transcript, asked about much later) and you can spare 7% of prefill, 4096 is the setting to try; measure it on your own prompts, because the retrieval battery is a retrieval test and not a general one. Reported as issue #57.

Composable context: an opt-in preview

An agent harness eventually compacts a conversation: it replaces the early turns with a short summary and keeps the recent tool results verbatim, so the context fits. Today that costs a full re-prefill of everything after the system prompt, because the kept results now sit at new positions. Composable context removes that cost for the kept results, and it is off unless you ask:

-e HALOGEN_COMPOSABLE_CONTEXT=1

With it on, each message at or above HALOGEN_COMPOSABLE_CONTEXT_FLOOR (default 2048 tokens) is retained in a host store (HALOGEN_COMPOSABLE_CONTEXT_BYTES, default 4 GiB, least-recently-used; in the server's memory, not on disk, and gone at restart) as it is first read. When a later request repeats that message at any offset behind the same system prompt, the server reuses the retained work instead of reading it again. On the test machine a compaction that kept ~8,700 tokens of tool results was restored in about 0.13 s where reading them fresh costs ~14 s, and the finish line reports composed 5 chunks.

It needs the resume-anywhere prompt cache and the KV pool, which are the serving defaults (HALOGEN_PROMPT_CACHE=2, HALOGEN_KV_POOL=1); it refuses image requests. /health reports it under composable_context.

It is not the prompt cache, and it is honest about that. The prompt cache is exact: a resume is byte-identical to a fresh run. Composable context is not: a reused answer is very close to, but not identical to, the one you would get by reading the text fresh, and quality is otherwise unchanged (retrieval in testing held at the same rate as reading fresh). That is why it is opt-in and its own switch. With the flag off, nothing changes and every byte-identical guarantee above still holds.

This is a preview, and the flag and defaults may change. On the roadmap for it:

• closer to exact — narrowing the small difference between a reused answer and reading the text fresh;
• broader reuse — reusing material across separate sessions and sub-agents working the same files, not only within one conversation's compaction;
• a smaller footprint and a durable store — less memory per retained message, and an optional on-disk store that survives a restart and holds far more than the in-memory one.

---

Troubleshooting

If the server will not start: "out of memory"

A start that ends in

dmalloc: FAILED requesting 0.750 GiB after 39.703 GiB in 647 allocations (out of memory)
HIP /src/halogen/src/flash_ops.h:122: out of memory

means the KV pool did not fit on this machine. HALOGEN_KV_SLOTS will not fix it and is the first thing most people try: since 0.3 the slots share one pool and each costs only about 115 MB, so one slot allocates as much as four. The knob is the pool:

-e HALOGEN_KV_POOL_POSITIONS=262144

That is 27.8 GB, the same layout 0.2.0 ran, and it still serves four conversations at once. 524288 is 35.0 GB and is the default. If it still will not start, halve the prefill call as well with -e HALOGEN_MAX_TOK=16384, which gives back about 8.8 GiB (the startup line's working memory, 21.3 to 12.5 GiB) for about 9% of prefill speed; the memory section has the measured table.

If the server starts but crawls on long prompts

A server that starts, answers short prompts, and then collapses to a few tokens per second on a long one, with the disk busy and the process stuck in uninterruptible sleep, is short of file cache rather than short of memory. The model keeps a large lookup table on disk and reads it through the page cache instead of holding it in RAM, so RAM the KV pool takes is RAM that table loses, and a longer prompt touches more of it. Since 0.6.3 the rows a prompt needs are read 64 at a time, so a table that is not in the cache costs seconds on an NVMe drive rather than minutes, and the log says how long each long prompt spent on it (lookup table: ... took N s on 64 threads). If that line still reads in the tens of seconds, the drive is the limit, and the same setting helps:

-e HALOGEN_KV_POOL_POSITIONS=262144

HALOGEN_HOST_RESERVE_GIB (default 20) is how much RAM the server leaves free for that cache when it sizes the pool at startup; raising it makes the server choose a smaller pool on its own.

Check the BIOS before you tune anything, if this machine carves memory out for the iGPU. A fixed block assigned to graphics in firmware is taken before the kernel boots, so it never shows up as missing anywhere on the host: the machine just reports itself smaller, and that RAM is gone from the file cache the lookup table depends on. This server does not need it. It drives the GPU through GTT and allocates from the same unified memory whichever way the setting is left, so a large carve-out buys nothing here and costs cache. Set the UMA frame buffer or dedicated graphics memory option back to Auto or its minimum, which reports about 512 MiB on this hardware.

You are paying for a carve-out even when nothing has thrashed yet. The pool sizes itself from the memory total the OS reports, which the carve-out has already made smaller, so the server quietly chooses a smaller pool and keeps fewer conversations resident than the pool table says. Pin the pool yourself and the file cache takes the whole loss instead. Either way the server prints what it found at startup, and warns when it is large:

memory: 16.0 GiB of this machine's RAM is carved out for the iGPU in firmware.
        That is not free memory the OS can lend to the file cache above, and
        it does not appear anywhere in /proc/meminfo: the machine simply
        reports itself smaller

Device memory here is system memory, and the ceiling is set by the kernel's resident-memory limit rather than by anything a driver reports: measured at about 47 GB on a 128 GB machine, and lower on machines carrying more besides this server. From 0.3.1 the server measures that budget at startup and lowers the pool itself when the configured one will not fit, printing what it chose; HALOGEN_KV_POOL_FIT=0 turns that off and allocates exactly what was asked for. HALOGEN_DMALLOC_LOG=1 prints every allocation over 64 MB with a running total, which is this configuration's memory budget measured rather than estimated, and HALOGEN_VERBOSE=1 turns on the fullest startup account the server can give. Both are off by default and both are useful to attach to a report. The two lines worth sending on their own are:

docker logs <container> 2>&1 | grep -E '^(dmalloc|kv pool):'

The host settings these numbers were measured on

Native Linux only. This server runs on the amdgpu/KFD driver stack and its memory design depends on it: the checkpoint is mapped and registered with the GPU in place, never copied, and every memory ceiling it knows about lives in that driver. WSL2 (ROCm through /dev/dxg) is not a supported host: the registration is refused there, and the server does not reach readiness. If you are on that stack, the same hardware booted into Linux is the path that works.

Kernel 7.0 or newer. The checkpoint is a read-only file mapping registered with the GPU as read-only, and that registration needs kernel support. The reference host runs 7.1.8 (Fedora 43 Server); every install reported working here is on 7.0.0 or later; on 6.18.6 (#37) the driver refuses every read-only mapping with invalid argument and the server cannot pin the weights. We have not bisected the exact kernel that added it; 7.0 is the oldest we have seen work.

Everything in Measured was measured on a machine booted like this, and the same command line has been in place unchanged for the whole life of this engine. This is our configuration, not a tuning guide: of the six settings we have A/B'd exactly one, and it is published here so the numbers can be reproduced and so a slow machine has somewhere to look.

amdgpu.vm_update_mode=0 amdgpu.noretry=0 amdgpu.gttsize=126976
ttm.pages_limit=32505856 amdgpu.sg_display=0 amd_iommu=off

amd_iommu=off is the one we have measured, and it is worth 13 to 16 percent of prefill. The numbers and the mechanism are in the conditions paragraph above. Two things to weigh before copying it: it turns off DMA translation machine-wide, which is a real change in posture on a host that is not dedicated to this, and it takes the NPU with it. On a box that exists to serve this model it is the right trade and it is the one we made.

ttm.pages_limit and amdgpu.gttsize are sizes, not constants. Do not paste ours. GTT is where every allocation this server makes on the GPU actually lands, and ttm.pages_limit sets that ceiling exactly: 32,505,856 pages times 4 KiB is 124 GiB, which is 99.3% of this machine's RAM, and it is precisely what the driver then reports as its GTT total. Both values say the same thing in different units, so set both to about your installed RAM:

Pasting the 128 GB row onto a 64 GB machine asks the driver for more GTT than the machine has. We have not measured what the kernel's own defaults are here, only that ours is what produced these numbers.

The remaining three, amdgpu.vm_update_mode=0, amdgpu.noretry=0 and amdgpu.sg_display=0, we have never run without. They are listed for completeness rather than recommended, and they are unmeasured in both directions: we make no claim about what they buy, and one report (#34) of an unkillable amdgpu deadlock came from a boot that had the first two set. One machine, one occurrence, not isolated to either flag, and none since on that machine without them. A second machine on the same issue, IOMMU off, ran the same workload with all three set and without: with them, 43 GiB of GTT stayed allocated after the container exited (Trying to push to a killed entity in dmesg) and every later start refused at the pin guard until a reboot; without them, GTT was back to 17 MiB within 5 s of every exit. Not isolated to one flag either. If you do not need them for something else, leave them off, and if a start refuses to pin right after a container exit, check

cat /sys/class/drm/card0/device/mem_info_gtt_used

before blaming the host's memory.

Check what you are on with:

cat /proc/cmdline

---

What this release is not

• Four conversations, not forty. The slot count is fixed at startup (HALOGEN_KV_SLOTS, up to 64) and a request waits for a free slot and for room in the pool; there is no preemption and no paging. Throughput past four streams grows slowly.
• Speculation is for a conversation on its own. With two or more conversations generating, every stream takes a batched step; the drafter resumes when a stream is alone again. Speculating inside a batch was measured to pay only for exactly two code-heavy streams and is not built.
• No response store. /v1/responses generates and streams; it does not keep responses, so previous_response_id, retrieval by id and cancellation by id are not available. Cancellation is by disconnect (since 0.10.2; #58): on every route, streaming or not, a request whose client closes the connection is cancelled within one decode step, its slot and KV reservation are released, and /health.in_flight and /metrics show it at once; the server log prints a "client disconnected" line with the timing. Its prefix stays in the prompt cache, so a retry resumes from it. Before 0.10.2 only streaming requests were cancelled; a non-streaming request ran to its natural end (EOS, max_tokens or the thinking budget).
• Images are read, not generated. There is no image output, and no audio or video input.
• One GPU, one model family. gfx1151 only. The build hard-rejects other architectures.

---

License

The engine is distributed under the terms in LICENSE.md. Third-party components and their licenses are listed in THIRD-PARTY-NOTICES.md. Model weights are licensed separately by their original authors.

"halogen" and "Peonist" are trademarks of Peonist, LLC (U.S. application pending). TRADEMARKS.md says how the names may be used; referring to the project, running it, and publishing numbers about it need no permission.