arcint

Intel Arc LLM inference engine, deliberately narrow: an OpenAI-compatible server for a short list of models on the Arc A770 and the Arc Pro B60, from the dense Qwen3.8-27B to the 512-expert Qwen3.8-Flash-Next.

arcint runs a short allowlist of hybrid Qwen models on two Intel cards — the Arc A770 (16 GiB) and the Arc Pro B60 (24 GB) — and tries to do that better than the general-purpose engines do. The inspiration is NInfer, a from-scratch engine that supports two checkpoints on one GPU and beats every generalist on that pair. arcint translates the idea to Intel: kernel work is delegated to OpenVINO's compiler stack, which already emits good Xe code, and arcint owns everything around the compute graph — the serving loop, the scheduler, the KV and recurrent-state memory, prefix caching, and speculative decoding. Where a kernel has to change, the change is a small, published patch series against a pinned OpenVINO build.

Every served model is a hybrid: most layers use linear attention (GatedDeltaNet), one in four uses full attention. Most of the design follows from that; see DESIGN.md. llm.txt is the machine-readable summary, CHANGELOG.md the per-release record and the runtime each release depends on.

Two engines

arcint has two inference backends behind the same HTTP surface, sampler, lanes and chat templates. Both services run on the second: the agent from 0.5.6, the coder from 2026-10-05, and Flash-Next from 0.6.0. The first carries Flash-Next's full-depth checkpoint blocks and stays as the coder's way back.

OpenVINO (the default). The model is an OpenVINO IR, or a GGUF fed through it. Runs on the patched marfrit-openvino runtime. arcint owns the whole memory picture here:

• paged KV in u8 or i8:u8;
• the exact prefix cache, in VRAM and in host RAM;
• admission by measured reservation;
• the Flash-Next expert tier between card and RAM;
• DFlash and MTP drafting.

That is what carried the coder service's 98,304 tokens until 2026-10-05 (and the agent's 122,880 before 0.5.6), and it carries Flash-Next at depth.

libllama (--engine llama --gguf FILE). llama.cpp at a pinned commit with ggml's OpenCL backend and arcint's Intel kernels (contrib/llama.cpp/patches: K-quant matvec and XMX GEMMs, int8 DPAS, gated delta-net, decode and prompt attention). It takes the GGUF as it is, with no export step, and drafts with the GGUF's own MTP head (--llama-mtp N, --llama-mtp-vocab). Its numbers (measured-here, 2026-10-04; the acceptance task 10/10 at temperature 0 on both):

The two engines run different weights: the services' int4 IRs are ~18 % fewer bytes than the GGUFs' Q4_K_M. The llama-engine figures are llama-bench prefill and served decode on the acceptance prompt, so they are not the services' long-context conditions.

Context on the llama engine (measured-here, 2026-10-04). The KV cache is f16 by default; --llama-kv q8_0 (or q8_0:q4_0) quantizes it, on arcint's attention kernels (contrib/llama.cpp 0015). --n-ctx defaults to 32,768.

• Dense 27B, B60: with f16 KV, 122,880 tokens without MTP: 24.4 of 25.7 GB of VRAM in use. A 30,065-token prompt prefills at 528 t/s at --n-ctx 122880 and at --n-ctx 32768 alike, so nothing pages. A 120,945-token prompt took 579 s (209 t/s on average). With MTP (5 drafts) and --llama-kv q8_0, 131,072 tokens:

  • peak VRAM 23.06 GB;
  • a 128,133-token prompt in 772 s (166 t/s), then decode at 7.5 t/s at that depth (0.5.6). Re-measured 2026-10-08 with the repository's source as context: 36.6 / 32.3 / 19.3 t/s at 4k / 29k / 130k prompt tokens with MTP 5, 18.4 / 15.1 / 9.7 without, prefill 252-272 t/s at 130k;
  • the acceptance task 8/10 at temperature 0 as deployed; sampled, q8_0 within the noise of f16 (docs/llama-engine.md).

  f16 KV with MTP at 131,072 overcommits the card.

• Coder, A770: bound by its weights, not its KV. The Q4_K_M GGUF is 14.9 GiB on the 16 GB card, while 98,304 tokens of its KV would be 1.9 GiB. With MTP, VRAM pages over the card's x4 link beyond ~16,384 tokens (prefill 585 t/s at 24,576).

Production (operator, 2026-10-04):

• The agent (dense 27B, B60) runs the libllama engine from 0.5.6, at 131,072 tokens with MTP and --llama-kv q8_0. Measured on llama-bench against the OpenVINO service's extension prefill:
  • decode with MTP is about twice the OpenVINO service's (47 against 24.6 t/s on the acceptance prompt);
  • prefill is slower: 896 t/s at 4k (OpenVINO 1,141) and 435 at 16k depth (852).
• The coder moved to libllama on 2026-10-05, with a searched GGUF.
  • Context: the Q4_K_M GGUF leaves ~16k tokens with MTP on the A770. An evolutionary search over per-layer expert types found one that serves 98,304 tokens and passes the answer-level bar (top-1 -0.10 points against Q4_K_M, KL lower). It scores 10/10 on the acceptance task (docs/llama-engine.md). The OpenVINO service stays as the way back.
  • Prefix reuse: the libllama engine has no shared prefix cache. A lane reuses the prefix it holds, and llama.cpp's context checkpoints let a hybrid model resume a follow-up or an edited message (--llama-checkpoints).
• Flash-Next (since 0.6.0): ISTA-DASLab's IQ2_XS GGUF with an expert cache that keeps the hot experts in VRAM slots and the rest in a USM bank in host memory, every expert computed on the card (--llama-expert-cache, --llama-expert-profile); the checkpoint's MTP layer drafts from a separate GGUF (--llama-mtp-gguf, tools/flash_next_mtp_gguf.sh). On the B60: 422 t/s prefill at 20k tokens, 33-35 t/s decode with MTP 2 (measured-here; Strata's own engine: 620 / 37.5 on the same card). At 131,072 tokens of context, 9,500 MiB of slots: 408 t/s and 19.8 t/s (contrib/systemd/arcint-flashnext.service). Since 0.7.3, with the bank stream (GGML_OPENCL_BANK_STREAM=1, contrib/llama.cpp 0032, 0033) in that configuration: 640-645 t/s prefill, 27-29 t/s decode.
• One model across two different Arc cards (since 0.7.0, --device GPU.0,GPU.1 --llama-layer-split K; docs/campaigns/layer-split-two-cards.md on the qfndev branch): Flash-Next split by layers across the B60 (Xe2) and the A770 (Xe-HPG).
  • Each card runs its own layer range with its own KV and its own expert cache, after Strata's layer split, which lists Intel GPUs as unsupported.
  • Intel's runtime lists the two generations as two OpenCL platforms; contrib/llama.cpp 0023 gives each its own context and builds every kernel for its own card.
  • At 131,072 tokens, against the B60 alone's 19.8 t/s (84.4 % of routed experts on the card): split at layer 40, 90.4 % on a card, 18.6 t/s with the MTP draft on the A770 and 20.4 with it on the B60 (--llama-mtp-card 0); split at layer 36 with the draft on the B60, 21.7 t/s and 93.5 %.
• An agent and a subagent lane on one set of weights (since 0.7.0, --served-model-name A,B --lane-ctx CA,CB): the request's model field picks the lane, an unknown name gets a 404, and each lane has its own context cap. One process and one KV pool serve qwen3.8-agent at 131,072 tokens and qwen3.8-subagent at 32,768 on the B60; contrib/llama.cpp 0026 gives each lane its own window of cells, so a lane attends over its own cells only. The subagent's 300-token answer decodes 32.0-32.1 t/s whether the agent lane holds 20k, 64k or 118k tokens; KL equal to one lane. Since 0.7.2, with MTP, two busy lanes share each draft and verify step in one decode (0028, 0029): 21.7 + 22.0 t/s at once, against 15.9 + 15.8 taking turns (docs/campaigns/lanes-agent-subagent.md on the qfndev branch).

docs/llama-engine.md has the details.

Building

C++20, CMake, no network at build time. third_party/ holds the two vendored single headers (cpp-httplib, nlohmann/json) and models/allowlist-raw.json holds the IR metadata the allowlist is pinned against.

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build
ctest --test-dir build --output-on-failure -L unit

./build/arcint --stub --port 8090 -v

This builds the stub, which serves the HTTP surface without a model and is what the device-free tests run against. In this configuration --model is refused at startup instead of starting something that cannot run. -L unit selects the device-free gate (arcint-test, the HTTP round trip, the stub-only concurrency stress test, and the acceptance enumeration's own consistency checks) — the only thing bare ctest ever runs. The card-requiring acceptance cells are a separate, enumerated target (tests/acceptance/, checklist in docs/release-checklist.md); see DEVELOPMENT.md for how to build and run them.

A build that serves a model needs OpenVINO:

cmake -S . -B build-ov -DARCINT_OPENVINO=ON -DCMAKE_BUILD_TYPE=Release \
      -DARCINT_GIT_SHA=$(git rev-parse --short=12 HEAD)
cmake --build build-ov -j"$(nproc)"
cmake --install build-ov --prefix ~/.local

The libllama engine needs a llama.cpp tree at the pinned commit (bed0a85) with contrib/llama.cpp/patches applied, plus the OpenCL headers and ICD loader:

git -C <llama.cpp> apply <arcint>/contrib/llama.cpp/patches/*.patch
cmake -S . -B build-ov -DARCINT_OPENVINO=ON -DARCINT_LLAMA=ON \
      -DARCINT_LLAMA_DIR=<llama.cpp> -DCMAKE_BUILD_TYPE=Release

The Debian recipe does exactly this (contrib/packaging/arcint/build-deb.sh).

The GPU runtime on a fresh host. Debian trixie ships no Intel compute runtime (intel-opencl-icd, libze-intel-gpu1) in any component; install Intel's release .debs from intel/compute-runtime with the IGC and gmmlib versions its release notes name. Every number here was measured on compute-runtime 26.27.39122.11 with IGC 2.38.2; a fresh trixie container with those, the .debs from these recipes and the cards bound in served the coder at the measured speed (0.6.0, measured-here).

-DARCINT_WERROR=ON gives the warning-clean build CI should use. Pass -DARCINT_GIT_SHA whenever the build tree has no .git; without it --version and /props report unknown.

The runtime. The OpenVINO engine's numbers below depend on a patched OpenVINO: marfrit-openvino +p25, the 2026.4.0 nightly at upstream commit 71640275 with the patch series 0003–0074 applied (+p27 adds 0076-0077 for the Flash-Next tier's opt-in switches). contrib/packaging/ holds the Debian recipes that build it and arcint itself, both engines in one binary, and contrib/packaging/marfrit-openvino/patches/README.md says what each patch does and what it measured. No .deb is published anywhere; the directory contains everything needed to build the same thing, and it is the shortest path to reproducing a number.

Containers. The same two halves are available as pull-and-run images: a tier-1 base carrying the patched OpenVINO runtime, and a thin tier-2 engine image on top, built with both engines. docker/Dockerfile.openvino-patched and docker/Dockerfile.arcint, with .github/workflows/build-toolbox.yml to build and publish both. See docs/docker.md for the run recipe and the tagging scheme.

Measured

The acceptance task is a Lua CSV parser to RFC 4180: ten named cases (CRLF and bare LF, a missing final terminator, quoted fields, an embedded comma, a doubled quote, an embedded newline, empty and empty-quoted fields, no trimming). The candidate code is executed, not read; one point per case. "task" below is the score out of 10. Every row was measured on this project's own hardware; the card, the configuration and the runtime are named, and the dated measurement is in DESIGN.md §7.

The two served configurations

The coder on the 24 GB card decodes at 66.5 t/s and prefills at 2,821 t/s at ~1k tokens — above the 60 t/s OpenVINO GenAI baseline on the same card and artifact that justifies the project.

Qwen3.6-35B-A3B on the 16 GiB card, all-resident

The 35B's int4 IR does not fit the A770 resident. Exported from the checkpoint's own GGUF with its expert blocks kept in their native formats (IQ2_S gate/up, IQ3_XXS down) and computed per routed expert on the card (--offload-ratio 0 --moe-per-expert-dispatch), it does, at 13.11 GiB:

Speculative decoding on the dense Qwen3.8-27B (B60)

The DFlash2 drafter is the public block-diffusion head incoai/Qwen3.8-27B-DFlash2, exported with tools/export_dflash.py. Drafters engage only under greedy, and a drafted token is accepted only when it equals what the sampler would have picked — but a multi-token verify pass and a single-token step are not bit-identical on this backend, so the answer can differ from plain greedy at a near-tie. Serve without a drafter where bit-exact reproducibility matters.

A GGUF through arcint (B60)

The dense Qwen3.8-27B as Unsloth's Q4_K_M, opened with --gguf on the dense IR as the topology template, u8 KV, one lane, MTP off, against Intel's own int4 IR of the same model:

The mixed form repacks the file's Q4_K rows at load into the runtime's own compressed form — every weight within a measured 1/64 of a quantisation step of ggml's value, the mins as exact extra columns — and runs Q5_K and Q6_K as the file's rows through a K-quant kernel carried in the patch series.

Supported models and formats

arcint loads an OpenVINO IR directory — openvino_language_model.{xml,bin}, openvino_text_embeddings_model.{xml,bin}, the tokenizer and detokenizer IRs, config.json and the chat template — and only one whose hashes and byte count match an allowlist entry. GPTQ or NVFP4 safetensors will not load.

A GGUF opens on top of an IR directory: --gguf FILE --model DIR takes the served IR of the same architecture as the topology template and replaces its projections with the file's own K-quant rows (Q4_K, Q5_K, Q6_K, Q8_0). --gguf-mode repack|native|mixed chooses the form (mixed, the default); --gguf-embed file|template where the token embeddings come from; --gguf-check once|always whether the repack's per-projection deviation check is cached between loads. Dense models of the allowlisted families.

Where to get the artifacts, how the served ones were exported and calibrated, the features, the non-goals and deployment notes are in FURTHER-READING.md.

This branch carries the current state. The dated measurement record, the campaign documents, design notes and acceptance windows behind every number here live on the development branch, qfndev.