Qwen3.5 CPU

A small C++20 inference engine specialized for Qwen3.5-0.8B text inference on CPUs, using calibrated H128/Q4-G32-DOT4 weights. No CUDA toolchain, GPU runtime or general-purpose model framework is required.

The project contains a native HTTP server, text-completion CLI, checkpoint packer, embeddable multi-request engine, CPU tests and comparison tools. It is an independent extraction of the validated CPU implementation in qwen35x; see provenance. The HTTP server supports a limited OpenAI-style /v1/completions API: raw prompts, greedy or sampled decoding and non-streaming responses. Chat completions and SSE streaming are not implemented.

CPU benchmarks: speed and quality

The engine uses the H128/Q4-G32-DOT4 checkpoint with a 424.9 MB tensor payload. It is generated using activation-weighted MSE16 fitting and error compensation within 128-channel blocks, calibrated on 256 mixed documents / 262,144 tokens. See the quantization recipe. Speed and quantization quality are measured separately.

Speed, size and quality

AMD Ryzen 9 9955HX3D · Arch Linux · eight physical V-Cache cores · GCC 16.2.1 Release · AVX-512/VNNI. Values are median tokens/s from three measured runs after one warmup. All candidates use identical fixed tokens, FP16 KV and full-vocabulary logits. Codex is minimized, XFCE compositing disabled, and the display runs at 2560×1600 / 240 Hz. IK uses runtime tensor repacking.

Sorted by single-request decode speed, fastest first. All speed columns are tokens/s; bold speed values mark the best result in each column. MB is tensor payload (decimal), not RAM usage. ↑ higher is better · ↓ lower is better.

PPL and KL use the quality evaluation described below; KL is in nats to the BF16 teacher. † Quality measured with llama.cpp on the identical GGUF, not with IK; all other quality values use the named backend. Quality scores are separate from the B16 speed measurement.

Prefill uses one request. Decode uses 512 input / 128 output tokens and counts 127 actual decode forwards per request. B16 is aggregate throughput across sixteen private requests. Loading, tokenization, HTTP and prefix-cache credit are excluded. IK B16 uses a local graph-capacity correction; IK B1, prefill and quality use the stock fork. These measurements describe this CPU and workload.

Speed on a second CPU

Intel Core i7-8750H · Ubuntu Linux · six physical cores · GCC 15.2.0 Release · AVX2. Values are median tokens/s from three measured runs after one warmup, pinned to physical cores 0–5 (0x3f). The fixed tokens, FP16 KV, full-vocabulary logits and timing boundaries match the table above. IK uses runtime tensor repacking; its B16 rows use the same documented graph-capacity correction. The laptop used the powersave governor with Intel P-state, so these results describe the measured system state rather than a fixed CPU clock.

Sorted by single-request decode speed, fastest first. MB remains tensor payload rather than file size or RAM usage. Bold values mark the best speed in each column.

The IQ4_K_R4 and IQ4_KS_R4 files were regenerated locally from the pinned Unsloth BF16 GGUF with the documented --pure recipe and ik_llama.cpp revision fe215a8c. Their tensor payloads match the Ryzen artifacts, but their hashes do not; no Ryzen quality score is attributed to these local files. Speed and quality therefore remain separate measurements. Absolute throughput between the two CPU tables should not be read as architecture-only scaling because the machines also differ in core count, ISA, compiler, power policy and desktop conditions.

Laptop performance CSV · Laptop benchmark provenance

Perplexity and KL divergence

8,192 scored tokens from 16 WikiText-2 test article windows, with identical prompts and scoring masks and a common BF16 teacher. This is an English-prose subset, not full-corpus WikiText perplexity. Lower PPL and KL are better. Each row identifies the backend used for scoring. Sorted by tensor size, largest first, then by lower PPL and lower KL for equal sizes. The BF16 teacher is the reference; the remaining rows are quantized checkpoints.

H128 and pure Q4_0 / IQ4_K_R4 share a 424,934,656-byte tensor budget. The smaller IQ4_KS_R4 trades some quality for size and decode speed. Tensor payload is not RAM usage: IK's loaded model tensors include additional storage; KV and compute buffers also require memory. Quantization labels alone do not establish equal quality.

The H128 checkpoint is calibrated on 256 mixed documents / 262,144 tokens. An independent mixed-language, code and math test scores 12.71898 PPL and 0.064178 KL over 1,280 tokens; it is a separate corpus from the table above.

Measurement details and reproduction · Performance CSV · B16 CSV · Provenance · Quantization recipe · Independent quality validation.

Download and run

The release workflow builds Windows x64 ZIP and Linux x64 tar.gz archives with the server, CLI, model download scripts and licenses. Runtime needs no compiler, Python or CUDA. Windows uses a static MSVC runtime; Linux releases target Ubuntu 22.04 or newer (glibc 2.35+), not every Linux distribution.

Model: danmoreng/Qwen3.5-0.8B-H128-Q4-G32-DOT4. Download binaries from GitHub Releases. The commands below pin the validated model revision, so later uploads cannot silently change the weights or tokenizer.

After extracting a release archive, run from its directory:

# Windows PowerShell 5.1+; no Python required
./download-model.ps1 -Repo danmoreng/Qwen3.5-0.8B-H128-Q4-G32-DOT4 -Revision f6cb5cf04a9094670f9041578a3f4e44b94a1395
./qwen35_cpu_server.exe --model-dir models/qwen3.5-0.8b --threads 8

# Linux: bash, curl and sha256sum
bash ./download-model.sh danmoreng/Qwen3.5-0.8B-H128-Q4-G32-DOT4 f6cb5cf04a9094670f9041578a3f4e44b94a1395
./qwen35_cpu_server --model-dir models/qwen3.5-0.8b --threads 8

The downloader verifies every file against the pinned revision's SHA-256 manifest. Download into a new directory. The checkpoint is already CPU-packed. The server defaults to localhost:8080, 16 resident requests, bounded queueing, FP16 shared KV pages and an 8192-token context limit.

curl http://127.0.0.1:8080/v1/completions -H "Content-Type: application/json" -d '{"prompt":"Once upon a time","max_tokens":64,"temperature":0}'

Use curl.exe on Windows, or Invoke-RestMethod with the same JSON body. For sampling (v0.1.1+), send e.g. "temperature":0.7, "top_p":0.8, "top_k":20, "repetition_penalty":1.05, "seed":42. Omitting temperature keeps greedy decoding for compatibility. Each request has its own random state. See server API and limits and publishing.

What is specialized

• One architecture: Qwen3.5-0.8B, including its hybrid DeltaNet/full-attention layers.
• Main weight paths: H128 transforms with Q4 groups of 32, or plain Q4_0, both using CPU DOT4 packing. Large projection weights use H128/Q4; embeddings and small retained tensors use their prescribed encodings. This is not an all-tensors-four-bit claim.
• H128 packing and sign conversion happen during checkpoint conversion. Plain Q4_0 GGUF weights are losslessly repacked once at model load. Inference reads the prepared blocks; loading also allocates metadata, scratch and projection groups.
• Experimental Q4_K_M GGUF execution is retained but slower at prefill; further K-quant optimization is paused. It does not use the optimized DOT4 path.
• FP16 KV caches, FP32 recurrent state, CPU-dispatched scalar/AVX2/VNNI/AVX-512 kernels.
• Batched decode, bounded scheduling, cancellation, output backpressure, exact-prefix reuse including recurrent state, and optional shared immutable KV pages.

The public C++ namespace remains qwen35x to keep the extraction small and traceable. See the engine API.

Build

Windows: Visual Studio 2022 C++ Build Tools, CMake 3.24+, Ninja and PowerShell 7. The server build downloads hash-pinned cpp-httplib and nlohmann/json headers; neither is required separately at runtime. Set -DQWEN35_BUILD_SERVER=OFF for an engine/CLI-only build with no downloaded dependencies.

./scripts/build.ps1
ctest --test-dir build --output-on-failure

Linux: a C++20 compiler, CMake and Ninja/pthreads.

cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build
ctest --test-dir build --output-on-failure

Windows/MSVC and Linux/GCC builds and kernel tests pass in GitHub CI. Real-model release tests are required on both platforms before a release tag publishes binaries; Intel i7-8750H hardware validation remains pending. ISA dispatch is designed for x86 portability, but this is not a claim of testing every x86 CPU. AVX2 or newer is the practical performance target; keep the portable core free of global -march=native flags.

Model and packing

Place the original Qwen3.5-0.8B Hugging Face checkpoint and tokenizer files in models/qwen3.5-0.8b. Models and build products are ignored by Git. The model has its own license; the engine's MIT license does not replace it.

./build/qwen35_cpu_pack.exe --hf-model-dir models/qwen3.5-0.8b --output models/qwen3.5-0.8b/model.q35h --quantizer mse16 --importance-dir benchmarks/bc-large-study/full-importance --covariance-dir benchmarks/bc-large-study/full-covariance
./build/qwen35_cpu.exe --model-dir models/qwen3.5-0.8b --weights models/qwen3.5-0.8b/model.q35h --prompt "Once upon a time" --threads 8 --max-new-tokens 128

The command above reproduces the calibrated recipe after collecting and fitting teacher inputs into the full importance and covariance directories above. The packer uses cpu-dot4 layout. Downloading the prepared standard model requires no calibration or conversion at runtime. See the standard recipe and conversion workflow. Runtime loading requires the matching .q35h, config.json and tokenizer files, not the original BF16 shards. The validated local artifact is about 425 MB (decimal); it is not shipped in this repository.

The CLI performs raw text completion, defaulting to greedy decoding. Use --temperature 0.7 --top-p 0.8 --top-k 20 --repetition-penalty 1.05 --seed 42 for sampling (v0.1.1+). It does not automatically render chat templates. For chat, pass an already rendered prompt using --prompt-file. Tokenization and teacher-forced logit export are also available through --help. On Linux, omit .exe in executable names.

Threads, batching and prefixes

Pass --threads explicitly for the CLI; use CpuEngineConfig::threads for the library. Eight threads is a starting point, not universal autotuning. Measure thread counts separately for single-request latency and batched throughput. On the development Ryzen, larger batches benefited from sixteen physical-core threads; using all SMT threads did not automatically improve performance.

The engine owns one executor per instance. Drive step() from one owner, drain output and release completed request records. Prefixes must match exact token IDs, model identity and trust namespace. Enable shared_kv_pages to share immutable FP16 prefix pages; hybrid recurrent state remains request-private.

Reproducibility and release status

Run all timed experiments through scripts/benchmark-inference-seq.ps1. It serializes processes, records commands and binary hashes, alternates case order, and writes profiles plus a completed CSV. Three measured runs and one warmup are the default. Quality/logit-dump runs are separate from speed measurements.

• CPU-only build and sixteen kernel/unit/evaluation tests: passed locally.
• Real-model scheduler/prefix/shared-page regression: passed.
• AVX-512 prefill: bitwise full-model logit equivalence over 24 prompts, including chunk boundaries and a 4,096-token prompt.
• Native text CLI and full-vocabulary logit export: smoke-tested.
• Native HTTP adapter, concurrent request/prefix correctness and packaged Windows server with the prepared Hugging Face model: passed locally.
• GitHub CI covers Windows/Linux builds, kernel tests and packaging. The public Hugging Face model is pinned for real-model tests and tag-triggered releases.
• Mixed 256-document calibration, error-compensated fitting and independent quality validation: passed.
• Broader context-length and corpus validation remains ongoing.

See validation details and the bounded release plan.

Calibrated MSE16 and block-128 error compensation run offline and preserves the DOT4 runtime and payload size. The standard download passes the free 2+2 = 4 regression. See the calibration and benchmark report for quality scores, sequential speed measurements, reproduction and remaining plan stages.