NInfer — gpillon fork

Why this fork

The upstream engine wasn't built with coding-agent workloads in mind. A tool like Qwen Code fires bursts of near-identical subagent requests that share the same long system prompt and tool schemas (often 98% identical) and diverge only in a short task-specific tail — but the engine had a single global prefill lane with no concept of "these requests are mostly the same": every subagent redid the full prefill from scratch, serialized one behind another, producing time-to-first-token ladders of several seconds per subagent even though almost none of that work was actually new. A long-running main conversation and a one-shot classifier call were also indistinguishable to the scheduler, so the conversation the user was waiting on could get evicted and re-prefilled at the worst possible moment. This fork exists to fix that: optimize the engine specifically for coding-tool usage patterns.

What's added in this fork

• Host-RAM KV cache tier (two-tier probation/protected eviction) that snapshots finished or still-decoding GPU lanes so a sibling request can restore instead of re-prefilling
• Prefix-reuse and admission logic so concurrent identical or shared-prefix requests (a subagent burst sharing one long system+tools prompt) skip the redundant part of prefill entirely
• Tagged request lanes (@main/@agents/@classifier, see below) so the scheduler knows which lane is the long-lived conversation and protects it from eviction by short-lived traffic
• DFlash2 speculative decoding made to actually run end-to-end on this tree (gather/selector addressing bugs fixed, RAM-tier KV carry for the drafter, working presets)
• Two silent KV-corruption bugs found and fixed (a rewrite-checkpoint restore and a hyperquant exact-key side store, both serving one request with another's state)
• Streaming/tool-call-parsing hardening, adaptive MTP verification-width calibration from measured round cost

Measured effect: siblings that used to pay a full prefill (hundreds to thousands of ms) now restore in tens to a few hundred ms, and a burst of subagents sharing a system+tools prefix reuses ~99% of it instead of queueing behind each other's redundant prefill.

Details, rationale, and file-level pointers: fork changes.

Thanks

• Thanks to cometkim: integration branch — kernel-perf (PDL decode chain, split-K prefill, per-request error boundary), DFlash2 base port, hyperquant KV cache, 1M-context envelope, NVFP4-full target
• Thanks to Mirko Covizzi: RTX 5090 Laptop compatibility and MTP adaptive verification-width tuning
• Thanks to mr-september: warmup/readiness decoupling and frontend streaming fixes
• Thanks to dylan (dylanbrodiefafard): RAM KV cache concept, LRU lane eviction, decode CPU-spin fix
• Thanks to Neroued: original NInfer engine

---

NInfer — cometkim fork

About this fork

This repository (cometkim/ninfer, branch cometkim/dev) is a personal integration workspace. Work happens on stacked feature branches and is re-applied here; nothing from this branch is pushed upstream directly.

Branch layout:

• master tracks origin/master and follows upstream/master (rebase; force-push own fork only).
• cometkim/dev is the primary experimentation workspace AND the integration branch: the fork-base commit (Windows toolchain pins, local build wrapper, bench prompts, handoff notes) plus one squashed commit per verified feature branch, applied in stack order. Anything may be committed here at any time — no discipline required.
• feat/* are the curated feature branches containing no machine-specific files (no absolute toolchain paths, local prompts, or agent workspaces) so any of them can become an upstream PR at any time. They are rewritten freely (git push --force-with-lease origin feat/...). Feature branches stack on master directly or on feat/windows-port — the Windows port layer (from natpate/ninfer-windows), which is engine work, not environment: local builds and local verification require it. It is itself PR-able; when upstream takes it, the branches above rebase onto master and the layer disappears.

The experimentation cycle:

1. Commit anything on cometkim/dev — it is the sandbox.
2. Extract / cherry-pick the meaningful pieces into feat/*, keeping those branches upstream-PR-able.
3. Squash verified feat branches back into cometkim/dev and dedup: reset dev to the fork base, re-apply each feat branch as one squashed commit in stack order — this both integrates the verified state and drops the superseded experimental commits. Squashed feature commits carry the prefix squash(feat/<branch>): — they are re-applied snapshots, not primary history, and are the only commits on this branch that a future rebuild may drop or reorder freely (git log --grep '^squash(feat/'). Exception: a feat branch that upstream has merged is never squashed into dev — sync master to upstream, rebase the remaining stack onto it, and the merged branch's content arrives through the lineage; drop its squash from the rebuild.

Ongoing feature experimentation:

When a row reaches merged upstream, remove its squash from the next dev rebuild and fold its row into the upstream lineage note.

Fork-local on cometkim/dev only (never in a feat branch): the /utf-8 source/execution charset flags — a CP949-locale build workaround owned by this environment, not engine work.

git switch cometkim/dev
git reset --hard <base-commit>          # the chore(dev) fork base, before any squash
git merge --squash feat/build-speed     && git commit   # re-apply in stack order
git merge --squash feat/qwen3.8-nvfp4full && git commit
git merge --squash feat/hyperquant     && git commit
git merge --squash feat/1m-context     && git commit

After every re-apply, verify content parity — the integration tree must differ from the feature tip only by fork-only files (environment and session docs) and content owned by sibling feature branches or upstream commits the feature lineage predates:

git diff feat/1m-context cometkim/dev --name-only    # expect fork files (.gitignore,
                                                    # Directory.Build.props, HANDOFF.md,
                                                    # ROADMAP-1m-context.md, build scripts,
                                                    # longprompt_*.json, README) plus
                                                    # sibling-branch files — never a
                                                    # feature-owned file

Machine-specific files live only in the environment commit (Directory.Build.props pins the CUDA 13.3 toolkit path for MSBuild; configure-ninja.ps1/build-ninja.ps1 are the Ninja fast-path build wrappers and build-live.ps1 the live-progress Visual Studio wrapper, with machine-local fallbacks for VCPKG_ROOT and CUDA_PATH; longprompt_*.json are fixed-context bench inputs; HANDOFF.md and ROADMAP-1m-context.md track active work).

Selected checkpoints. Maximum single-GPU inference performance.

This fork carries Neroued/ninfer, a from-scratch C++/CUDA inference engine for explicitly registered Qwen checkpoints on a single NVIDIA GeForce RTX 5090, and runs text, image, and video prompts through a local CLI, OpenAI-/Anthropic-compatible HTTP APIs, or the included llama.cpp webui. Development and measurement happen on Windows 11 x64 today (native builds via the port lineage); the Linux build path is unchanged from upstream and is the intended next home for this environment.

NInfer deliberately supports a closed set of model artifacts instead of acting as a general model runtime:

Qwen3.6-27B and Qwen3.8-27B each expose two registered weight profiles. The version-2 artifact identity selects the profile without a separate runtime flag; Qwen3.8 uses target key qwen3_8_27b while sharing the 27B execution package. The Qwen3.6 nvfp4 profile uses W4A4 Tensor Core MMA for prefill and A16 NVFP4 kernels for decode. The Qwen3.8 nvfp4 profile preserves its source's mixed allocation: NVFP4 MLP weights in Text layers 0–55 and row-scaled FP8 for the token embedding, attention input/output projections, GDN Q/K/V/Z and output projections, output head, and remaining MLP weights. All four 27B artifacts retain the same Text, Vision, MTP, prefix-reuse, CLI, and serving routes.

Upstream lineage

NInfer is Neroued's project (Neroued/ninfer); that repository remains the reference implementation, and master here follows upstream/master directly. This workspace builds on natpate/ninfer-windows, the native Windows port of that engine — the port core below is inherited from it (the compatibility layer itself ported from Don-Chad/ninfer-3090, without its RTX 3090 retargeting). The WebUI and context-reporting items live on this fork's feat/windows-port lineage — natpate's dev branch has already absorbed them, and PR #2 delivers them to its master.

Windows-port lineage:

• Native Windows 11 x64 build and run — CMake with Visual Studio 2022 (MSVC), with vcpkg resolving FFmpeg, libcurl, and zlib via the vcpkg.json manifest; the CUDA runtime is statically linked, so the CUDA Toolkit is only needed at build time.
• Windows porting of the runtime — memory-mapped artifact reading with unbuffered overlapped I/O (the Windows counterpart of POSIX O_DIRECT/pread, with the same 4096-byte alignment contract), portable console logging and load progress, and portable media acquisition for image and video input.
• MSVC/TMA kernel compatibility — fixes that let the upstream Blackwell kernels compile under MSVC: device-pointer NVFP4 TMA descriptors, the pair-row SwiGLU TMA epilogue, and MSVC move-construction details in the target runtime.
• Stock llama.cpp WebUI — the HTTP server additionally accepts the stock llama.cpp WebUI's API dialect, and ninfer-serve can serve the unmodified WebUI in-process: --webui downloads the latest build from the ggml-org/llama-ui bucket on first start, or --webui-dir DIR serves an existing local copy.
• Context window reporting — ninfer-serve advertises the served context ceiling in the OpenAI dialect: the objects returned by /v1/models and /v1/models/{id} carry meta.n_ctx = the --max-context value in force.
• Portable Windows release — a self-contained zip containing the executables and all runtime DLLs; see Prebuilt Windows release.

What this fork adds on top (one squashed commit per feature branch, per About this fork):

• feat/qwen3.8-nvfp4full — the Qwen3.8-27B NVFP4 full-precision-requant artifact: unified activation scale space, conversion tooling, model card, and the published gpqa comparison.
• feat/hyperquant — the hq-e8-2b KV cache: E8-lattice + Rice entropy coding at 2.25 bits/scalar (3.7× smaller payload than INT8 at 262k context), prompt attention over one-shot decoded rotated bf16 scratch, an 8-lane cooperative k=0 Rice decoder with a parallel unary packer, and a tensor-core tile-source decode kernel (the hq route is a KV-source policy of the bf16 TC kernel). After the residual-window + dither work below, hq decode matches or beats same-session INT8 at the measured cells (tg128 80.6 vs 71.7, pp32k+tg64 62.2 vs 58.1 tok/s).
• feat/1m-context — the 1M-context track: engine envelope raised to 1,048,576 keys on the U8 cache (banded rotated scratch, 1M + --kv-capacity auto resolves the full pool with ~3.3 GiB free), YaRN rope scaling (--rope-scaling yarn:F[,t=][,bf=][,bs=], q-side temperature, FP64 angle reduction), the bf16/int8 linear envelope raised to 524,288 keys, and the WI-8 quality fix — a BF16 sink+recent residual window (S=32/W=512 exact side rows with per-slot ring validity) plus half-cell subtractive dither, which fixed the >262k hq garble: greedy needle retrieval is exact at 32k/304k/390k/592k true tokens (592k runs yarn:4 at 916–922 prefill / 15.2 decode tok/s; MTP3 at 390k commits 62.9 tok/s). Beyond ~592k the 1M cell still garbles (depth-independent; dense-YaRN-at-1M is the leading suspect — see HANDOFF).

Everything else — the Linux build path, the RTX 5090 (sm_120a) target, the CUDA 13.1 requirement, and the NVFP4/W4A4 Blackwell execution paths — is unchanged from upstream.

Performance

The published measurements cover the three Qwen3.6 artifact profiles and the Qwen3.8-27B NVFP4 profile. The Qwen3.8-27B groupwise-int profile is supported by current NInfer builds but is not yet included in a published benchmark campaign.

Concurrent MTP3 decode

Saturated decode was measured on an RTX 5090 with INT8 group-64 KV cache, CUDA Graphs, MTP3, and one 8,192-token generation per active request. The values below are aggregate committed decode throughput from complete one-second intervals in which the actual decode batch remained equal to the configured concurrency. MTP acceptance is aggregated over the complete request wave. Each concurrency cell reports decode tok/s / MTP acceptance; profiles should be read independently.

At C=8, Qwen3.6-35B-A3B reaches 1,313.8 aggregate decode tok/s. Qwen3.6-27B NVFP4 reaches 1,146.9 tok/s and 5.67× its C=1 throughput. Qwen3.8-27B NVFP4 has 45.8–48.9% MTP acceptance, versus 67.2–71.4% across the other measured profiles, so aggregate committed throughput reflects both execution performance and speculative acceptance.

Single-request serving

The single-request corpus was measured on the same GPU with INT8 group-64 KV cache, CUDA Graphs, and a 1,024-token prefill chunk. Each reported fixture uses five fixed seeds after server warm-up. Targets and weight profiles are reported independently rather than as cross-target comparisons. Requests were submitted serially to a persistent server. The Qwen3.8-27B NVFP4 MTP0 results use the same dedicated serial corpus runner as the Qwen3.6 profiles; its MTP3 results come from the C=1 point of the fixed concurrent-corpus campaign documented in Performance.

Qwen3.6-35B-A3B

• MTP0 at a 7,680-token prompt: 15,544.3 prefill tok/s and 271.1 decode tok/s.
• MTP0 at a 260,096-token prompt: 5,157.1 prefill tok/s and 188.2 decode tok/s.
• MTP3 long reasoning: 620.3–726.2 decode tok/s with 72.7–82.8% acceptance.
• MTP3 structured output: 770.9 decode tok/s, 89.1% acceptance, and 3.67 tokens/round.

Qwen3.6-27B (groupwise-int)

• MTP0 at a 7,680-token prompt: 3,218.1 prefill tok/s and 77.6 decode tok/s.
• MTP0 at a 260,096-token prompt: 1,614.8 prefill tok/s and 54.8 decode tok/s.
• MTP3 long reasoning: 161.9–175.4 decode tok/s with 73.4–78.8% acceptance.
• MTP3 structured output: 193.0 decode tok/s, 88.7% acceptance, and 3.66 tokens/round.

Qwen3.6-27B (nvfp4)

• MTP0 at a 7,680-token prompt: 11,191.5 prefill tok/s and 86.4 decode tok/s.
• MTP0 at a 260,096-token prompt: 2,510.6 prefill tok/s and 59.9 decode tok/s.
• MTP3 long reasoning: 213.1–231.0 decode tok/s with 76.3–81.1% acceptance.
• MTP3 structured output: 252.2 decode tok/s, 89.8% acceptance, and 3.69 tokens/round.
• Against groupwise-int on the same corpus and runtime options: 3.48× the 7,680-token prefill throughput, 1.55× the 260,096-token prefill throughput, and 30–32% higher MTP3 decode throughput.

Qwen3.8-27B (nvfp4)

• MTP0 at a 7,680-token prompt: 8,340.4 prefill tok/s and 71.2 decode tok/s.
• MTP0 at a 260,096-token prompt: 2,203.1 prefill tok/s and 52.9 decode tok/s.
• MTP3 long reasoning: 151.4–195.2 decode tok/s with 56.2–76.0% acceptance.
• MTP3 structured output: 219.8 decode tok/s, 90.8% acceptance, and 3.72 tokens/round.

See Performance for the full methodology, variability, reproduction command, and per-fixture results.

Evaluation

Capability scores were measured through NInfer's OpenAI-compatible serving route with thinking enabled, MTP=3, and EvalScope 1.9.0 (0-shot, rule scoring, one sample per problem):

The Qwen3.6 rows used temperature 0.6 and presence penalty 1.0; the Qwen3.8-27B rows used temperature 1.0 and presence penalty 0.0. The multimodal columns (ERQA and RealWorldQA) ran with --vision at a 81,920-token context limit; the text columns used a 262,144-token limit except Qwen3.8-27B NVFP4, which needs 252,928 to fit the RTX 5090 after weights.

These are single-sample results under that NInfer evaluation profile, not pass@k. See the model cards and full performance document for correct/total counts and evaluation notes.

Requirements

NInfer currently requires:

• 64-bit Linux or Windows 11 x64;
• NVIDIA GeForce RTX 5090 (sm_120a);
• NVIDIA driver support for CUDA 13.1 and the CUDA Toolkit 13.1 or newer;
• CMake 3.28 or newer and a C++20-capable host compiler (GCC or Clang on Linux, MSVC from Visual Studio 2022 on Windows);
• FFmpeg development libraries: libavformat >= 60, libavcodec >= 60, libavutil >= 58, and libswscale >= 7;
• libcurl >= 7.85;
• pkg-config on Linux, or vcpkg on Windows (the repository pins the dependency baseline in vcpkg.json);
• Ninja, when using the commands below.

The build rejects CUDA architectures other than 120a. On Linux, NInfer is run from its source build tree; on Windows, the prebuilt portable release provides the same binaries without a toolchain.

Prebuilt Windows release

Windows users who would rather not build can use the portable release instead of the build steps below. The zip is self-contained — executables, all runtime DLLs (FFmpeg, libcurl, zlib, and the VC++ runtime; the CUDA runtime is statically linked), launcher scripts, a models\ folder, a README.txt, and SHA256SUMS:

1. Download the latest ninfer-windows-<version>-win64-cuda131.zip from GitHub Releases. Verify files against SHA256SUMS, e.g. Get-FileHash ninfer-serve.exe -Algorithm SHA256.
2. Extract it anywhere — the launcher scripts use relative paths and work from any location.
3. Download a model into models\ as in Download a model.
4. Run the matching launcher, e.g. .\qwen3_8_27b.bat. This starts ninfer-serve on http://127.0.0.1:8080 (API at /v1) and serves the WebUI at the root URL; --webui downloads the WebUI on first start, so the first run needs an internet connection (later runs reuse the local copy).
5. Or run .\ninfer-serve.exe models\<model>.ninfer [flags] directly — the options are identical to a source build (see Run the HTTP server).

The launchers default to a 150,000-token context (--max-context / --default-max-tokens) to leave VRAM headroom for the Windows desktop. On the 32 GB RTX 5090, the smaller models (qwen3_6_27b, qwen3_6_27b_nvfp4, and qwen3_8_27b) can be safely raised to 200,000 when VRAM is completely free at startup; the two larger models (qwen3_8_27b_nvfp4 and qwen3_6_35b_a3b) do not fit at 200,000 and should stay at 150,000. Hardware requirements are unchanged: Windows 11 x64, RTX 5090, and an NVIDIA driver supporting CUDA 13.1.

Build

Linux

git clone https://github.com/cometkim/ninfer.git
cd ninfer

cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build --parallel

The default configuration builds:

build/apps/ninfer
build/apps/ninfer-serve

Tests, benchmarks, and maintainer tools are excluded from the default build.

Windows

The fork's build entry point is a pair of plain-PowerShell Ninja wrappers — no developer prompt needed. configure-ninja.ps1 (run once per build directory) imports the MSVC (vcvars64) environment itself, resolves cmake/ninja from PATH with VS-bundled and winget fallbacks, points the vcpkg toolchain at VCPKG_ROOT (falling back to this machine's local checkout) and nvcc at CUDA_PATH, and enables tests and benchmarks by default (-NoTests / -NoBenchmarks exclude them). build-ninja.ps1 builds everything or one target:

git clone https://github.com/cometkim/ninfer.git
cd ninfer

powershell -ExecutionPolicy Bypass -File configure-ninja.ps1
powershell -ExecutionPolicy Bypass -File build-ninja.ps1
powershell -ExecutionPolicy Bypass -File build-ninja.ps1 -Target ninfer_bench

The default configuration builds:

build-ninja/apps/ninfer.exe
build-ninja/apps/ninfer-serve.exe

with tests under build-ninja/tests/ and benchmarks under build-ninja/bench/. The GQA launcher TUs are split per dtype (and per geometry for the hq codec kernels), so a cold build takes ~15 minutes on the RTX 5090 box versus ~50 minutes through the Visual Studio generator, and touching a launcher dispatcher rebuilds in ~90 seconds. MSVC compiles with /utf-8 for source and execution charset, matching the Linux toolchain default.

The same configuration works through the Visual Studio generator without the wrappers — Visual Studio 2022 (MSVC) and vcpkg; the manifest in the repository root pins curl, ffmpeg, and pkgconf:

cmake -S . -B build-windows -G "Visual Studio 17 2022" -A x64 `
  -DCMAKE_TOOLCHAIN_FILE=C:/path/to/vcpkg/scripts/buildsystems/vcpkg.cmake `
  -DVCPKG_TARGET_TRIPLET=x64-windows
cmake --build build-windows --config Release --parallel

building build-windows/apps/Release/ninfer.exe and build-windows/apps/Release/ninfer-serve.exe (tests and benchmarks require -DBUILD_TESTING=ON / -DNINFER_BUILD_BENCHMARKS=ON; the Ninja wrapper turns both on by default). build-live.ps1 wraps the Visual Studio flow with live progress output and remains supported.

See the Windows guide for complete setup instructions, vcpkg installation, and notes on the resulting DLL layout.

Docker

Build the runtime image on a 64-bit Linux host with an RTX 5090, a CUDA 13.1-compatible NVIDIA driver, Docker, and the NVIDIA Container Toolkit.

docker build --tag ninfer:local .

Download a model into models/ as described below, then run the HTTP server:

docker run --rm \
  --gpus '"device=0"' \
  --publish 8080:8080 \
  --volume "$PWD/models:/models:ro" \
  ninfer:local \
  ninfer-serve /models/qwen3_6_27b.ninfer \
  --host 0.0.0.0

Run the CLI from the same image:

docker run --rm \
  --gpus '"device=0"' \
  --volume "$PWD/models:/models:ro" \
  ninfer:local \
  ninfer /models/qwen3_6_27b.ninfer \
  --prompt "Explain prefill and decode in three sentences." \
  --max-new 256

Download a model

Use the Hugging Face CLI to download one of the registered artifacts:

hf download neroued/Qwen3.6-27B-NInfer \
  qwen3_6_27b.ninfer \
  --local-dir models

# Or the 27B NVFP4 weight variant:
hf download neroued/Qwen3.6-27B-nvfp4-NInfer \
  qwen3_6_27b_nvfp4.ninfer \
  --local-dir models

# Or Qwen3.8-27B:
hf download neroued/Qwen3.8-27B-NInfer \
  qwen3_8_27b.ninfer \
  --local-dir models

# Or Qwen3.8-27B NVFP4:
hf download neroued/Qwen3.8-27B-nvfp4-NInfer \
  qwen3_8_27b_nvfp4.ninfer \
  --local-dir models

# Or:
hf download neroued/Qwen3.6-35B-A3B-NInfer \
  qwen3_6_35b_a3b.ninfer \
  --local-dir models

Current NInfer builds accept only the version-2 artifact container, and all five downloads above are version 2. Migration applies only to Qwen3.6 artifacts downloaded before their version-2 publication; both Qwen3.8-27B profiles were published directly as version 2. Migrate an older exact local file in place:

python3 -m tools.artifact.migrate_v1_to_v2 models/qwen3_6_27b.ninfer

Use the same command with qwen3_6_27b_nvfp4.ninfer or qwen3_6_35b_a3b.ninfer for those artifacts. The migration updates only container metadata; it does not rewrite the weight payload. Alternatively, download the current version-2 file again from its Hugging Face repository.

Each .ninfer file contains the weights and frontend resources needed by NInfer. It is not a Transformers checkpoint, Safetensors distribution, or GGUF file.

Each artifact is complete, while GPU residency is fixed at process startup. Speculative decoding is disabled by default, so MTP/DFlash state and the optimized proposal head are not uploaded. Vision is also disabled by default, so its weights, Vision scratch phase, and frozen request-transient allocation are omitted. Add --vision to the CLI or server process that must accept image or video input. Disabled capabilities cannot be enabled by a later request. DFlash is available only for the 35B-A3B target and is text-only.

Run the CLI

./build/apps/ninfer models/qwen3_6_27b.ninfer \
  --prompt "Explain prefill and decode in three sentences." \
  --max-context 16384 \
  --max-new 256 \
  --spec mtp --draft-tokens 3 \
  --lm-head-draft

Use --messages FILE instead of --prompt for chat history, images, or videos:

./build/apps/ninfer models/qwen3_6_27b.ninfer \
  --messages examples/cli/messages/image_chart.json \
  --max-context 8192 \
  --max-new 128 \
  --vision

Answer content is written to stdout. Loading progress, reasoning, timing, throughput, memory, and speculative-decoding statistics are written to stderr. See the CLI guide and committed examples for structured input and runtime options.

Run the HTTP server

./build/apps/ninfer-serve models/qwen3_6_27b.ninfer \
  --max-context 16384 \
  --kv-capacity auto \
  --max-concurrency 2 \
  --spec mtp --draft-tokens 3 \
  --lm-head-draft

The public model ID defaults to the artifact's identity.model_id; use --model-id only to publish a deployment-specific alias.

Then send an OpenAI-style request:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen3.6-27b",
    "messages": [{"role": "user", "content": "Reply with one short sentence."}],
    "max_tokens": 64
  }'

The server also implements OpenAI Responses Core (typed Items, semantic SSE, local continuation state, and function calls) plus Anthropic Messages, token counting, and multimodal input. See HTTP serving.

Request lanes for coding agents (@main / @agents / @classifier)

(gpillon fork addition — see About this fork above for why this exists.)

Append @main, @agents, or @classifier to the model field to tell the scheduler what kind of request this is, so lane eviction and prefix reuse can treat it accordingly:

# The user-facing conversation — protect this lane from eviction.
curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen3.6-27b@main",
    "messages": [{"role": "user", "content": "Continue the conversation."}],
    "max_tokens": 512
  }'

# A subagent spawned by the main conversation — freely recyclable, and if its
# prompt shares a long system+tools prefix with a sibling already running
# (the common case for a burst of subagents from the same tool), it reuses
# that prefix instead of re-prefilling.
curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen3.6-27b@agents",
    "messages": [{"role": "system", "content": "<same long system+tools prompt>"},
                 {"role": "user", "content": "<subagent task N>"}],
    "max_tokens": 256
  }'

# A one-shot intent classifier — same recycling tier as @agents, kept
# separate so its churn is visible on its own in the throughput log.
curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen3.6-27b@classifier",
    "messages": [{"role": "user", "content": "Classify: <text>"}],
    "max_tokens": 8
  }'

Benefits in practice: fire ten subagent requests against @agents while a @main conversation is mid-decode, and the main conversation's lane is never chosen as the eviction victim — the subagents recycle each other's lanes first. Subagents sharing a system+tools prefix restore most of it from the host-RAM cache instead of re-prefilling, so time-to-first-token drops from full-prefill latency (hundreds of ms to multiple seconds, depending on prompt length) to the cost of re-hashing and copying a cached prefix (tens to a couple hundred ms). Untagged requests keep working exactly as before, classified as @agents.

Capabilities

All three registered model IDs support:

• text generation with thinking and non-thinking prompt modes;
• image, multi-image, video, and mixed multimodal messages;
• chunked prefill and CUDA Graph decode;
• startup-bounded small-scale concurrent serving with true batched decode;
• fixed or signal-adaptive MTP speculative decoding with draft windows from one to eight on 27B targets and one to five on 35B-A3B;
• BF16 and INT8 group-64 KV cache;
• model- and thinking-mode-aware official sampling defaults, with explicit greedy, temperature, top-k, top-p, min-p, and presence/frequency-penalty overrides;
• compatible-prefix reuse;
• OpenAI Responses Core, OpenAI Chat Completions, and Anthropic Messages, including streaming and usage accounting;
• prompt-rendered function tools and parsed tool calls.

The 35B-A3B target additionally supports text-only DFlash speculative decoding with draft windows from one to fifteen.

Current limits

• Only the five (model_id, weights_id) artifact identities listed above are accepted product identities.
• Execution is specialized for one RTX 5090 and one CUDA device.
• One Engine owns one resident model and supports a startup-fixed capacity of 1–8 active requests. Decode-ready requests are compacted at round boundaries and executed in one batched model traversal.
• NInfer does not provide large-scale or preemptive continuous batching, priority/QoS scheduling, multi-GPU execution, CPU/GPU offload, or distributed serving.
• --max-context is the logical ceiling of each sequence and is configurable up to the registered models' per-dtype context envelope: 524,288 tokens with bf16/int8 KV and 1,048,576 with hq-e8-2b (YaRN --rope-scaling yarn:F past the checkpoint's trained 262,144 positions). --kv-capacity N explicitly sizes the shared Main Text KV pool for all active and retained sequences, while --kv-capacity auto selects the largest usable capacity from the memory remaining after weights are loaded while preserving 1 GiB of sizing headroom. Omission defaults to one --max-context worth of pages. The resolved pool is fixed at startup and is not divided statically among request lanes.
• Tool calls are parsed and returned to the client; NInfer does not execute tools.
• The C++ headers are used by the in-tree applications and are not distributed as an installed SDK.

Documentation

• Contributing
• Documentation index
• CLI
• HTTP serving
• Performance
• Windows
• CLI examples

License

NInfer is licensed under the Apache License 2.0.

The published artifacts are derived from Qwen/Qwen3.6-27B, Qwen/Qwen3.8-27B, and Qwen/Qwen3.6-35B-A3B. The Qwen3.6-27B NVFP4 artifact also uses the fixed packed weights from rdtand/Qwen3.6-27B-PrismaSCOUT-Blackwell-NVFP4-BF16-vllm. The Qwen3.8-27B NVFP4 artifact also uses the fixed mixed FP8/NVFP4 weights from unsloth/Qwen3.8-27B-NVFP4. These source repositories are distributed under Apache-2.0. Vendored dependencies retain their own license files under third_party/.