NInfer — cometkim fork

About this fork

This is cometkim's personal fork of NInfer.

Selected checkpoints. Maximum single-GPU inference performance.

NInfer is a from-scratch C++/CUDA inference engine for supported Qwen architectures on a single NVIDIA GeForce RTX 5090. It runs text, image, and video prompts through a local CLI or OpenAI-/Anthropic-compatible HTTP APIs.

NInfer implements Qwen3.5 Dense and MoE architectures through explicit native model/Op paths, not arbitrary model graphs. These five official recipes are available:

The current engine requires v3 .ninfer: configuration, encoded weights, logical bindings and frontend resources select execution, not a fixed (model_id, weights_id) registry. Custom recipes can choose supported format/shape combinations without adding checkpoint-specific engine identities. Downloaded v2 files need an offline upgrade; published download checksums do not describe upgraded files.

Upstream lineage

NInfer is developed by Neroued/ninfer. The Windows port and WebUI originate from natpate/ninfer-windows.

Performance

These are upstream-published RTX 5090 measurements. The performance index links to per-model run records and the measurement rules. The tables below are excerpts from those detailed results.

Concurrent MTP3 decode

Saturated decode used INT8 group-64 KV, CUDA Graphs, MTP3, and one 8,192-token generation per active request. Throughput uses aggregate committed decode tokens from complete intervals whose actual decode batch equaled the configured concurrency. Acceptance covers the complete request wave; these rates are steady decode (tok/s).

Single-request serving

The serial serving corpus used INT8 group-64 KV, CUDA Graphs, a 1,024-token prefill chunk, and five fixed seeds after warm-up. The table keeps one short-prefill, one extreme-prefill, and one structured-output MTP3 point for each published profile; the full context and scenario matrices are linked from each model below.

Evaluation

Capability scores were measured through NInfer's OpenAI-compatible serving route with thinking enabled, MTP=3, and EvalScope 1.9.0 (0-shot, rule scoring, one sample per problem):

The Qwen3.6 rows used temperature 0.6 and presence penalty 1.0; the Qwen3.8-27B rows used temperature 1.0 and presence penalty 0.0. The multimodal columns (ERQA and RealWorldQA) ran with --vision at a 81,920-token context limit; the text columns used a 262,144-token limit except Qwen3.8-27B NVFP4, which needs 252,928 to fit the RTX 5090 after weights.

These are single-sample results under that NInfer evaluation profile, not pass@k. See the model cards and full performance document for correct/total counts and evaluation notes.

Requirements

NInfer currently requires:

• 64-bit Linux or Windows 11 x64;
• NVIDIA GeForce RTX 5090 (sm_120a);
• NVIDIA driver support for CUDA 13.1 and the CUDA Toolkit 13.1 or newer;
• CMake 3.28 or newer and a C++20-capable host compiler (GCC or Clang on Linux, MSVC from Visual Studio 2022 on Windows);
• FFmpeg development libraries: libavformat >= 60, libavcodec >= 60, libavutil >= 58, and libswscale >= 7;
• libcurl >= 7.85;
• pkg-config on Linux, or vcpkg on Windows (the repository pins the dependency baseline in vcpkg.json);
• Ninja, when using the commands below.

The build rejects CUDA architectures other than 120a. Run NInfer from its source build tree.

Prebuilt Windows release

Prebuilt packages must match the engine and artifact version you intend to run. See the Windows guide for setup and build instructions, and Download a model for artifact compatibility.

Build

Linux

git clone --branch cometkim/dev https://github.com/cometkim/ninfer.git
cd ninfer

cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build -j

The default configuration builds:

build/apps/ninfer
build/apps/ninfer-serve

Tests, benchmarks, and maintainer tools are excluded from the default build.

Windows

See the Windows guide for Visual Studio, CUDA, and vcpkg setup and native Windows build instructions.

Docker

Build the runtime image on a 64-bit Linux host with an RTX 5090, a CUDA 13.1-compatible NVIDIA driver, Docker, and the NVIDIA Container Toolkit.

docker build --tag ninfer:local .

Download a model into models/ as described below, then run the HTTP server:

docker run --rm \
  --gpus '"device=0"' \
  --publish 8080:8080 \
  --volume "$PWD/models:/models:ro" \
  ninfer:local \
  ninfer-serve /models/qwen3_6_27b.ninfer \
  --host 0.0.0.0

Run the CLI from the same image:

docker run --rm \
  --gpus '"device=0"' \
  --volume "$PWD/models:/models:ro" \
  ninfer:local \
  ninfer /models/qwen3_6_27b.ninfer \
  --prompt "Explain prefill and decode in three sentences." \
  --max-new 256

Download a model

Use the Hugging Face CLI to download an official artifact:

hf download neroued/Qwen3.6-27B-NInfer \
  qwen3_6_27b.ninfer \
  --local-dir models

# Or the 27B NVFP4 weight variant:
hf download neroued/Qwen3.6-27B-nvfp4-NInfer \
  qwen3_6_27b_nvfp4.ninfer \
  --local-dir models

# Or Qwen3.8-27B:
hf download neroued/Qwen3.8-27B-NInfer \
  qwen3_8_27b.ninfer \
  --local-dir models

# Or Qwen3.8-27B NVFP4:
hf download neroued/Qwen3.8-27B-nvfp4-NInfer \
  qwen3_8_27b_nvfp4.ninfer \
  --local-dir models

# Or:
hf download neroued/Qwen3.6-35B-A3B-NInfer \
  qwen3_6_35b_a3b.ninfer \
  --local-dir models

Current builds accept only v3 containers. Existing official v2 downloads can be upgraded locally without downloading the weights again. Filenames alone do not identify the container version.

Each .ninfer file contains the weights and frontend resources needed by NInfer. It is not a Transformers checkpoint, Safetensors distribution, or GGUF file.

Artifacts contain only the components selected at conversion; GPU residency is fixed at startup. Speculative decoding is disabled by default, so MTP/DFlash state and the optimized proposal head are not uploaded. Vision is also disabled by default, so its weights, Vision scratch phase, and frozen request-transient allocation are omitted. Add --vision to the CLI or server process that must accept image or video input. Disabled capabilities cannot be enabled by a later request. DFlash is available for 35B-A3B and DFlash2 for Qwen3.8-27B artifacts containing the companion component. Both can accelerate generated-text decode after Text or Vision prefill; Vision encoding itself is not speculative.

Run the CLI

./build/apps/ninfer models/qwen3_6_27b.ninfer \
  --prompt "Explain prefill and decode in three sentences." \
  --max-context 16384 \
  --max-new 256 \
  --spec mtp --draft-tokens 3 \
  --lm-head-draft

Use --messages FILE instead of --prompt for chat history, images, or videos:

./build/apps/ninfer models/qwen3_6_27b.ninfer \
  --messages examples/cli/messages/image_chart.json \
  --max-context 8192 \
  --max-new 128 \
  --vision

Answer content is written to stdout. Loading progress, reasoning, timing, throughput, memory, and speculative-decoding statistics are written to stderr. See the CLI guide and committed examples for structured input and runtime options.

Run the HTTP server

./build/apps/ninfer-serve models/qwen3_6_27b.ninfer \
  --max-context 16384 \
  --kv-capacity auto \
  --max-concurrency 2 \
  --spec mtp --draft-tokens 3 \
  --lm-head-draft

The public model ID defaults to the artifact's model name; use --model-id only to publish a deployment-specific alias.

Then send an OpenAI-style request:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen3.6-27b",
    "messages": [{"role": "user", "content": "Reply with one short sentence."}],
    "max_tokens": 64
  }'

The server also implements OpenAI Responses Core (typed Items, semantic SSE, local continuation state, and function calls) plus Anthropic Messages, token counting, and multimodal input. See HTTP serving.

Capabilities

The supported model architectures provide these capabilities when corresponding artifact components are present and enabled at startup:

• text generation with thinking and non-thinking prompt modes;
• image, multi-image, video, and mixed multimodal messages;
• chunked prefill and CUDA Graph decode;
• startup-bounded small-scale concurrent serving with true batched decode;
• MTP speculative decoding;
• BF16, INT8 group-64, FP8, NVFP4, and K8V4 KV cache;
• offline causal-perplexity scoring;
• model- and thinking-mode-aware official sampling defaults, with explicit greedy, temperature, top-k, top-p, min-p, and presence/frequency-penalty overrides;
• exact-prefix reuse with Device/Host State and KV retention;
• OpenAI Responses Core, OpenAI Chat Completions, and Anthropic Messages, including streaming and usage accounting;
• prompt-rendered function tools and parsed tool calls.

35B-A3B supports DFlash and Qwen3.8-27B supports DFlash2 with companion weights, each with draft windows from one to fifteen and optional Vision residency.

Current limits

• Model architectures and format/shape combinations require implemented native paths; v3 custom recipes do not make the engine an arbitrary-model runtime.
• Execution is specialized for one RTX 5090 and one CUDA device.
• One Engine owns one resident model and supports a startup-fixed capacity of 1–8 active requests. Decode-ready requests are compacted at round boundaries and executed in one batched model traversal.
• NInfer does not provide large-scale or preemptive continuous batching, priority/QoS scheduling, multi-GPU execution, CPU/GPU offload, or distributed serving.
• --max-context is the logical ceiling of each sequence and is configurable up to the registered models' execution envelope. This is not a guarantee of VRAM fit or long-context quality. --kv-capacity N explicitly sizes the shared Main Text KV pool for all active and retained sequences, while --kv-capacity auto selects the largest usable capacity from the memory remaining after weights are loaded while preserving 1 GiB of sizing headroom. Omission defaults to one --max-context worth of pages. The resolved pool is fixed at startup and is not divided statically among request lanes.
• Tool calls are parsed and returned to the client; NInfer does not execute tools.
• The C++ headers are used by the in-tree applications and are not distributed as an installed SDK.

Documentation

• Contributing
• Documentation index
• CLI
• HTTP serving
• Performance
• Weight conversion and custom recipes
• Perplexity evaluation
• Resource scheduling and context cache
• Windows
• CLI examples

License

NInfer is licensed under the Apache License 2.0.

The published artifacts are derived from Qwen/Qwen3.6-27B, Qwen/Qwen3.8-27B, and Qwen/Qwen3.6-35B-A3B. The Qwen3.6-27B NVFP4 artifact also uses the fixed packed weights from rdtand/Qwen3.6-27B-PrismaSCOUT-Blackwell-NVFP4-BF16-vllm. The Qwen3.8-27B NVFP4 artifact also uses the fixed mixed FP8/NVFP4 weights from unsloth/Qwen3.8-27B-NVFP4. These source repositories are distributed under Apache-2.0. Vendored dependencies retain their own license files under third_party/.