Anbeeld's BeeLlama.cpp — CMP 90HX Edition

BeeLlama.cpp (or just Bee) is a performance-focused llama.cpp fork for squeezing more speed and context out of local GGUF inference. It adds variance-normalized KV-cache quantization (KVarN), KV cache precision tail for recent tokens, low-bit cache types, adaptive draft control for speculative decoding, reasoning-loop protection, and more.

This fork additionally patches the CUDA backend for NVIDIA CMP 90HX (GA102, sm_86), replacing firmware-throttled instructions (DP4A, FP32 FFMA) with unthrottled equivalents (IMAD, HFMA2).

Not quite a pegasus, but close enough.

---

CMP 90HX Optimizations

The CMP 90HX (GA102, sm_86) has a firmware that throttles certain CUDA instructions:

Applied patches

Cumulative gain vs unpatched: ~+58% on Q4_K models, ~+87% on dense Q4_K_XL models.

Benchmark data in bench/cmp90hx/. Future patch roadmap in FUTURE_PATCHES.md.

Build for CMP 90HX

cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 \
  -DGGML_CUDA_FA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON \
  -DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(nproc) --target llama-bench llama-server

---

Fork Features

• Variance-normalized KV-cache quantization (KVarN): provides higher precision at similar memory costs. Independent K and V bit widths at kvarn2, kvarn3, kvarn4, kvarn5, kvarn6, and kvarn8, set with --cache-type-k and --cache-type-v.
• KV cache precision tail: keep most of the KV cache quantized while storing recent tokens in F16/BF16, enabled with --kv-tail-tokens. A single global softmax merges the quantized body and the precision tail under FlashAttention, without materializing the whole cache.
• Standard low-bit KV cache types: q2_0, q2_1, q3_0, q3_1, q6_0, and q6_1, usable for either target or draft caches alongside the upstream q4/q5/q8 types.
• Adaptive draft-max for DFlash: adjusts the active DFlash draft horizon at runtime instead of using a fixed --spec-draft-n-max, comparing speculative throughput against a no-spec baseline.
• Reasoning-loop protection: the server detects repeated hidden reasoning output and intervenes.

For the full feature and public-repo comparison, read docs/beellama-features.md. For the complete argument reference, read docs/beellama-args.md.

KV Cache Quantization

K and V cache types are set independently with --cache-type-k and --cache-type-v. The research is covered in articles: KVarN KV Cache: Implementation and Benchmarks, KV Cache Precision Tail: Implementation and Benchmarks, and KV Cache Quantization Benchmarks: KVarN, Precision Tail.

KV Cache Recommenation Ladder

The measurements come from Qwen 3.6 27B Q5_K_S at 64K context on Wikitext-2 raw with -b 2048 -ub 512 on an RTX 3090. Median KLD is the primary quality metric; lower is better.

A 1024-token tail is a useful starting point for low-bit Qwen caches. Prefer more body precision when old and recent tokens matter equally; consider 2048 only when the newest two thousand tokens are genuinely the privileged working set. Standard caches generally have slightly faster prefill.

Gemma 4 needs a separate decision. Its 1024-token sliding window makes a 1024 tail exact across most layers, sharply changing memory and throughput. Standard q8_0 / q8_0 without a tail is the safer Gemma default when throughput and older-context coverage matter, and the benchmark results recommend avoiding quantized KV cache when Gemma quality is non-negotiable.

Standard-Only KV Cache Ladder

Use this generic fallback ladder when KVarN or precision tails are unavailable. It is based on the same Qwen 3.6 27B benchmark, with every standard cache row using tail 0.

KV Cache Type Reference

All KV cache types available in BeeLlama

Type	Origin	bpv	Size vs bf16
q8_0	upstream	8.5	53.1%
kvarn8	Huawei / fork	8.375	52.3%
q6_1	fork	7.0	43.8%
q6_0	fork	6.5	40.6%
kvarn6	Huawei / fork	6.375	39.8%
q5_1	upstream	6.0	37.5%
q5_0	upstream	5.5	34.4%
kvarn5	Huawei / fork	5.375	33.6%
q4_1	upstream	5.0	31.3%
q4_0	upstream	4.5	28.1%
iq4_nl	upstream	4.5	28.1%
kvarn4	Huawei / fork	4.375	27.3%
q3_1	fork	4.0	25.0%
q3_0	fork	3.5	21.9%
kvarn3	Huawei / fork	3.375	21.1%
q2_1	fork	3.0	18.8%
q2_0	fork	2.5	15.6%
kvarn2	Huawei / fork	2.375	14.8%

Standard ratios come directly from each block format. KVarN ratios describe the compressed record body, including scale and zero-point metadata; the permanent exact sink, exact suffix, staging, and alignment add overhead.

Installation

Prebuilt

Current release binaries are on the releases page.

Windows CUDA archives contain a ggml-cuda.dll backend; download the matching beellama-<version>-cudart-win-cuda-*-x64.zip runtime archive and extract it into the same folder. Windows SYCL and HIP archives ship as standalone packages with all required runtime DLLs bundled.

Docker images are published to ghcr.io/anbeeld/beellama.cpp:

Building from source with -DGGML_NATIVE=ON may result in a tiny bit better performance, so it might still be a good idea to do that if/when you decide to use this fork long-term.

CUDA Build

# Linux (GCC + CUDA)
cmake -B build -DGGML_CUDA=ON -DGGML_NATIVE=ON \
  -DGGML_CUDA_FA=ON \
  -DCMAKE_BUILD_TYPE=Release
cmake --build build -j

# Windows (MSVC + CUDA)
cmake -B build -DGGML_CUDA=ON -DGGML_NATIVE=ON ^
  -DGGML_CUDA_FA=ON ^
  -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release --parallel

# macOS (Metal)
cmake -B build -DGGML_METAL=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build -j

The default CUDA FlashAttention build covers 50 standard cache pairs and 15 KVarN fast-decode pairs, including the homogeneous F16 and BF16 pairs needed by precision tails. Add -DGGML_CUDA_FA_ALL_QUANTS=ON to compile all 169 standard and 36 KVarN pairs, or -DGGML_CUDA_KVARN=OFF to build without KVarN kernels.

Other Backends

Bee inherits llama.cpp backend support, including Metal, HIP, Vulkan, SYCL, BLAS, CANN, MUSA, OpenVINO, OpenCL, and RPC. Use the upstream-style build docs in docs/build.md and backend-specific pages under docs/backend.

Common Commands

Local CLI

llama-cli -m model.gguf
llama-cli -m model.gguf -cnv --chat-template chatml
llama-cli -m model.gguf -n 256 --grammar-file grammars/json.gbnf -p "Request: schedule a call at 8pm; Command:"

OpenAI-Compatible Server

llama-server -m model.gguf --port 8080
llama-server -m model.gguf -c 16384 -np 4
llama-server -m model.gguf -md draft.gguf

DFlash Speculative Decoding

llama-server -m target.gguf --spec-type draft-dflash \
  --spec-draft-model drafter.gguf \
  --spec-draft-ngl all \
  --spec-dm-controller profit \
  --flash-attn on --cache-type-k q5_0 --cache-type-v q4_1

Keep the draft context on a standard cache type; KVarN is target-cache only.

KVarN Target Cache

# Balanced general starting point from the benchmark ladder
llama-server -m model.gguf --flash-attn on \
  --cache-type-k kvarn5 --cache-type-v kvarn4 \
  --kv-tail-tokens 1024

Router Mode With Presets

llama-server --models-dir /path/to/models
llama-server --models-preset presets.ini

Documentation

• BeeLlama features and public repo diff
• BeeLlama args reference
• Build docs
• Server docs
• Docker docs
• Performance troubleshooting

Contributing

Keep PRs small and scoped. Run the narrowest relevant tests or benchmarks before opening a PR, and include the exact commands. For fork-specific changes, update the corresponding docs when behavior or args change.

Read CONTRIBUTING.md for inherited llama.cpp contribution conventions and this fork's AI usage policy.

Dependencies

• yhirose/cpp-httplib - HTTP client/server library used by llama-server - MIT
• stb-image - single-header image decoder used by multimodal code - public domain
• nlohmann/json - single-header JSON library - MIT
• miniaudio.h - single-header audio decoder - public domain
• subprocess.h - process launching helper - public domain
• Intel OpenVINO - frontend header used in OpenVINO backend (ggml/src/ggml-openvino/openvino/frontend.h) - Apache-2.0
• Intel SYCL/oneAPI - SYCL backend (ggml/src/ggml-sycl/) - Apache-2.0 WITH LLVM-exception

See the licenses/ directory for full license texts.