llama.cpp

[!IMPORTANT] This is the PrismML fork of llama.cpp, the main line behind the Bonsai models (branch prism, developed as prism-v7). It tracks current mainline llama.cpp and adds the fork's low-bit formats and runtime features on top.

New here? Start with the Bonsai-demo repo. It downloads the right models and the correct prebuilt binaries for your hardware/backend automatically.

Which ternary model file to use:

• *-PQ2_0.gguf (fork group-128, ggml id 142): preferred on Metal, CUDA, HIP and CPU. About 6% smaller than group-64.
• *-Q2_0_g64.gguf / 27B *-Q2_g64.gguf (official group-64, ggml id 42): runs on every backend here AND on mainline llama.cpp. If unsure, use this. Newer model releases name this file plain *-Q2_0.gguf.
• *-Q2_0.gguf on OLDER model repos is the deprecated legacy format (group 128 stored as id 42). It does not load on these builds; the error tells you which file to get instead. If you must run it, use the frozen prism-v5 line and its final release prism-b9601.

Speculative decoding (dspark) is supported via mainline's draft-dspark plus fork patches. Drafters published for older model releases need a one-time conversion with gguf-dspark-to-dflash (see SPECULATIVE.md in Bonsai-demo); newer releases ship ready-to-use drafters.

Do NOT build from prism-v6 (stale mid-migration snapshot) and do NOT mix this fork's ggml-* libraries with a stock llama.cpp build.

[!NOTE] This repository is a fork of the PrismML fork above. It is prism-ml llama.cpp plus one addition: the TurboQuant KV-cache quantization family, backported from AmesianX/TurboQuant. Everything from PrismML-Eng/llama.cpp (branch prism) is unchanged; the TurboQuant sources are vendored in-tree, so no submodule is needed to build.

What is added on top of prism-ml llama.cpp:

• New ggml types GGML_TYPE_TBQ3_0 … TBQP4_4, AMX3_1/AMXV3_1 (ids 43–64) with quant/dequant codecs, CPU type traits and ops.
• KV-cache selection: -ctk/-ctv tbq3 | tbq4 | tbqp3 | tbqp4 | amx3, with the _0/_1/_2/_3 block-size variant picked automatically from head_dim. External attn_rot_k/v rotation is disabled for these types (the codec applies its own WHT). The speculative draft cache accepts -ctk tbqp3 too.
• CUDA FlashAttention: K/V dequant and in-kernel quantizers (set_rows), plus FA vec/tile/mma kernels for 45 K-V type combinations (incl. all tbq*/tbqp* pairs) in GGML_CUDA_FA_QUANTS — quantized caches run under flash attention without an f16 fallback.
• Verified on *-PQ2_0.gguf (head_dim 256): -ctk tbq3 -ctv tbq3 → K (tbq3_0) = V (tbq3_0) = 24.50 MiB per 4096-cell cache, flash_attn = enabled, correct generation.

---

Quick start

A few options to get llama.cpp installed on your machine:

• Visit https://llama.app and follow the instructions
• Run with Docker - see our Docker documentation
• Download pre-built binaries from the releases page
• Build from source by cloning this repository - check out our build guide

Once installed:

# Download and run a model directly from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF

# Launch OpenAI-compatible API server
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

Description

The main goal of llama.cpp is to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.

• Plain C/C++ implementation without any dependencies
• Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
• AVX, AVX2, AVX512 and AMX support for x86 architectures
• RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures
• 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
• Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
• Vulkan and SYCL backend support
• CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity

The llama.cpp project is build on top of the ggml library.

Supported backends

Documentation

Tools

• cli
• completion
• server
• GBNF grammars

Development

• How to build
• Running on Docker
• Build on Android
• Multi-GPU usage
• Performance troubleshooting
• GGML tips & tricks
• XCFramework
• Completions
• Models
• Release process

Contributing

• Contributors can open PRs
• Collaborators will be invited based on contributions
• Maintainers can push to branches in the llama.cpp repo and merge PRs into the master branch
• Any help with managing issues, PRs and projects is very appreciated!
• Read the CONTRIBUTING.md for more information

Acknowledgements

• yhirose/cpp-httplib - Single-header HTTP server, used by llama-server - MIT license
• nothings/stb - Single-header image format decoder, used by multimodal subsystem - Public domain
• nlohmann/json - Single-header JSON library, used by various tools/examples - MIT License
• mackron/miniaudio - Single-header audio format decoder, used by multimodal subsystem - Public domain
• sheredom/subprocess.h - Single-header process launching solution for C and C++ - Public domain