llama.cpp-mirai-s

llama.cpp with support for Mirai S, Mirai Labs' 2.4-bit Qwen3.8-27B. The compressed trellis codes load from a GGUF bit for bit and run on CUDA, and the model fits a 12 GB GPU with 128K context. The fork also speeds up long-context decode with a q8_0/q4_0 KV cache, adds a chunked DeltaNet prefill kernel, and keeps the prompt cache when an agent's history re-tokenizes differently. With a 1.3 MB vector file it can also remove the model's refusals at run time (--cvec-mode project).

This is the reference fork: close to upstream, and the one the GGUF was verified on. For daily use on a 12 GB card look at mirai-s-ada by Cary Palmer first. It ports this codec to a serving engine with the full 262K window at q8_0 KV and MTP drafting, and it ships a Windows bundle. If that project changes or breaks, this fork still runs the same file.

• Build, run, formats, numbers: tools/mirai-s/README.md
• GGUF, vision mmproj and the refusal vector: alesha-pro/Qwen3.8-27B-S-mirai-GGUF
• Base: ggml-org/llama.cpp at d834d44e6. Everything below this line is the upstream README.

---

llama.cpp

Quick start

A few options to get llama.cpp installed on your machine:

• Visit https://llama.app and follow the instructions
• Run with Docker - see our Docker documentation
• Download pre-built binaries from the releases page
• Build from source by cloning this repository - check out our build guide

Once installed:

# Download and run a model directly from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF

# Launch OpenAI-compatible API server
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

Description

The main goal of llama.cpp is to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.

• Plain C/C++ implementation without any dependencies
• Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
• AVX, AVX2, AVX512 and AMX support for x86 architectures
• RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures
• 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
• Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
• Vulkan and SYCL backend support
• CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity

The llama.cpp project is build on top of the ggml library.

Supported backends

Documentation

Tools

• cli
• completion
• server
• GBNF grammars

Development

• How to build
• Running on Docker
• Build on Android
• Multi-GPU usage
• Performance troubleshooting
• GGML tips & tricks
• XCFramework
• Completions
• Models
• Release process

Contributing

• Contributors can open PRs
• Collaborators will be invited based on contributions
• Maintainers can push to branches in the llama.cpp repo and merge PRs into the master branch
• Any help with managing issues, PRs and projects is very appreciated!
• Read the CONTRIBUTING.md for more information

Acknowledgements

• yhirose/cpp-httplib - Single-header HTTP server, used by llama-server - MIT license
• nothings/stb - Single-header image format decoder, used by multimodal subsystem - Public domain
• nlohmann/json - Single-header JSON library, used by various tools/examples - MIT License
• mackron/miniaudio - Single-header audio format decoder, used by multimodal subsystem - Public domain
• sheredom/subprocess.h - Single-header process launching solution for C and C++ - Public domain