llama.cpp

Maple Preview — CPU Setup

Build llama.cpp

On Debian/Ubuntu ARM, install GCC/G++ if needed:

sudo apt update
sudo apt install -y build-essential

git clone git@github.com:deepgrove-ai/llama.cpp.git
cd llama.cpp

rm -rf ./build

uvx cmake -B build \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_METAL=OFF

uvx cmake --build build -j

macOS uses Apple Clang. uvx provides CMake, so no separate CMake installation is needed.

Download the GGUF

uvx --from huggingface_hub hf download \
  deepgrove/maple-preview-GGUF \
  maple-preview-TQ2_0-head-Q4_K.gguf \
  --local-dir models

Hugging Face: deepgrove/maple-preview-GGUF

TQ1_0 and TQ2_0 are different ternary packing schemes. Use TQ2_0 for generally faster speeds but slightly higher memory footprint.

Start chatting

./build/bin/llama-completion \
  -m models/maple-preview-TQ2_0-head-Q4_K.gguf \
  --threads 16 \
  --temp 1.0 \
  --top-p 0.95 \
  --jinja \
  --conversation

--jinja applies Maple's embedded chat template exactly, including its thinking prefix.

Benchmark

M5 Pro, CPU-only, 16 threads, 512 prompt tokens, 128 generated tokens, 3 repetitions:

Adjust -t for #of threads.

FP16 LM head:

./build/bin/llama-bench \
  -m models/maple-preview-TQ2_0-head-F16.gguf \
  -p 512 \
  -n 128 \
  -r 3 \
  -t 16 \
  -dev none \
  -ngl 0

Q4_K LM head:

./build/bin/llama-bench \
  -m models/maple-preview-TQ2_0-head-Q4_K.gguf \
  -p 512 \
  -n 128 \
  -r 3 \
  -t 16 \
  -dev none \
  -ngl 0

Quick start

A few options to get llama.cpp installed on your machine:

• Visit https://llama.app and follow the instructions
• Run with Docker - see our Docker documentation
• Download pre-built binaries from the releases page
• Build from source by cloning this repository - check out our build guide

Once installed:

# Download and run a model directly from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF

# Launch OpenAI-compatible API server
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

Description

The main goal of llama.cpp is to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.

• Plain C/C++ implementation without any dependencies
• Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
• AVX, AVX2, AVX512 and AMX support for x86 architectures
• RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures
• 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
• Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
• Vulkan and SYCL backend support
• CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity

The llama.cpp project is build on top of the ggml library.

Supported backends

Documentation

Tools

• cli
• completion
• server
• GBNF grammars

Development

• How to build
• Running on Docker
• Build on Android
• Multi-GPU usage
• Performance troubleshooting
• GGML tips & tricks
• XCFramework
• Completions
• Models

Contributing

• Contributors can open PRs
• Collaborators will be invited based on contributions
• Maintainers can push to branches in the llama.cpp repo and merge PRs into the master branch
• Any help with managing issues, PRs and projects is very appreciated!
• Read the CONTRIBUTING.md for more information

Acknowledgements

• yhirose/cpp-httplib - Single-header HTTP server, used by llama-server - MIT license
• stb-image - Single-header image format decoder, used by multimodal subsystem - Public domain
• nlohmann/json - Single-header JSON library, used by various tools/examples - MIT License
• miniaudio.h - Single-header audio format decoder, used by multimodal subsystem - Public domain
• subprocess.h - Single-header process launching solution for C and C++ - Public domain