llama.cpp

Quick start

This fork adds inference support for the internally developed Spark2_5ForCausalLM model. The following commands build llama.cpp, convert a local Hugging Face checkpoint to GGUF, and run it on CPU or an NVIDIA GPU.

1. Build

For an NVIDIA CUDA build:

cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j 8

For a CPU-only build, use -DGGML_CUDA=OFF instead. The CUDA build also contains the CPU backend, so the same binaries can be used for both examples below.

2. Convert the Spark2_5 checkpoint to GGUF

Install the Python conversion dependencies:

python -m pip install -r requirements.txt

The Spark2_5 checkpoint stores its tokenizer under v8_2_token. The converter expects the tokenizer files next to config.json and the model .safetensors files, so copy them to the checkpoint root before conversion:

mkdir -p models
cp /path/to/spark2_5-hf/v8_2_token/{tokenizer.json,tokenizer_config.json,merges.txt} \
    /path/to/spark2_5-hf/

python convert_hf_to_gguf.py /path/to/spark2_5-hf \
    --outfile models/spark2_5-1.7b-bf16.gguf \
    --outtype bf16

3. Run inference

CPU:

./build/bin/llama-completion \
    -m models/spark2_5-1.7b-bf16.gguf \
    -ngl 0 -t 16 -c 1024 \
    -cnv -st --jinja --simple-io --no-display-prompt \
    -p '请用三句话解释什么是计算图。' \
    -n 96 --temp 0 --seed 1

NVIDIA GPU (GPU 0):

CUDA_VISIBLE_DEVICES=0 ./build/bin/llama-completion \
    -m models/spark2_5-1.7b-bf16.gguf \
    -ngl 99 -t 16 -c 1024 \
    -cnv -st --jinja --simple-io --no-display-prompt \
    -p '请用三句话解释什么是计算图。' \
    -n 96 --temp 0 --seed 1

-ngl 99 offloads all Spark2_5 layers to the selected GPU. Use an integer for -ngl; values such as all are not accepted by llama-bench.

4. Verify the architecture and GPU backend

./build/bin/test-llama-archs -a spark2_5

CUDA_VISIBLE_DEVICES=0 ./build/bin/llama-bench \
    -m models/spark2_5-1.7b-bf16.gguf \
    -ngl 99 -p 32 -n 8

The architecture test should report OK for the CPU and CUDA backends. A Roundtrip: SKIP result is expected because model-saver roundtrip support is currently disabled for Spark2_5.

Description

The main goal of llama.cpp is to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.

• Plain C/C++ implementation without any dependencies
• Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
• AVX, AVX2, AVX512 and AMX support for x86 architectures
• RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures
• 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
• Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
• Vulkan and SYCL backend support
• CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity

The llama.cpp project is build on top of the ggml library.

Supported backends

Documentation

Tools

• cli
• completion
• server
• GBNF grammars

Development

• How to build
• Running on Docker
• Build on Android
• Multi-GPU usage
• Performance troubleshooting
• GGML tips & tricks
• XCFramework
• Completions
• Models
• Release process

Contributing

• Contributors can open PRs
• Collaborators will be invited based on contributions
• Maintainers can push to branches in the llama.cpp repo and merge PRs into the master branch
• Any help with managing issues, PRs and projects is very appreciated!
• Read the CONTRIBUTING.md for more information

Acknowledgements

• yhirose/cpp-httplib - Single-header HTTP server, used by llama-server - MIT license
• nothings/stb - Single-header image format decoder, used by multimodal subsystem - Public domain
• nlohmann/json - Single-header JSON library, used by various tools/examples - MIT License
• mackron/miniaudio - Single-header audio format decoder, used by multimodal subsystem - Public domain
• sheredom/subprocess.h - Single-header process launching solution for C and C++ - Public domain