Safe, ergonomic and complete Rust bindings for llama.cpp.

---

Installation

# Rust crate
cargo add llama-crab

# HTTP server (optional)
cargo install llama-crab-server --features mtmd --force

For backend selection (Metal, CUDA, Vulkan, ROCm, OpenCL, ...), see the installation guide.

Quickstart

use llama_crab::{Llama, LlamaParams};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut llama = Llama::load(
        LlamaParams::new("models/model.gguf")
            .with_n_ctx(2048)
            .with_n_gpu_layers(99),
    )?;

    let response = llama.create_completion("The capital of France is", 32)?;
    println!("{}", response.text);

    Ok(())
}

Runnable end-to-end examples live in the separate llama-crab-examples repo.

What is in the box

• Safe high-level API for text completion, chat completion, infill and embeddings.
• Low-level FFI bindings to llama.cpp, ggml, gguf and mtmd.
• Sampling chains, grammar-constrained decoding and JSON-Schema to GBNF conversion.
• Tool-call parsing for ChatML, Mistral, Llama 3, Functionary and plain JSON.
• Multimodal support (vision and audio) through mtmd.
• Hardware backends for CPU, Metal, CUDA, Vulkan, ROCm, OpenCL and KleidiAI.
• HTTP server (llama-crab-server) and Tauri plugin (tauri-plugin-llama-crab).
• TypeScript contracts and client (@llama-crab/core, @llama-crab/tauri).

Crates and Packages

Documentation

• Official documentation — installation, examples, server, multimodal and Tauri guides.
• API reference (Rust) — published rustdoc for the current release.
• API reference (latest) — rustdoc for main / latest version.
• Examples — guide for the runnable example crates.
• Examples repository — runnable end-to-end projects.
• Troubleshooting — common build and runtime issues.

The documentation site is published at https://llama-crab.nlp.rocks/.

Contributing

Contributions are welcome. Read CONTRIBUTING.md before opening a pull request and follow the Code of Conduct. Bug reports and security issues are tracked through GitHub Issues and SECURITY.md.

Clone with submodules:

git clone --recursive https://github.com/DominguesM/llama-crab.git
cd llama-crab

License

Licensed under the MIT License.

llama-crab builds on llama.cpp.