bitnet-llm

Safe Rust bindings to Microsoft's BitNet b1.58 inference engine (bitnet.cpp).

Provides a clean, ergonomic API with both full-response and streaming inference, multi-turn chat with incremental KV cache, targeting macOS (Apple Silicon) and Linux (x86-64).

---

Installation

Add to your Cargo.toml:

[dependencies]
bitnet-llm = "1.0.3"

Note: the first build compiles bitnet.cpp from source via CMake which takes a few minutes. Subsequent builds use the cached output.

---

Prerequisites

macOS (Apple Silicon) only: Xcode Command Line Tools (xcode-select --install).

---

Model setup

There are two options for obtaining the model.

Option 1 — Pre-packaged GGUF (simplest)

Download the pre-converted GGUF directly from Hugging Face:

hf download microsoft/BitNet-b1.58-2B-4T-gguf \
    --local-dir models/BitNet-b1.58-2B-4T

The model file is models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf.

Note: this file is missing pre-tokenizer metadata which triggers a warning at load time. Output quality on Linux x86 has been verified to be correct despite this warning.

Option 2 — Convert from BF16 weights (a must for ARM systems)

The converted GGUF preserves full tokenizer metadata:

# Install conversion dependencies
pip install huggingface_hub numpy torch

# Download the BF16 master weights (~5 GB)
hf download microsoft/bitnet-b1.58-2B-4T-bf16 \
    --local-dir models/bitnet-b1.58-2B-4T-bf16

# Clone the BitNet repository for the conversion script
git clone https://github.com/microsoft/BitNet /tmp/bitnet
pip install -r /tmp/bitnet/requirements.txt

# Convert to GGUF format
python3 /tmp/bitnet/utils/convert-helper-bitnet.py \
    models/bitnet-b1.58-2B-4T-bf16

The output file is models/bitnet-b1.58-2B-4T-bf16/ggml-model-i2s-bitnet.gguf.

---

Build

macOS (Apple Silicon)

cargo build --release

Linux (x86-64)

cargo build --release

The build script auto-detects AVX-512 and enables the TL2 BitNet kernel if available, falling back to the TL1 kernel automatically. Build time is a few minutes on first run as bitnet.cpp is compiled from source. Subsequent builds use the cached CMake output.

---

Performance

Benchmarked on Hetzner Cloud using the BitNet-b1.58-2B-4T model with max_tokens = 512, TopP sampling.

TTFT includes prompt prefill. Short tok/s is averaged over responses under 30 tokens where prefill overhead dominates; long tok/s is averaged over responses of 100+ tokens where generation dominates.

---

Usage

Running the examples

# Single prompt streaming inference
cargo run --release --example inference_streaming -- \
    models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf \
    "What is the capital of France?"

# Single prompt, returns full response at once
cargo run --release --example inference_standard -- \
    models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf \
    "What is the capital of France?"

# Interactive multi-turn chat
cargo run --release --example chat -- \
    models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf \
    "You are a helpful assistant."

Library usage

Single-turn inference

use bitnet_llm::{
    init, suppress_warnings, ContextParams, GenerateParams,
    Model, ModelParams, SamplingStrategy,
};
use std::io::Write;

fn main() -> Result<(), bitnet_llm::Error> {
    init();
    suppress_warnings();

    let model = Model::load(
        "models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf",
        ModelParams::default(),
    )?;

    let mut session = model.session(ContextParams::default())?;

    let params = GenerateParams {
        max_tokens: 200,
        sampling: SamplingStrategy::TopP {
            temperature: 0.8,
            top_p: 0.95,
            seed: u32::MAX,
        },
    };

    // Streaming — callback fires for each token piece as it is produced
    session.generate_streaming(
        "<|start_header_id|>user<|end_header_id|>\n\
         What is the capital of France?<|eot_id|>\
         <|start_header_id|>assistant<|end_header_id|>\n",
        &params,
        |piece| {
            print!("{piece}");
            let _ = std::io::stdout().flush();
        },
    )?;

    // Or collect the full response at once
    let response = session.generate(
        "<|start_header_id|>user<|end_header_id|>\n\
         What is the capital of France?<|eot_id|>\
         <|start_header_id|>assistant<|end_header_id|>\n",
        &params,
    )?;
    println!("{response}");

    bitnet_llm::deinit();
    Ok(())
}

Multi-turn chat

Sessions track KV cache position across turns. Only new tokens are encoded on each turn — history is never re-encoded. This keeps subsequent turns fast regardless of conversation length.

use bitnet_llm::{
    init, suppress_warnings, ContextParams, GenerateParams,
    Model, ModelParams, SamplingStrategy,
};
use std::io::Write;

fn main() -> Result<(), bitnet_llm::Error> {
    init();
    suppress_warnings();

    let model = Model::load(
        "models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf",
        ModelParams::default(),
    )?;

    let mut session = model.session(ContextParams::default())?;
    let params = GenerateParams::default();

    // Turn 1 — BOS added automatically because kv_pos is 0
    session.generate_streaming(
        "<|start_header_id|>system<|end_header_id|>\n\
         You are a helpful assistant.<|eot_id|>\
         <|start_header_id|>user<|end_header_id|>\n\
         Hello!<|eot_id|>\
         <|start_header_id|>assistant<|end_header_id|>\n",
        &params,
        |piece| { print!("{piece}"); let _ = std::io::stdout().flush(); },
    )?;
    // Close the assistant turn in the KV cache
    session.encode("<|eot_id|>")?;

    // Turn 2 — only new tokens encoded, full history stays in KV cache
    session.generate_streaming(
        "<|start_header_id|>user<|end_header_id|>\n\
         How are you?<|eot_id|>\
         <|start_header_id|>assistant<|end_header_id|>\n",
        &params,
        |piece| { print!("{piece}"); let _ = std::io::stdout().flush(); },
    )?;
    session.encode("<|eot_id|>")?;

    // Start a fresh conversation on the same session
    session.reset();

    bitnet_llm::deinit();
    Ok(())
}

---

API overview

ModelParams

ContextParams

GenerateParams

---

Important notes

CPU only. BitNet's ternary lookup-table kernels only run on the CPU. The ARM TL1 kernel is used on Apple Silicon and the x86 TL2 kernel on Linux. GPU offloading is not supported.

Single-token processing. The BitNet kernels produce valid output only when processing one token at a time. This library handles this correctly internally.

Chat template. This model uses the Llama 3 chat template with <|start_header_id|>, <|end_header_id|>, and <|eot_id|> control tokens. See the examples for complete implementations.

Context window. The model has a 4096 token context window. The library warns at 80% usage and returns an error when full. Call session.reset() to start a fresh conversation.

Multi-turn correctness. Always call session.encode("<|eot_id|>") after each assistant turn. Always call session.reset() between separate conversations. Do not call session.encode() before the first generate_streaming call on a fresh session as this will prevent BOS from being added.

Generation. The pre-packaged GGUF (Option 1) produces word-salad output on ARM. Use Option 2 (convert from BF16) on ARM Linux.