Cera

A Rust-native LLM inference engine. Load a GGUF model and run it locally, on your laptop's CPU, an Apple GPU, a cross-platform Vulkan/DX12 GPU, a phone, or in the browser, from a single dependency-free core.

[!NOTE] The 0.7.0 release adds the native Qualcomm Hexagon NPU backend, dynamic CPU topology discovery and worker threadpool resizing, and core lifecycle enhancements. See the API guide for usage and compatibility limits, and Releases for published builds.

Why Cera

• No Python, no runtime. Pure Rust. The CLI is a single binary; the library has zero required system dependencies on a default build.
• Runs everywhere. The same core drives a desktop CLI, Android/iOS apps (via UniFFI), and the browser (via WebAssembly). Pick CPU or GPU at runtime.
• Loads standard GGUF. Point it at a .gguf file, a LeapBundles manifest, or a bundle id; it can auto-download and cache models from Hugging Face.
• Multimodal. Text, vision (image → text), and audio (in/out) models all load through the same session API.
• Wake word & speech recognition. Native Keyword Spotting (KWS), Silero VAD, OpenAI Whisper ASR, and the unified stateful AudioPipeline in pure Rust, linked through a single unified binary across mobile and desktop.
• Structured output. Constrain generation to a GBNF grammar, or one flag for grammar-constrained JSON (check for incomplete output on token limits).
• Tool calling. Give the model a set of tool schemas and parse the calls it makes back out: format-aware (LFM2 Pythonic, Hermes/Qwen JSON), with an optional grammar constraints on tool-call syntax. From the CLI and every binding.
• LoRA adapters & embeddings. Load LoRA adapters at runtime (GGUF or PEFT safetensors), hot-swap / unload per session (applied on CPU, Metal, and wgpu, never merged) and pull per-token hidden states out of the engine for classifier and embedding heads. From every binding.

Supported models

Dispatch is on the GGUF general.architecture string. Selected supported families are listed below; tensor layouts and quantization must also match the loader:

CPU loading also supports families such as Gemma 2/4, Olmo 2/3, Granite Hybrid, Falcon H1 and Mamba-2. See the loading dispatch for architecture aliases and backend admission rules.

Metal and wgpu support lfm2, lfm2moe, llama, qwen2, qwen3, granite, minicpm, minicpm5, nanbeige, phi3, phi and mistral3 / ministral3 (YaRN is folded into the rope kernel's frequency table; attention temperature scaling is not implemented on GPU, so those models are limited to the context below its floor, the original context length, and warn when that caps a longer request). Qualcomm Hexagon NPU supports lfm2, lfm2moe, the dense families (llama, qwen2, qwen3, granite, gemma2, olmo2/olmo3, mistral3, phi3 and others listed in docs/HEXAGON_NPU.md) and qwen35 (HexagonLfmModel); those paths are host-tested and still need device validation. The listed bailingmoe3 family uses CPU; requesting an unsupported GPU or NPU backend explicitly returns an error. Auto selection can fall back to CPU. The LFM2 and supported dense-transformer paths use batched-GEMM prefill, including on CPU, with tiled flash attention for long prompts where supported.

lfm2moe's routed feed-forward block is the one exception to that batched-GEMM sentence, and it is deliberate: each token picks its own 4-of-32 experts, so an expert's weights are re-read once per routed token rather than once per prompt. (The GPU backends still issue one dispatch per projection for the whole chunk, not one per token; what they do not do is share a weight row between two tokens that chose the same expert.) Grouping tokens by expert is a follow-up, kept separate from getting the routing arithmetic right. On the two GPU backends its expert kernels are also Q4_0-only, and its routed-FFN LoRA targets (the router and the per-expert projections) apply on CPU only: an adapter carrying them is refused rather than half-applied.

Modalities

• Text → text: every supported architecture.
• Vision (image → text): LFM2-VL models. CeraEngine auto-attaches the vision encoder; Session::append_image runs image → ViT → projector → prefill. The ViT encoder runs on the GPU (Metal or wgpu) with a CPU fallback. Verified against LFM2.5-VL-450M.
• Audio (in / out): LFM2-Audio (lfm2-audio-v1): feed PCM audio in, and (with a vocoder) decode audio out.
• Wake word & ASR: Pure-Rust streaming Keyword Spotting (cera::hotword), Silero VAD v5 (cera::vad), and OpenAI Whisper speech-to-text (cera::model::whisper).

Platforms & backends

Cera dispatches to the fastest available backend at runtime (--device auto), or you can pin one:

--device auto prefers native Metal on macOS and iOS, Qualcomm Hexagon NPU on supported Snapdragon devices, and wgpu where a GPU is available, falling back to CPU otherwise. A routed-expert model too large for the DSP's mappable memory (LFM2.5-8B-A1B) skips the NPU under auto, since paging it is slower than the CPU; set CERA_HEXAGON_PAGE_EXPERTS=1, or ask for --device hexagon, to page it.

Qualcomm Hexagon NPU backend

Cera provides a native backend for Qualcomm Hexagon NPUs and Compute DSPs on Snapdragon 8 Gen 2 (v73), Snapdragon 8 Gen 3 (v75), Snapdragon 8 Elite (v79), and next-generation Snapdragon architectures (v81/v85). See the Hexagon NPU Guide for full architecture and Android packaging details.

• FastRPC Unsigned Process Domain: Runs within Qualcomm's CDSP unsigned user domain (/dev/fastrpc-cdsp), enabling third-party Play Store applications to execute on HTP/HVX hardware without root privileges or vendor signing keys.
• Zero-Copy Shared Memory: Allocates model weights, activations, and KV cache buffers in shared rpcmem (DMA-BUF / ION) mapped into both CPU and DSP address spaces, eliminating bus copying.
• Embedded Skeleton Libraries: Embeds prebuilt, 16 KB page-aligned DSP worker libraries (libcera-htp-v{73,75,79,81,85}.so) into the host binary and extracts them automatically at startup. Skeletons incorporate extended operators (Conv1D, ConvTranspose1D, Snake, UnaryStep, Sum).
• Accelerated Kernels:
  • LLM Text Generation: 32x32 tiled Q4_0 and Q8_0 matrix repacking for HTP matrix units, forward decode batched to eliminate most synchronization boundaries (the default is one uncapped batch per token with a static batch template, and CERA_HEXAGON_BATCH_TENSORS=<n> caps each batch at n tensors for bisection), ping-pong scratch memory isolation, FastRPC latency QoS (FASTRPC_CONTROL_LATENCY = 100 µs), FastRPC driver wakelock (FASTRPC_CONTROL_WAKELOCK), and Q8_0 quantized KV cache (~47% memory reduction over F16).
  • On-DSP Argmax: Offloads greedy token argmax reduction directly to Hexagon HTP (HtpOpCode::Argmax, opcode 62), reading only 4 bytes of token ID from rpcmem rather than copying or invalidating 128 to 256 KB of float32 logits, eliminating host CPU allocations and memory traffic during greedy decode.
  • Speculative Decoding Verification: Batches LM-head projections across all draft token rows via parallel HTP matrix dispatch (forward_prefill_logits_all). When paired with prompt-lookup drafting (ngram=2, k=4), speculative decoding lifts 350M decode throughput from 164.0 tok/s to 259.5 tok/s (peak 260.2 tok/s) and 2.6B decode from 26.7 tok/s to 47.7 tok/s (peak 48.5 tok/s) on Snapdragon 8 Elite.
  • Multimodal Vision (ViT): Dispatches all 24 Vision Transformer blocks in a single batched submission with on-NPU Flash Attention, F16 KV scratch handling, and quantized MLP projector execution.
  • Whisper Speech Recognition: 64-token chunked Conv1D and LayerNorm dispatches adhering to Snapdragon 8 Elite's 8 MB physical VTCM ceiling, paired with FlashAttnExt autoregressive decode.
  • Audio Synthesis & Vocoder: HexagonDepthformer executes all 8 autoregressive passes (48 transformer layers per audio frame) entirely on the NPU using HTP GEMV, RMSNorm, RoPE, and FlashAttnExt, alongside on-DSP audio detokenization (LayerNorm, linear GEMM, SwiGLU, Conv1D).

Quantization

Weights run in Q4_0, Q4_1, Q8_0, Q4_K, Q5_K, and Q6_K, plus dense F32. Activations are dynamically quantized to Q8_0 for fast integer GEMV on CPU.

Those are GGML tensor types, and dispatch is per tensor, so what decides whether a file runs is the mix inside it, not its Q4_K_M-style label. K-quant downloads usually work, but the label is not a guarantee: llama.cpp substitutes Q5_0 / Q5_1 / IQ4_NL for tensors whose rows are not a multiple of 256, and cera has no kernel for those. cera inspect --model <file> lists the per-tensor types, and an unsupported one fails at weight resolution naming the tensor rather than failing anonymously, with the type name too, for the types cera knows about (an IQ type reports its numeric id).

Backends are not uniform, so a file that runs on one may not run on another. On native Metal, Q4_1 works as a projection weight but not as token_embd / output; --device auto falls back to wgpu or CPU on its own, while --device metal reports the gap. F16/BF16 tensors parse, and are dequantized on the LoRA, vision, and audio paths, but the transformer weight and token-embedding paths have no kernel for them; an F16-weight LLM is not a supported configuration.

Language bindings

One Rust core, consumed from many places:

A complete SwiftUI example app (streaming chat + embeddings + LoRA) that consumes the published Swift Package lives in examples/CeraChat; it doubles as a real-device Metal validation harness.

Structured output (GBNF grammars)

Force the model's output to match a grammar, useful for JSON, tool calls, or any schema. Cera ships a byte-level GBNF engine (mirroring llama.cpp's) that masks the sampler each step so only grammar-valid tokens can be produced.

# Constrain generation to JSON (a token limit can still truncate it)
cera run -m model.gguf -p "List 3 colors as JSON" --json

# Any custom GBNF (inline or @file)
cera run -m model.gguf -p "..." --grammar @schema.gbnf

Supports literals, character classes, alternation, grouping, and repetition (* + ? and bounded {n,m}). Available from every binding, not just the CLI: in Rust set GenerateOpts.grammar to a compiled Grammar (Grammar::parse(gbnf)?), while the Kotlin/Swift FFI (GenerateOpts.grammar) and browser/Node WASM (GenerateOpts.setGrammar(gbnf)) take the GBNF string directly and compile it natively, so mobile and web apps use the same grammar rules. A token limit or cancellation can leave an incomplete value; parse and validate the final output.

Native JSON Schema compilation

Cera compiles a subset of JSON Schema directly to GBNF without external tooling:

• Rust: let grammar = Grammar::from_json_schema_str(&schema_json)?; or opts = opts.with_json_schema(&schema_json)?;
• Swift: let constrainedOpts = try opts.withJsonSchema(schema)
• Kotlin: val constrainedOpts = opts.withJsonSchema(schema)
• Python: constrained_opts = opts.with_json_schema(schema)
• Dart: chat.streamJson(opts, schemaJson)
• Browser / Node WASM: opts.setJsonSchema(schema) or const constrainedOpts = GenerateOpts.withJsonSchema(opts, schema)

Pass the returned constrained options to generation. The Swift/Kotlin/Python helpers and WASM withJsonSchema return copies; WASM setJsonSchema updates opts in place.

Supports nested properties and required keys, bounded arrays, scalar enums, local $defs/definitions and chained $ref resolution. allOf accepts only supported scalar wrappers and compatible object merges. Numeric ranges, string patterns and other unimplemented keywords are not enforced. See the supported subset before relying on a schema.

Tool calling

Give the model a set of tools (OpenAI "function" schemas) and get the calls it makes back out as structured data. Cera renders the tools into the model's chat template and parses tool calls from the reply, format-aware, since the wire format isn't interchangeable: LFM2/LFM2.5 emit Pythonic calls ([get_weather(city="Paris")]) while Hermes-style Qwen2.5/Qwen3 emit JSON (<tool_call>{"name": …, "arguments": {…}}</tool_call>). The format is detected from the GGUF architecture.

# Let the model decide whether/how to call a tool (free text or a call)
cera run -m model.gguf -p "What's the weather in Paris?" \
  --tools '[{"name":"get_weather","description":"Get weather for a city",
             "parameters":{"type":"object",
               "properties":{"city":{"type":"string"}},"required":["city"]}}]'

# Constrain tool-call syntax after the model emits a call marker
cera run -m model.gguf -p "..." --tools @tools.json --constrain-tools

With --constrain-tools a lazy grammar trigger keeps generation free until the model starts a tool call, then constrains function/argument names and outer value syntax. Its separate tool grammar does not enforce required arguments, uniqueness or nested item/property schemas. Prose and truncated calls remain possible; parse and validate arguments before executing a tool. In --tools mode stdout is machine-readable: only the JSON array of calls ([] when the model answered in prose); the assistant reply and timing stream to stderr, so … --tools tools.json | jq just works.

Available from every binding, not just the CLI: Rust (cera::tools), Kotlin/Swift (applyChatTemplateWithTools, parseToolCalls, toolGrammar, detectToolFormat), and browser/Node WASM (the same names); see each crate's README for the API surface.

Conversational chat coordination

Cera promotes the high-level SessionChat coordinator (cera::session::chat / ChatSession in FFI) to the public API:

• Explicit lifecycle: Ingestion moves Idle to PromptReady; generation reaches TurnComplete only on a profile terminal. Token limits, cancellation and nonterminal stops can leave Interrupted; zero-token/no-progress calls can preserve PromptReady for retry. Recovery may restore, reset or mark execution unusable.
• Delta-only prompt evaluation: Continuation turns evaluate only the newly appended user message delta, preserving warm KV cache resident state without repeating full prompt history prefill.
• Validated template profiles: Discovers framing for ChatML, Llama 3, Gemma and compatible Jinja templates (TemplateFamily, ProfileBuilder); unsupported continuation profiles are rejected.
• First-class tool loop: Register tool schemas directly on the coordinator via chat.set_tools(tools); tool calls are parsed into TurnResult.tool_calls and tool execution outputs are ingested with chat.ingest_tool_response(...).

use cera::{Message, SessionPhase};

let mut chat = session.into_chat().map_err(|(_, err)| err)?;
chat.ingest(&Message::user("Hello!"))?;
let result = chat.complete(&opts)?;
println!("Assistant: {}", result.text);

// Continue only after the profile's terminal marker ended the turn.
if chat.phase() == SessionPhase::TurnComplete {
    chat.ingest(&Message::user("What did I just say?"))?;
    let reply = chat.complete(&opts)?;
    println!("Assistant: {}", reply.text);
} else {
    // Start a fresh conversation; alternatively replace the full history.
    chat.reset()?;
}

Language-native reactive streaming

Cera provides native reactive streaming wrappers tailored for each language ecosystem:

• Swift: AsyncThrowingStream<String, Error> via chat.stream(opts: opts) and chat.streamJson(opts: opts, schemaJson: schema).
• Kotlin: Flow<String> via chat.stream(opts) and chat.streamJson(opts, schema).
• Python: Python generator Iterator[str] via chat.stream(opts) and chat.stream_json(opts, schema).
• Dart: Stream<String> via chat.stream(opts) and chat.streamJson(opts, schema).
• Rust: chat.stream_text(&opts, |chunk| { ... })? yielding real-time text fragments into a callback closure.

Stream completion does not imply a completed chat turn. Inspect the phase before the next user message; reset or replace interrupted conversations. Abandoning an active stream requests cancellation. Let that operation finish before reusing the handle. See streaming and cancellation.

Session checkpointing & persistence

CPU Session and Chat support resumable snapshots. Native Metal/Hexagon/wgpu Session checkpoint and restore operations reject backend-owned state; browser WebGpuSession has a separate device snapshot API.

• Binary serialization: SessionCheckpoint / ChatCheckpoint formats with magic headers (CERASCHK / CERACHAT), structural fingerprints and versioned headers. Fingerprints are not cryptographic model identities.
• Atomic file operations: checkpoint.save_to_file(path) and SessionCheckpoint::load_from_file(path) with atomic writes and clean error recovery.
• Browser WebGPU checkpointing: Async GPU readback/export and synchronous restore/import into a compatible session.
• Continuation state preservation: Restores live sequence length, KV cache states, and terminal token commitment for seamless multi-turn resumption.

Version 0.6.2 and newer validates row geometry, KV precision and compression identity, including TurboQuant seeds. Recreate older f16/TurboQuant snapshots whose fingerprints lack compression identity; plain f32 fingerprints are unchanged. See checkpoint compatibility.

LoRA adapters & hidden states

Runtime LoRA. Load a LoRA adapter, a llama.cpp GGUF (from convert_lora_to_gguf) or a PEFT .safetensors, and attach it to a session. The delta is applied at inference time (y += scale·B·(A·x)), never merged into the weights, so the base model stays quantized and adapters hot-swap / unload per request at ~no cost. Runs on CPU, Metal, and wgpu (batched-GEMM prefill + decode) and is dimension-checked against the model at attach. The one gap is lfm2moe's routed-FFN targets, which apply on CPU only; on a GPU backend such an adapter is refused with CeraError::LoraUnsupportedByBackend rather than partially applied.

Hidden-states extraction. Pull the per-token last-layer hidden state (post-final-RMSNorm, the llama.cpp --pooling none vector) straight out of the engine, reflecting the active adapter. This unblocks classifier / extractor / embedding heads, e.g. a section router: LFM2.5 + a route_section LoRA + a small linear head over the mean-pooled hidden state.

Both are available from every binding:

// Swift / Kotlin (UniFFI): same shape on both
let adapters = try LoraAdapters.fromSafetensors(path: p, alpha: nil)
try session.attachLora(adapters: adapters)               // hot-swap-able
let pooled = try session.hiddenStatesMeanPooled(tokens: toks)  // [hidden_size]

// Browser / Node (WASM)
const adapters = LoraAdapters.fromSafetensorsBytes(bytes, undefined);
session.attachLora(adapters);
const pooled = session.hiddenStatesMeanPooled(tokens); // Float32Array

TurboQuant KV-cache compression

Cera includes the first implementation of TurboQuant (arXiv:2504.19874, Google Research 2025: TurboQuant: Extreme KV Cache Compression) for LFM2, compressing the KV cache to ~3 bits/key + ~2 bits/value (~12× vs f32) with near-lossless quality and no calibration. On a 1.6B LFM2 model at 4K tokens that's ~192 MB → ~16 MB of KV, with decode staying within ±5% of f32.

The default KV cache is f16 on the CPU backend (llama.cpp's default cache type; --kv-cache-keys auto), which halves the bytes a decode streams at depth; --kv-cache-keys f32 keeps full precision. Backends that keep their own cache (wgpu, Metal, Hexagon) are unaffected.

Enable TurboQuant on the CLI:

cera run -m lfm2.gguf -p "Hello" --kv-cache-keys tq3 --device cpu
cera run -m lfm2.gguf -p "Hello" --kv-cache-keys tq3 --device gpu     # wgpu
cera run -m lfm2.gguf -p "Hello" --kv-cache-keys tq3 --device metal   # native Metal

Supported on all three backends (CPU, wgpu, and native Metal) for both decode and chunked prefill. On the GPU backends the compressed cache lives in GPU buffers and the prefix-cache snapshots are byte-compatible with the CPU's, so all three write the same TQK1/TQV1 blob format. Two GPU-specific restrictions: head_dim must be a power of two ≤ 128 and a multiple of 32 (every supported model is 64 or 128), and only the both-sides mode (tq3) is available; the single-sided debug modes tq3-keys / tq3-values fall back to the backend's uncompressed KV there (f32 on wgpu, f16 on Metal) with a warning. --n-keep context shift is not supported with any TurboQuant mode on any backend.

See cera/src/turboquant.rs for the algorithm (PolarQuant + QJL).

Prefix cache anchors

cera::kv_cache::KvPrefixCache retains token-prefix snapshots in warm memory and, when disk caching is enabled and configured, a cold disk tier. Its insert_anchor and find_deepest_semantic_anchor APIs let callers tag and select matching prefix snapshots using caller-supplied boundary metadata. The standard engine cache path does not automatically select semantic anchors.

FlatBuffers v2 persistence preserves each snapshot's existing precision, including TurboQuant data when already compressed. Callers choose compression before capturing snapshots; moving a snapshot to disk does not recompress it.

DSpark: Neural Speculative Decoding

Cera includes support for DSpark (arXiv:2407.08608, 2024: DSpark: High-Throughput Speculative Decoding with Lightweight Drafters), accelerating inference throughput via parallel multi-token draft verification:

• Lightweight Sidecar Drafters: Load compact draft models (cera::model::dspark) that share embedding and GGUF weight backings with the target model without redundant memory mapping.
• Parallel GPU Verification: Validates draft token sequences in a single forward pass with batched LM-head verification on CPU, Metal, and WebGPU.
• Automatic Drafter Discovery: Bundle loader automatically discovers and attaches paired DSpark sidecar models from LeapBundles and Hugging Face repositories.

Sharing a loaded GPU or NPU model

Metal, Hexagon, and wgpu permit one live Session per loaded model. A second session returns Busy until the first is dropped; reset and cancellation keep ownership. CPU models continue to support shared weights across concurrent sessions. Load separate GPU or NPU models for simultaneous conversations. The session ownership walkthrough shows the public API, cleanup behavior and executable checks.

LFM2-Audio transcription during a live GPU conversation uses a cached secondary model built from retained weights. This preserves conversation KV and costs additional model memory and first-use setup; the walkthrough includes a native Dart consumer for transcription and seeded generation.

Whisper speech recognition

The standalone FfiWhisperModel loads a Whisper GGUF from a path or owned bytes and transcribes 16 kHz mono float PCM through Swift and Kotlin. Use transcribeAsync for a background decode; whisperDefaultTranscribeOpts supplies language, translation, timestamps, token limit and temperature defaults. The Swift/Kotlin recording examples are compiled and executed against the generated bindings. Build matching wrappers and native libraries from the same revision.

# Exercise loading, nonempty decoding, async calls and ownership without downloads.
cargo test -p cera-ffi --test whisper_ffi --locked --offline

Each call handles at most 30 seconds; segment longer recordings first. This is separate from LFM2-Audio's CeraEngine.transcribe. Dropping its Rust future cooperatively cancels decoding. Kotlin propagates coroutine cancellation; the pinned Swift wrapper does not propagate Task.cancel().

Voice Activity Detection (Silero VAD v5)

Cera includes a native, pure-Rust implementation of Silero VAD v5 (cera::vad), enabling zero-dependency, real-time voice activity detection across desktop, mobile, and web:

• Zero runtime dependencies: Pure Rust tensor operations with no ONNX Runtime or Python required.
• Streaming audio evaluation: State-machine-based chunk iterator (VadIterator) processing 512-sample frames (32ms at 16kHz) with state persistence.
• Speech segment timestamping: Automatic speech segment extraction, speech padding, and silence trimming.
• Cross-Platform Bindings: Available in Rust (cera::vad::SileroVad), CLI (cera vad), Swift/Kotlin via UniFFI, and Dart/Flutter via cera_ffi.

# Detect speech timestamps from a WAV file
cera vad --model models/silero_vad.gguf --audio speech.wav --json

# Convert official Silero VAD v5 ONNX model to GGUF
python scripts/convert_silero_vad.py silero_vad.onnx models/silero_vad.gguf

Keyword Spotting (Native Wake Word Engine)

Cera includes a native, pure-Rust Keyword Spotting (KWS) engine (cera::hotword) designed for always-on, low-power wake word detection on mobile and edge devices:

• Self-describing GGUF containers: Stores acoustic configurations, label lists, window and hop dimensions, and detection thresholds in GGUF metadata under kws.*.
• Reusable inference buffers: LogMelFrontEnd and HotwordDetector retain scratch buffers for acoustic features and inference. Event strings, growing stream buffers and foreign argument/result marshaling can still allocate; process chunks serially on a background audio worker.
• VAD-gated streaming iterator (HotwordIterator): Interleaves Silero VAD gating (512 samples at 16 kHz or 256 at 8 kHz) with configurable hop KWS checks. It skips acoustic evaluation when the buffered window has no recent speech; other model sample rates disable VAD gating.
• Adaptive Automatic Gain Control (AGC): Peak normalization scales soft or far-field speech up to nominal reference levels with a 30.0x gain ceiling, preventing ambient noise floor amplification while maintaining sensitivity across desk or room distances.
• Cross-platform bindings: Available in pure Rust (cera::hotword), Kotlin/Android, Swift/iOS, Python via cera-ffi, and Flutter via cera_ffi_flutter.

# Export and fold PyTorch wake word weights into a self-describing GGUF container
python scripts/convert_kws.py --checkpoint model.pt --output models/hey_liquid.gguf

# Train a custom wake word model with automated phonetic near-miss negative synthesis
python tools/wake_word/train_kws.py --phrase "Hey Liquid" --epochs 25 --output models/hey_liquid.gguf

Whisper Speech Recognition (ASR)

Cera provides a pure-Rust implementation of OpenAI Whisper ASR (cera::model::whisper), exposed across all foreign language bindings via UniFFI (cera-ffi):

• Unified mobile deployment: Mobile applications link a single native library (libcera_ffi.so or CeraFFI.xcframework) for wake word detection, speech activity gating, and full speech-to-text transcription without external C++ or JNI dependencies.
• Cooperative async cancellation: Dropping a transcribe_async Rust future signals a running decoder to stop at its next cancellation check. Kotlin propagates coroutine cancellation; the pinned Swift wrapper does not propagate Task.cancel(). See the Whisper examples and limitations.
• Multilingual and timestamp options: Multilingual models support language detection and selection from the standard 100-code table. The timestamp option controls decoder tokens; transcription returns plain text without structured segment or word timestamps.
• Cross-platform bindings: Available in pure Rust (cera::model::whisper), Swift, Kotlin, Python, and Dart/Flutter.

Unified Audio Pipeline (AudioPipeline)

Cera provides a unified stateful streaming audio pipeline (cera::audio_pipeline::AudioPipeline and cera-ffi FfiAudioPipeline) connecting Silero VAD v5, Keyword Spotting, and Whisper ASR:

• State machine lifecycle: Tracks ListeningForHotword, ListeningForSpeech, SpeechActive, and Transcribing states with independent audio buffer ownership.
• Pre-roll sample preservation: Automatically captures and preserves pre-roll audio ring buffers upon wake word detection so trailing speech onset is never clipped.
• Utterance chunking with VAD state preservation: Splits long continuous speech at max_utterance_ms without resetting Silero VAD recurrent hidden states across continuation segments.
• Wait-free cancellation: Fast cooperative cancellation via atomic handles without mutex contention during active Whisper inference.
• Speaker labels (optional): Attach a Streaming Sortformer diarizer (AudioPipelineBuilder::with_diarizer, or FfiAudioPipeline.fromFilesWithDiarizer), or the Nemotron-3-Diarization add-on (with_diarizer_nemotron3 / fromFilesWithDiarizerNemotron3, 8 speakers at 10 ms frames), and every transcribed utterance is followed by an UtteranceLabeled event with its speaker slot (0 to 3, or 0 to 7), once the diarizer has covered it. The diarizer hears the same samples as the pipeline, so both share one clock, and either diarizer can run on the Hexagon NPU (prefer_npu, with a GGUF converted using --tail-outtype q8_0; see scripts/sortformer/README.md and scripts/nemotron3_diarization/README.md). Compare the two side by side with cera diarize --compare.
• Cross-platform bindings: Available in Rust (AudioPipeline), Swift, Kotlin, Python, and Dart/Flutter (FfiAudioPipeline).

Hugging Face Models & Streaming Quantization

Cera supports direct loading and streaming execution from Hugging Face model repositories:

• Direct Repo Loading: Point Cera directly at Hugging Face model IDs or URLs (--hf org/repo).
• Streaming Quantization: Convert remote SafeTensors models tensor by tensor into a cached GGUF file without storing complete source shards (--quant-strategy fast-mse, hqq, quarot).
• LeapBundles Integration: Seamless automatic download, manifest parsing, and local caching under $HOME/.cache/cera.

# Run directly from a Hugging Face GGUF repository
cera run --hf LiquidAI/LFM2.5-1.2B-Instruct-GGUF --quant Q4_0 --prompt "Explain quantum computing."

# Quantize and run on-the-fly from a remote SafeTensors repository
cera run --hf LiquidAI/LFM2.5-350M --quant Q4_0 --quant-strategy fast-mse --prompt "Hello"

Quick start

# Install the CLI (CPU-only build)
cargo install cera-cli --locked

# ...or with a GPU backend
cargo install cera-cli --locked --features metal   # or: --features gpu

# ...or build from source
just release   # optimized LTO build → target/release/cera

# Generate from a local GGUF
cera run --model model.gguf --prompt "Explain quantization in one sentence."

# Auto-download a bundle from Hugging Face and generate
cera run --bundle-id LFM2.5-1.2B-Instruct --quant Q4_0 --prompt "Hello"

# Interactive multi-turn chat (keeps the prefix cache warm across turns)
cera chat --bundle-id LFM2.5-1.2B-Instruct --quant Q4_0

# Or with no model source at all: pick one from the published catalog
# interactively (needs a terminal, and not --no-tui)
cera chat

# Pick an accelerator backend
cera run -m model.gguf -p "Hi" --device metal   # or: gpu, npu, hexagon, cpu, auto

Using the library directly:

use cera::{CeraEngine, EngineConfig, GenerateOpts, Message, SessionConfig};

let engine = CeraEngine::from_path("model.gguf", EngineConfig::default())?;
let mut session = engine.new_session(SessionConfig::default())?;

// Raw prompt completion:
session.append_text("Once upon a time")?;
let opts = GenerateOpts { max_tokens: 128, ..Default::default() };
let summary = session.generate(&opts, &mut sink)?; // sink: your ModalitySink

// Or conversational chat coordination (delta-only prefill, live KV retention):
// Start a new conversation after the raw completion above.
session.reset()?;
let mut chat = session.into_chat().map_err(|(_, err)| err)?;
chat.ingest(&Message::user("What is the capital of France?"))?;
let reply = chat.complete(&opts)?;
println!("Assistant: {}", reply.text);

See the cera crate README for the full library API. This checkout also exposes explicit loading alongside the existing constructors:

use cera::{ModelLoader, ModelSource, SessionConfig};

let model = ModelLoader::new(ModelSource::path("model.gguf")).build_generative()?;
let mut session = model.create_session(SessionConfig::default())?;
drop(model); // The session retains its loaded resources and live KV state.
session.append_text("Once upon a time")?;

Run the complete text-completion example: cargo run -p cera --example explicit_loading -- model.gguf "Once upon a time". For multi-turn chat, run the conversational chat example: cargo run -p cera --example chat -- model.gguf. The API refactor also provides runnable multi-language examples, including conversational chat (Rust, Swift, Kotlin, Python, Dart), session continuation, and vision/draft loading. The audio walkthrough demonstrates PCM input and output with locally generated test weights. The remote companion examples run HF discovery, downloads, cache repair and retained vision/audio/draft execution against a local HTTP fixture. The HF revision examples load changed GGUF revisions into separate cache entries, verify defaults and keep a CPU session live across another revision load. They also exercise failed metadata resolution and explicit commit checks. The SafeTensors conversion examples exercise real conversion, corruption repair and option-sensitive cached reloads. The examples verify checkpoint bytes, pin inputs to one upstream commit, and refresh changed revisions. Retained models execute after replacement, and sessions continue after their parent model is released. The persistent cache examples demonstrate same-path weight replacement, unchanged-weight disk reuse and retained live sessions.

CLI commands

Tuning

Defaults are chosen to be correct-or-harmless across a wide range of hosts, not optimal on any one. The knobs below override them.

How the decode thread count is chosen

Decode width is sized from the loaded model, because there is no single best value, and, less obviously, model size does not predict it. Measured on a Ryzen AI MAX+ 395 (16 physical / 32 logical), decode tok/s change going from 12 to 20 workers:

Note the middle two: near-identical weight bytes, opposite answers. What separates them is how many pool dispatches a token is split across; LFM2 issues 99 per token against SmolLM-360M's 257, so each dispatch amortizes 2.6× more work against the same fixed barrier cost. Below ~2.5 MB per dispatch decode is barrier-bound and wants fewer workers; above it decode is bandwidth-bound and wants more.

cera computes that ratio from the GGUF at load and picks a narrow or wide width accordingly. A/B against the old flat cap of 12 (same binary, interleaved, CERA_DECODE_SIZING=off vs on, the five models above at two prompt depths) gave +19% mean over those 10 cells with no cell slower (small models gain too: they get fewer than 12 workers). Scored differently, against each model's own best measured width across the full 12-model set, the rule reaches 98.1% of peak versus 90.1% for the flat cap; one combo near the threshold (LFM2.5-350M Q4_0) lands on the narrow arm when its measurement mildly preferred the wide one, which is the cost that average already includes.

The pool itself is a process-wide singleton, built once on the first decode dispatch, so the width comes from the first model loaded into a process and stays there for any loaded after it; it does not re-size per load. (The thread count was already inherited that way before this sizing existed.) Embedders running several models in one process should pick the knobs below for whichever model's decode matters, or load that one first.

This is calibrated on one x86 host. The sizing steps aside, leaving the previous flat-cap behaviour, in two cases. On heterogeneous parts, because decode there measured best across all big cores (that evidence is from ARM big.LITTLE; x86 hybrid parts decline by the same argument, without the direct measurement). And on hosts whose physical core count cannot be detected (Windows, BSD, Intel macOS), where deriving a width from the logical count would overshoot badly. Pinning a width forces it on anyway, for sweeping a machine it has not been tuned against: one arm is enough on a heterogeneous host, both are required where the physical core count is undetectable.

Apple Silicon is sized, not declined. M-series parts are heterogeneous in hardware, but cera detects only their P-cores, so neither decline case applies: an M4 Max gets 12 workers for large models (its full P-core count; there is no SMT for the wide arm to spend) and 6 for small ones. That narrow arm is the part of this that has not been measured off x86.

If you run one model repeatedly and care about decode latency, it is still worth measuring: cera bench --model <path> with CERA_DECODE_THREADS set sweeps a fixed width, and CERA_DECODE_SIZING=off gives you the old default to compare against. Interleave the arms rather than sweeping; laptop clocks drift enough to invent a trend that isn't there.

Other features

• Speculative decoding: opt-in greedy speculation with prompt-lookup (n-gram) drafting: no draft model, so no extra weight memory. Verifying K drafted tokens in one forward amortizes the single weight-read over the accepted run, which is where bandwidth-bound decode gets its win. (As of #327 the LM head is projected for all verified positions in one batched GEMM, so it too is read once per round.) Every emitted token is the target's own argmax (a poor draft costs acceptance rate, never correctness), though a near-tie can land differently than a sequential greedy run, since the verifier forwards a different batch shape. It engages on eligible CPU, native Metal, Qualcomm Hexagon NPU, and wgpu models with all-position logits, plain greedy decoding, uncompressed KV and no audio decoder. Speculative decoding defaults to disabled (off) unless explicitly configured or when a draft sidecar model is present. See speculative decoding for backend and execution limits. The CLI exposes speculative decoding knobs on run, chat, and bench (--spec and --no-spec).
• Streaming & cancellation: tokens (and audio frames) arrive through a ModalitySink as they decode; Session::cancel() interrupts long prompts responsively via chunked prefill.
• Prefix caching: warm (in-memory) and cold (on-disk) KV reuse across sessions, namespaced by model fingerprint, so repeated prompt prefixes skip re-prefill.
• Chat templates: Jinja2 (minijinja) rendering straight from GGUF metadata, including multimodal (image + text) messages.
• Context shifting: RoPE re-rotation with n_keep prefix pinning, on CPU and GPU, keeps generation going past the context window.
• Built-in BPE tokenizer: vocab, merges, and special tokens loaded directly from GGUF; no external tokenizer files.

Architecture

Cera is a Cargo workspace. The core library does GGUF parsing, quantization, the compute backends, the models, and the tokenizer; everything else is a thin adapter over it.

• cera: core library
• cera-cli: CLI binary (clap)
• cera-ffi: UniFFI bindings (Kotlin / Swift / Python)
• cera-ffi-kotlin: Android packaging (AAR)
• cera_ffi · cera_ffi_flutter: Dart bindings, and the Flutter plugin that ships a native library for them
• cera-wasm: wasm-bindgen browser / Node bindings
• cera-parity: cross-binding parity harness (runs one prompt through every binding and reports drift)

See the cera crate README for the module layout, the model trait, and the inference loop.

Performance

Cera is competitive with (and on decode, often faster than) llama.cpp on the LFM2 family. On an M1 Max with Q4_0 weights, the native Metal backend decodes roughly 2× faster than llama.cpp across tested VL and Audio models; prefill leads at short prompts and trails at long ones.

On the cross-platform wgpu backend, decode is 1.78x faster than it was on the same model and machine (LFM2-VL-450M Q4_0, M1 Max: 63.4 to 112.8 tok/s), from four changes: removing register spills in the quantized GEMV kernels, merging the LFM2 conv block into one compute pass, deleting two per-token GPU round trips that carried almost no work, and running the LM head on the weight as GGUF stores it instead of a dequantized f16 copy (which also gives back ~79 MB of VRAM on a 230M model). A fifth change since then, word-loading the Q6_K GEMV instead of reading it byte-at-a-time, measured +11.6% decode (109.5 to 122.1 tok/s, ABBA-ordered) on LFM2.5-230M-Q4_K_M, a different model from the 1.78x row above, so the two do not multiply. That gain is entirely conditional on the change before it: measured without #320, the same kernel was flat (110.0 vs 110.5 tok/s), because the only Q6_K matrix in the decode path was one ffn_down of 42. #320 makes the tied Q6_K embedding the LM head, and that is what the faster loads then pay off on. The per-kernel breakdown, and what did not work, are in benchmarks/BASELINE.md.

On CPU, rows dispatch through a persistent, affinity-pinned threadpool with dynamic chunk-stealing rather than a per-GEMV fork-join. This fixes the multi-core decode collapse on Android big.LITTLE and scales decode across the performance cores (Tensor G5, LFM2 Q4_0, warm: CPU decode matches or beats llama.cpp). Thread placement and count auto-detect per device; the CERA_* override knobs are documented under CPU threading & tuning.

Detailed methodology, per-model tables (decode + prefill vs llama.cpp), the Accelerate/AMX BLAS results, and the backend optimization notes live in benchmarks/README.md. Numbers there are tagged with the commit and device they were measured on; some sections are older than others, and the Android GPU row in particular predates the wgpu work above.

License

Licensed under either of Apache License, Version 2.0 or MIT License at your option.