laya-candle

Use this crate to classify, score or check a piece of text from your Rust code: give it a message and a few questions, and it answers each one with a label, a score or a yes/no probability you can branch on. It helps with triaging support tickets, sorting emails, moderating posts, guarding prompts before they reach an LLM, and any other case where you need a reliable decision rather than generated text.

Typed decisions in one forward pass, in pure Rust.

Ask typed questions about any state (string, email, ticket, JSON) and get back a label, a score or a probability, each with a calibrated confidence. Because nothing is generated, the answer never needs parsing and cannot hallucinate.

Pure-Rust port of Laya. Loads Laya's checkpoints directly from the Hugging Face Hub via candle.

All questions are answered in a single batched forward pass. Cost scales with input length, not output length.

Install

[dependencies]
laya-candle = "0.1"
serde_json = "1"

Rust 1.88+. A C compiler is required to build (candle-core enables tokenizers' onig feature, which compiles Oniguruma).

Library only, no CLI:

laya-candle = { version = "0.1", default-features = false, features = ["hub"] }

Quickstart

use laya::{ModelName, Question, Questions, Router};
use serde_json::json;

let router = Router::builder().preload([ModelName::English, ModelName::Multilingual]).build()?;

let questions = Questions::new()
    .with("department", Question::choice("Which department should handle this request?")
        .option("billing",   "invoices, payments, refunds")
        .option("technical", "bugs, outages, system errors")
        .option("other",     "everything else"))
    .with("urgency", Question::score("How urgent is this request?")
        .level("not urgent").level("soon").level("critical deadline or blocking issue"))
    .with("churn_risk", Question::noul("Does the user threaten to cancel or leave?"));

let out = router.predict(json!({
    "subject": "Duplicate charge on invoice #4411",
    "body": "We were billed twice for March. Refund the duplicate today or we cancel our plan.",
}), &questions)?;

out.get("department").unwrap().as_choice();     // Some("billing")
out.get("department").unwrap().confidence();    // 0.86
out.get("urgency").unwrap().as_score();         // Some(1.44)
out.get("churn_risk").unwrap().as_noul();       // Some(0.825)

Other ways to build:

// Questions from the same JSON schema as the Python package
let questions = laya::Questions::from_json(&std::fs::read_to_string("questions.json")?)?;

// One checkpoint, no routing
let agent = laya::Agent::from_hub("convaiinnovations/laya", Some("multilingual"))?;

// From a local directory
let agent = laya::Agent::from_dir("checkpoints/laya")?;

Agent is immutable and predict takes &self: share one behind an Arc.

Gate on confidence:

for id in out.below_confidence(0.85) {
    escalate_to_a_human(id);
}

Where Rust matters

The decision model is the same one the Python package runs. What changes with a native crate is where it can go and what it can sit inside of:

• A request-path middleware. predict takes &self, so one checkpoint behind an Arc serves every worker of an axum, actix or tonic service. Screen prompts, route tickets or gate a webhook inside the process that received it, without a round trip to a sidecar service.

  let agent = Arc::new(Agent::from_hub("convaiinnovations/laya", None)?);
  let guard = presets::guard();
  
  // in each handler:
  let out = agent.predict(json!({"prompt": body}), &guard)?;
  if out.get("prompt_injection").and_then(|a| a.as_noul()).unwrap_or(0.0) > 0.9 {
      return Err(StatusCode::FORBIDDEN);
  }
  

• A single binary at the edge. There is no interpreter or runtime to ship alongside it. The crate, a weights directory and Agent::from_dir run on a box with no network, a kiosk, or a container whose image is the binary plus five files.

• Text that already flows through Rust. A proxy, an API gateway, a Kafka consumer, a Discord or Slack bot, a log shipper: a typed decision can be added where the text is, without introducing a second language to the deployment.

• Multilingual traffic without a model registry. Router detects the script and language in microseconds of pure Rust before touching any weights, then runs the checkpoint that can read the text. One code path handles 100+ languages.

• Batch jobs and shell pipelines. The laya CLI reads questions as JSON and states from stdin, so a cron job or a find | xargs laya predict labels a corpus with no code at all.

• Confidence you can branch on. Every answer comes back as a distribution rather than a string. The gating logic (escalate, retry, hand to a human) is ordinary Rust over ordinary numbers rather than a regex over generated prose.

CLI

$ cargo install laya-candle

$ laya predict -s @examples/data/email.json -q triage
routed to english — English Latin text

intent            choice  refund      confidence 0.99
is_urgent         noul    0.222       confidence 0.78
frustration       score   1.72 / 3    confidence 0.32
refund_requested  noul    0.891       confidence 0.89
churn_risk        noul    0.360       confidence 0.64

• laya predict -s <text|@file|-> -q <preset|@file|-> [--model NAME] [--lang xx] [--checkpoint DIR] [--json]
• laya route -s <state>: show the routing decision without running the model
• laya presets [name]: list built-in question sets (triage, email, guard, moderation, router)

Examples and benchmarks

cargo run --release --example quickstart
cargo bench --bench latency        # load time, memory, cost per question
cargo bench --bench reliability    # checks that opposite states actually separate

Checkpoints

Weights come from convaiinnovations/laya, cached by hf-hub on first use. Any directory with the same five-file layout loads, including your own fine-tunes.

Routing

The English checkpoint collapses to near-random on non-Latin scripts while staying confident, so confidence gating cannot catch it. Router detects script and language in microseconds before the forward pass and picks the right checkpoint.

laya::detect_language(&json!("Der Kunde wurde zweimal belastet")).is_english;  // false

Pass RouteOptions::lang("de") or RouteOptions::model("multilingual") when you already know. Call preload on servers seeing mixed languages, or the router rebuilds a checkpoint on every language switch.

Confidence caveats

• Only valid inside a checkpoint's competence. Route first.
• Shipped temperatures below 1 sharpen instead of soften. They are clamped to [0.5, 5.0]; clamped buckets are reported at load and listed by Agent::clamped_temperatures(). Treat their confidence as uncalibrated.
• Measure before you gate. The reliability bench flags harm_severity in the guard preset as not separating attacks from benign prompts.

Performance

Everything runs in f32 on the CPU by default. On 24 cores, no BLAS:

Enable the feature that matches your hardware before judging speed:

laya-candle = { version = "0.1", features = ["mkl"] }        # Intel CPU
laya-candle = { version = "0.1", features = ["accelerate"] } # macOS
laya-candle = { version = "0.1", features = ["cuda"] }       # NVIDIA
laya-candle = { version = "0.1", features = ["metal"] }      # Apple GPU

cuda needs the CUDA toolkit (nvcc) at build time; recent GPUs need a recent toolkit (CUDA 12.8+ for RTX 50-series).

Choose the device at run time with auto (the default), cpu, cuda, cuda:N, metal or metal:N:

laya predict -s "..." --device cuda:0       # CLI
LAYA_DEVICE=cuda:1 my-service               # any program using the crate's defaults

let router = laya::Router::builder().device("cuda:0".parse::<laya::DeviceChoice>()?.resolve()?).build()?;

An accelerator asked for by name fails loudly if it cannot be opened (feature not compiled, driver missing, bad index); auto falls back to the CPU and prints a warning when a compiled-in accelerator did not open.

Each question is its own batch row carrying a copy of the state, so 15 questions cost about 10x one. Sequences pad to the longest row in the batch, not to max_len.

Fidelity to the reference

Sequence format, decision head, calibration and routing are ported from Laya's implementation:

[CLS] <type> question: <instructions> [SEP] [MASK] opt0 [MASK] opt1 ... [SEP] <state> [SEP]

Differences from the Python package on GPU: f32 instead of autocast fp16 (small probability shifts, same answers), and clamped temperatures.

cargo test                                          # offline, < 1 s
cargo test --release -- --ignored --nocapture       # end-to-end, downloads ~1.7 GB

Licence

Apache-2.0, matching Laya. See NOTICE for attribution.