PESTI - Portable Execution Substrate for Transformer Inference

▄▄▄▄▄▄▄▄▄     ▄▄▄▄▄▄▄▄▄         ▄▄▄▄▄     ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ ▄▄▄▄▄
█████ ▀█████▄  █████ ▀████▄   ▄▓▓▓▓▀▓▓▓▓▄  ▐▓█▓█▀▓███▓▀▓▓█▓▌ █▓█▓█
▓███▓  ▐█▓██▓▌ ▓█▓█▓  ▐▓█▓▓▌ ▐▓▓▓▓▌ ▐▓▓▓▓▌ ▐▓▓▓▌ ▓▓▓▓▓ ▐▓▓▓▌ ▓▓▓▓▓
▓▓▓▓▓   ▓▓▓▓▓▓ ▓▓▓▓▓   ▒▓▒▒░ ▓▒▓▒▒   ▒▒▓▒▓ ▀     ▒▓▒▓▓     ▀ ▓▓▓▒▓
▓▓▓▒▓   ▒▒▒▒▒▒ ▒▓▓▒▓         ▓▒▓▓▒▄              ▒▒▒▒▓       ▓▒▓▓▒
▒▓▒▓▒ ▄▒▒▒▒▒▒▌ ░▓▒▓▒▓▒       ▀▀▀▓▒▓▒░▒▒▀▄        ▒░▒▓▒       ░▒░▓▒
▓░▓▒▒▓▒▒▒░▒░▀  ░░▒░░   ░░░░░        ▀░▒░▒▒       ░░░░░       ░░░░░
░▀█░▀          ▀░▀░░   ▀░▀░▀ ░░▀░░   ▀░░░▄       ░░░▀░       ▀░▀░░
  ▀               ▀     ▀     ▀  ▀   ▒▀ ▀░       ▀ ▀          ▀  ▀
▓ ▓▄▀          ▄▓ ▄▓   ▓ ▄▓▄ ▓▄ ▓▄   ▓▄▄▓▓       █ ▄▓        ▓▄ ▓▄
▒▄▒▒▒          ▒▒▄░▒  ▓▒▒▒▒▓ ▄▒▓▒▄  ▄▄▒▒▒▒       ▓▓▓▒▒       ▄▒▓▒▄
░░░░░          ░░░▒░▄░░░░░░▀ ▀▄░░░░░░░▄▀      ▄▒▒▒░░▄      ░░░░░

Portable Execution Substrate for Transformer Inference *A learning-first Rust substrate for GGUF inference: parse, dequantize, and run transformer forward passes from scratch (with a numpy conformance oracle to prove it)

---

What This Is

PESTI is a Rust workspace for understanding LLM inference internals, built the way you'd build it if you wanted to contribute back to llama.cpp / candle / burn rather than just call them:

• GGUF parsing + pure-Rust dequantization of all K-family quantizations (Q2_K → Q8_0), verified byte-exact against a reference.
• A CPU transformer forward pass (RMSNorm → GQA attention → RoPE → SwiGLU FFN, all 24 layers + output head) that is numerically conformant with an independent pure-numpy reference.
• A CUDA dispatch path (cudarc-based) for GPU kernels, feature-gated so the crate compiles and the parser/dequant layer runs without a GPU.
• A self-contained GGUF-embedded tokenizer — no external tokenizer.json downloads; a Qwen2 GGUF carries its complete tokenizer.

The headline capability is conformance: every forward-pass fix is verified against conformance-corpus/ref_forward.py (an independent numpy oracle), not just "it compiles" or "it runs."

---

Quick Start

# Clone the workspace
git clone https://github.com/crombojambo/pesti.git
cd pesti

# Build the library (CPU path; no GPU required)
cargo build -p pesti-runner --lib

# Run the library test suite (62 unit tests: dequant, tokenizer, kvcache, ...)
cargo test -p pesti-runner --features cuda --lib

# CPU end-to-end generation on the conformance model (coherent text)
cargo run -p pesti-runner --release --features cuda \
  --example cpu_e2e_generate \
  -- conformance-corpus/qwen2.5-0.5b-instruct-q4_k_m.gguf "The capital of France is" 48

# Dump all 24 layers' hidden states + logits (the verified full-model forward
# pass; prints per-layer norms, top-8 tokens, argmax). This is the example the
# conformance check below diffs against the numpy oracle.
cargo run -p pesti-runner --release --features cuda \
  --example dump_all_layers \
  -- conformance-corpus/qwen2.5-0.5b-instruct-q4_k_m.gguf

# Reproduce the 24-layer conformance check (Rust vs numpy) — see below

Note on --features cuda: the transformer module (and therefore the forward-pass examples) is gated behind the cuda feature in src/lib.rs. The library compiles without it (parser + dequant + CPU primitives), but running the transformer examples requires --features cuda even when they execute on the CPU path.

---

Current Status (honest)

✅ Verified working

⚠️ Frontier (not yet done — tracked in docs/ROADMAP.md)

• GPU decode tok/s is real but slow: 0.52-0.60 tok/s on RTX 3070 Ti with fused attention kernel. Far from llama.cpp's ~218 tok/s baseline, but this is the first measured GPU decode throughput — no more synthetic projections.
• GPU end-to-end correctness (Week 17, ongoing) — failed GPU matmuls fall back to CPU GEMM with a gpu_fallback_count() counter (no more silent zeroed buffers). Remaining: per-layer oracle diff against numpy reference, divergence fixes, zero-fallback assertion.
• KV-cache updates during autoregressive generation (paged attention).
• FP8 quantization, multi-GPU scaling.

---

Conformance

The full forward pass is verified against an independent pure-numpy float32 reference on qwen2.5-0.5b-instruct-q4_k_m.gguf (Qwen2.5-0.5B, 24 layers, hidden 896). Full methodology + reproduce commands live in conformance-corpus/CONFORMANCE.md.

Result (2026-08-22, Q4_K_M, 10-token "fox" prompt, last position):

VERDICT: PASS — pesti's full 24-layer forward pass is numerically conformant with the independent numpy reference to within f32 accumulation order. The sub-1e-3 deltas are the expected difference between pesti's Rust Q4_K dequant/accumulation order and gguf's numpy order.

The wiring that was historically buggy is now conformant end-to-end: QKV bias applied, SwiGLU = silu(gate) * up (sigmoid-based SiLU, not silu²), RoPE at the true token position, and per-head GQA (n_head=14, n_head_kv=2).

Reproduce:

MODEL=conformance-corpus/qwen2.5-0.5b-instruct-q4_k_m.gguf

# numpy oracle
python3 conformance-corpus/ref_forward.py "$MODEL" > /tmp/ref_all_layers.txt

# Rust dumper (same output grammar)
cargo build -p pesti-runner --release --features cuda --example dump_all_layers
./target/release/examples/dump_all_layers "$MODEL" > /tmp/rust_all_layers.txt

# head-8 / norm / top-8 / argmax diff
python3 conformance-corpus/compare_all_layers.py \
  /tmp/rust_all_layers.txt /tmp/ref_all_layers.txt --tol=1e-3

# gold-standard full-vector diff (all 896 dims × 24 layers + full logits)
python3 conformance-corpus/probe_all_layers.py "$MODEL" --out /tmp/probe_all_layers
./target/release/examples/dump_all_layers --dump /tmp/rust_probe "$MODEL"
python3 conformance-corpus/compare_full_vectors.py \
  /tmp/rust_probe /tmp/probe_all_layers --tol=1e-3

---

Architecture

┌─────────────────────────────────────────────────────────────┐
│                     pesti-runner                             │
├─────────────────────────────────────────────────────────────┤
│  GGUF v3 parser + pure-Rust K-family dequantization         │
│  (Q2_K → Q8_0, byte-exact)                                  │
│              ↓                                               │
│  Self-contained GGUF-embedded tokenizer (BPE + Qwen2 regex) │
│              ↓                                               │
│  Transformer forward pass (CPU path — verified)             │
│    RMSNorm → GQA attention → RoPE → SwiGLU FFN × 24 layers  │
│              ↓                                               │
│  Output head → logits                                        │
│                                                              │
│  CUDA dispatch path (feature-gated, --features cuda)        │
│    cudarc kernels: GEMM, attention, softmax                 │
└─────────────────────────────────────────────────────────────┘

Feature flags (pesti-runner/Cargo.toml):

• default — empty (CPU parser + dequant + primitives)
• cuda — enables cudarc + the transformer forward-pass module
• rust-tokenizer — optional pure-Rust Qwen2 BPE tokenizer (qwen2-bpe crate)

The MistralRs backend is a tokenizer source (it builds the real tokenizers::Tokenizer from GGUF-embedded arrays), not a separate inference engine. There is no mistralrs inference feature — mistralrs-core is a hard dependency used for tokenization only.

---

Performance

Measured

Projected (synthetic micro-benchmarks — not real decode)

The Week 12/13 numbers below come from isolated kernel micro-benchmarks and backend.sync() proxy timing. They do not represent end-to-end transformer decode throughput. Week 14 replaces these with real measurement.

---

Development Workflow

# Build the library (CPU)
cargo build -p pesti-runner --lib

# Build with CUDA (forward pass + GPU kernels)
cargo build -p pesti-runner --features cuda

# Run the library test suite
cargo test -p pesti-runner --features cuda --lib

# Full test suite (lib + integration tests)
cargo test -p pesti-runner --features cuda --no-fail-fast

# Format the whole crate (rustfmt, edition 2024)
cargo fmt -p pesti-runner

# Conformance tooling (numpy oracle + Rust dumper + diffs)
python3 conformance-corpus/ref_forward.py conformance-corpus/qwen2.5-0.5b-instruct-q4_k_m.gguf

The conformance loop is the core workflow: change a kernel or op → run dump_all_layers → diff against ref_forward.py → confirm the per-layer deltas stay within f32 accumulation order before committing.

---

Project Structure

pesti/
├── pesti-runner/          # Main crate: parser, dequant, transformer, CUDA dispatch
│   ├── src/
│   │   ├── transformer/   #   forward pass (model, layer, rms_norm, linear, kv_cache, tokenizer)
│   │   ├── kernel/        #   CUDA kernels (PTX via include_str!) + dispatch
│   │   ├── dequantize.rs  #   pure-Rust K-family dequantization
│   │   └── gguf_weight_loader.rs
│   └── examples/          #   cpu_e2e_generate, dump_all_layers, ...
├── pesti-gguf/            # GGUF parser crate (workspace member)
├── pesti-safetensors/     # Safetensors crate (workspace member)
├── llm-plug-in/           # llama.cpp FFI runner (workspace member)
├── pesti-gguf-cli/        # GGUF CLI (workspace member)
├── crates/qwen2-bpe/      # Optional pure-Rust Qwen2 BPE tokenizer
├── conformance-corpus/    # numpy oracle + diff tools + canonical Q4_K_M model
│   ├── ref_forward.py     #   24-layer numpy reference
│   ├── probe_all_layers.py
│   ├── compare_all_layers.py
│   ├── compare_full_vectors.py
│   └── CONFORMANCE.md
├── docs/                  # Design docs, benchmarks, roadmap
│   ├── ROADMAP.md         #   Upcoming work, known issues, failure modes
│   ├── benchmarks/        #   Weekly benchmark reports
│   └── history/           #   Archived week results
├── scripts/               # Benchmark and setup scripts
├── tests/                 # Standalone test files
└── examples-disabled/     # quarantined broken examples (git-tracked, not deleted)

pesti-conformance is a standalone crate excluded from the workspace (Cargo.toml exclude list), not a workspace member — cargo test --package pesti-conformance will not resolve from the root.

---

Hardware Requirements

CPU-only (parser + dequant + forward pass)

• Any modern x86_64 or ARM64
• RAM: 8GB+
• No GPU required

CUDA dispatch path

• NVIDIA GPU, CUDA 12.5+
• Tested: RTX 4070 Ti SUPER (sm_8.9, Ada) — primary dev hardware; RTX 5060 Ti (sm_12.0, Blackwell) in a dual-GPU setup

---

Roadmap

For the full milestone-by-milestone plan (including engineering lessons and known gaps), see docs/ROADMAP.md.

Completed

• GGUF v3 parsing + byte-exact K-family dequantization
• CPU transformer forward pass, 24-layer conformant vs numpy
• Self-contained GGUF-embedded tokenizer
• CUDA dispatch path (feature-gated)
• Conformance tooling (numpy oracle + full-vector diffs)

Next (frontier)

• GPU end-to-end correctness + decode tok/s
• Measured llama.cpp baseline on same model/prompt/hardware
• KV-cache updates during autoregressive generation (paged attention)
• FP8 quantization, multi-GPU scaling
• Contribute back to llama.cpp / candle / burn

---

License

AGPL-3.0-or-later Open-source, copyleft, designed for learning and contribution.

---

Last updated: September 11, 2026 (Week 21 — cleanup complete, roadmap restructured) This README will change as I learn more. If it looks perfect, it's lying.