VITRIOL

"Visita Interiora Terrae Rectificando Invenies Occultum Lapidem"

(Visit the Interior of the Earth, by Rectifying you will find the Hidden Stone)

Why VITRIOL?

VITRIOL is my attempt at using every optimization possible to run modern AI models on old hardware that would have no business running them otherwise. I am talking here specifically about my old desktop PC with a narrow PCIe bus, an outdated GPU, a CPU without AVX2 instructions, so offloading isn't even practical. With those constraints in mind I had a simple question: Could I stream directly from system RAM to GPU. It took a while, but I managed through a solid direct memory access system and a custom copy engine. Every optimization I am doing now is really one big mad computer science experiment to make old hardware punch above its weight, and possibly introduce those optimizations for newer hardware. This means VITRIOL is not a stable repo in the slighest. Not yet at least. I am trying to use the latest papers and research on LLM inference to squeeze every bit of performance out of my silicon, and I hope this comes to your benefit as well.

Regarding the name, in alchemy, vitriol was considered the ultimate catalyst for transmuting matter. This is something this application aims to do as well. To transform old hardware into a high performant AI transformer. Lead into gold. You know the drill. Aditionally, around the 15th century, the esoteric backcronym was formed: "Visita Interiora Terrae Rectificando Invenies Occultum Lapidem". In a way, this is what we are doing. We are reaching down into the bowels of the computer, rectifying the data streams, and running inference on our newfound philosopher's stone. Yes, I know how incredibly silly this all sounds, but it makes me happy to be using archaic alchemical terminology. Regarding the logo: In alchemical texts and artwork, vitriol was often depicted as a "Green Lion devouring the Sun". This is a metaphor for sulfuric acid dissolving base metals (symbolically represented by the green lion) to extract and purify precious gold (the alchemical symbol for which is the sun).

What VITRIOL Is

VITRIOL is a fork of llama.cpp whose goal is optimal inference on machines that are normally too constrained to run the model at all — VRAM-starved GPUs, narrow PCIe buses, CPUs without modern vector instructions, DDR3-era memory, single-digit-gigabyte boxes. Every optimization is measured against one standard: does it make the model run, or run faster, on silicon that the model has no business running on?

The operating point changes with the hardware:

• Resident mode (default, VITRIOL_MODE=off). When the model fits in combined VRAM, weights stay fully VRAM-resident. This is the residency rule: streaming a fitting model is a measured pessimization on narrow buses. Optimizations then target the context — TurboQuant KV, attention-probe sparse eviction, MTP speculative decoding — to stretch depth per byte.
• Streaming mode (optional, VITRIOL_MODE=stream). When weights exceed combined VRAM, MoE expert tensors live in page-locked host RAM and stream to the GPU over PCIe DMA, accelerated by an LRU VRAM cache, expert pinning, predictive prefetching, and an approximate expert output cache. This is the "RAM Shot" line of work — for the model class that does not fit, not for the one that does.

The same tree also ships Officina, a full-screen agentic coding workshop welded to the engine over its HTTP surface, and an operational layer (launcher, calibration tool, profiles, systemd units) that turns the engine into a durable appliance instead of a process that dies with the first memory crisis.

What sets VITRIOL apart

Depth, not window. A context window allocation says nothing about usable filled-context depth. Every VITRIOL capability claim carries a filled token count measured by chunked/single-shot prefill plus decode-at-depth. Shallow benchmarks do not certify deep-context operation.

⚗️ Officina — the built-in coding workshop

VITRIOL is not just an inference engine — it ships Officina, a full-screen agentic programming environment that is detached-but-coupled: it runs as its own workspace in whatever project directory you point it at, yet it is welded to the engine over the HTTP surface and uses that coupling for real optimization.

vitriol serve          # the engine wakes up (any lane, any model)
cd ~/my-project        # the workshop opens for THIS directory
vitriol officina       # the stone greets you; start working

What the tight coupling buys you — none of which a generic agent harness can do:

• Live engine telemetry in your editor. Braille gradient gauges (the same ones the engine TUI draws) show slot occupancy, decode tok/s, and boot decode totals, streamed from /metrics and /slots. You never open a second terminal to know how far along your model is.
• Context tuned to the real window. Officina knows the engine's certified context — it probes /props at startup and registers the actual n_ctx, not a catalog guess.
• Checkpoint-aware. Warm KV checkpoint/restore and /rewind (worktree
  • context from one turn key) ride the engine's checkpoint endpoints.
• The composer, the panel, the stone. A braille watermark of the VITRIOL logo greets an empty workshop and dissolves on first input; a cordoned panel shows your coupling, folder, token flow, and touched files; the composer sits pinned at the floor of the screen.
• Couplings. The default coupling is Lapis Occultus – VITRIOL: whatever the engine is serving, we use. Alternative providers (e.g. a euro-capped cloud escalation) hot-swap mid-session with the conversation intact.

Officina is self-contained in officina/ and docs/OFFICINA.md is its manual: extensions, Plan/Build agent modes (TAB), /history, /panel, /resume, couplings, and the design law — upstreams are mined, the workshop is ours. Related reading: docs/SELF-SUFFICIENCY-2026-08-31.md (the standalone guarantee), docs/PROVENANCE.md (every borrowed thing, cited), docs/SYSTEMS-MAP-2026-09-01.md (who consumes what), docs/LAYOUT-FORK-2026-08-31.md (the docked-shell build), and docs/HANDOFF-2026-08-31.md (session handoff).

History

The project has lived three eras. Each one is preserved as history in docs/VERDICTS.md — every dead idea carries its own measurement and reason.

• War I — fit the model (35B MoE era). A Qwen3.6-35B-A3B needs ~12 GB of weights; the GPUs here had 8 and 12 GB. The answer was to not fit it: page- locked host RAM ("RAM Shot"), a custom copy engine, Chimera dual-backend (CUDA+Vulkan), expert streaming. It worked — 23.3 tok/s where the baseline was "doesn't boot". Then the project measured its own darling: on DDR3 + narrow PCIe, streaming a model that fits VRAM is a pessimization. That verdict became the residency rule, and most of War I's machinery became a tombstone — a correct answer for its era, outlived by hardware.
• War II — afford the model (27B era). A Qwen3.8-27B nearly fits across two GPUs. The answer was to buy the missing 8 GB (a secondhand GTX 1070 Ti beside the RTX 3060), then make the remainder behave: TurboQuant KV at 3.5 bpw, LULL attention-probe scoring to decide which KV cells earn their keep, slot tenancy so tenants share one server without stealing context. Certified depth: 96,836 filled tokens @ 11.32 tok/s — measured, not window-allocated.
• War III — keep it alive (current). A model that runs on a red-lined 16 GiB box dies differently: OOM kills, swap-thrash hangs, task-queue jams. VITRIOL became an appliance: disk checkpoints that survive crashes, a sidecar that replays conversation warmth into fresh instances within seconds, a hang watchdog, a proactive bounce that restarts cleanly before memory exhaustion wedges the box, and an oom-shield that works backwards — since unprivileged users cannot protect their own processes, it marks everything else as more killable.

Reading the repo: docs/ARCHITECTURE.md is the single source of truth for current behavior. docs/VERDICTS.md holds every dead idea and why it died. Sections of this README dated to Wars I–II are preserved as history.

Hardware assumptions (what's tuned where)

This tree is tuned for one specific machine: i7-3770 (no AVX2), 16 GiB DDR3 + zram, RTX 3060 12 GiB + GTX 1070 Ti 8 GiB. Tensor splits (ts 24,12), KV quant choices, cache caps, and sidecar thresholds are all this-box numbers. They live in profiles/ (personal) and profiles/examples/ (generic starting points) — re-tune there, not in code.

Upstream posture

main is the canonical daily-driver branch in both repos: the outer repo and the inner llama.cpp/ fork. The inner vitriol branch is a frozen pre-port archive; master tracks the fork's published state. Merge cadence: when upstream grows something we want, not before.

Quick Start

# 1. Clone with submodule (llama.cpp is pinned)
git clone --recursive https://github.com/Randozart/VITRIOL.git
# Or if already cloned: git submodule update --init --recursive

# 2. Build. The dual-GPU daily driver needs BOTH archs (sm_61 Pascal + sm_86 Ampere).
cd vitriol/llama.cpp && cmake -B build -DGGML_CUDA=ON -DGGML_NATIVE=ON \
  -DCMAKE_CUDA_ARCHITECTURES="61;86" \
  -DLLAMA_BUILD_TESTS=OFF -DLLAMA_BUILD_EXAMPLES=OFF \
  && cmake --build build -j$(nproc)

# 3. One-time capability grant (CAP_IPC_LOCK; optional on hosts where mlock is free)
./vitriol setup

# 4. Configure + calibrate
./vitriol calibrate --quick
./vitriol config

# 5. Run (or load a blessed profile)
./vitriol config load qwen38-mtp-131k
./vitriol run

The engine is designed to run as a managed service:

sudo systemctl restart vitriol-server.service   # the daily driver

The launcher refuses bare vitriol serve while the systemd unit is active — restart via the unit, never bare.

Configuration

VITRIOL's feature flags control memory, context efficiency, and retrieval. Each has measurable trade-offs between throughput, context size, and recall quality. Config lives in ~/.vitriol/config; profiles (vitriol config save|load) switch between blessed operating points.

Production speed profiles use q4_0 KV; the master deep-context profile uses tq3_0. Measure per profile — TurboQuant trades a decode penalty for depth on some configs. See docs/CONFIG_REFERENCE.md for every flag and docs/RECOMMENDED_SETTINGS.md for the blessed operating points.

Memory mode (the Flask shim + SQLite emulated-memory era) is superseded by Officina's built-in memory — see the Officina manual.

How It Works

Resident mode (the daily driver)

Weights live in VRAM, split across both GPUs. The context is where the squeeze is:

• TurboQuant KV. K and V caches quantized to 3.5 bpw (tq3_0) with Walsh–Hadamard-rotated quantizers, per-device asymmetric overrides.
• Sparse KV eviction. An attention-probe graph (q·K softmax, exponential decay) runs on a sibling GPU during decode lulls, every 16 steps. Lowest- scored middle cells are evicted first; sinks and recent cells are protected. The eviction counter surfaces as ⤓ Nk in Officina's context row.
• MTP speculative decoding. The embedded MTP head drafts one token ahead; verification is parallelized. Single-token decode measured +40% shallow, +31% at depth on Qwen3.8-27B. Draft tokens are excluded from probe scoring.
• Context checkpoints + slot save/restore. Disk checkpoints (slotN.bin) carry KV + recurrent state; warm-resume after a crash takes ~300 ms. The persistence chain is gated on a text-only engine — loading a multimodal projector (--mmproj) shuts it off.

Streaming mode (MoE that does not fit)

When weights exceed combined VRAM, MoE expert tensors live in page-locked host RAM and the GPU reads them over PCIe DMA:

VITRIOL buffer type (CUDA experts)
  ├─ 1. Allocation
  │     mmap(hugepage) → mlock → cudaHostRegister (page-locked, DMA-accessible)
  ├─ 2. Model load: experts land in the host-RAM buffer, base model in VRAM
  └─ 3. Inference: is_host=true routes MUL_MAT_ID to CUDA; GPU reads experts
        over PCIe (~12 GB/s), accelerated by:
        ├─ LRU VRAM cache   — hot experts cached in VRAM, async DMA prefetch
        ├─ expert pinning   — first N layers' experts preloaded to VRAM
        ├─ predictive prefetch — cross-layer + temporal union, zero training
        └─ output cache     — approximate reuse of routed experts' FFN outputs

Streaming is gated per-op: hooks fire only when the expert tensor lives in a VITRIOL buffer. Resident operation sees zero routing change.

Dual-GPU operation

-ts 24,12 splits the model across the RTX 3060 (12 GB) and GTX 1070 Ti (8 GB). Per-device KV quantization (VITRIOL_KV_QUANT_K_GPU<d> etc.) and per-device pin ranges (VITRIOL_PIN_FIRST_N_LAYERS_GPU<d>) let each card be tuned to its own headroom. The calibration tool (vitriol calibrate --quick) computes VRAM from GGUF tensor data — no hardcoded model constants.

Distributed operation (row-split over RPC)

A second machine can host a slice of the layers over the llama.cpp RPC backend (raw TCP on a Tailscale mesh): [gpu] rpc_servers = <host:port> and a 3-way -ts [boxB, GPU0, GPU1]. Load qwen38-distributed profile to get the measured split (5% to box B). The win is capacity — box B's large unified memory holds KV/context box A's VRAM cannot — at a decode cost (~10-11 t/s vs 18.94 local; prefill is box-B bandwidth-bound). Full setup, measured table, and limits: docs/DISTRIBUTED_INFERENCE.md.

Hardware & Compatibility

Tested daily driver

• GPU pair: RTX 3060 12 GiB (sm_86) + GTX 1070 Ti 8 GiB (sm_61), PCIe Gen3
• CPU: Intel i7-3770 (Haswell, no AVX2) — orchestrates, does not compute experts
• RAM: 16 GiB DDR3 + zram
• OS: Ubuntu 24.04, kernel 6.17+, NVIDIA driver 535.288.01, CUDA 12.2
• Model: Qwen3.8-27B-Q3_K_M (unsloth, qwen35 arch, embedded MTP head)
• Certified depth: 96,836 filled tokens @ 11.32 t/s (IQ3_S, tq3_0, ts 26,10)

Likely works

• NVIDIA GPUs CC ≥ 5.0: Pascal, Turing, Ampere, Ada, Blackwell. The cudaHostRegister + PCIe DMA path is architecture-agnostic.
• Maxwell (CC 5.x): may lack some kernels — exclude or set exclude_secondary = true.
• ROCm/HIP (AMD): mechanical port of CUDA driver calls (hipHostRegister, hipMalloc); ggml already has GGML_USE_HIP guards.
• Windows (NVIDIA): NVCC + cudaHostRegister work identically; the CLI launcher needs a PowerShell wrapper.
• Intel GPUs (SYCL): a full VITRIOL SYCL integration exists (LRU cache, predictive prefetch, hot-expert profile, zero-copy mmap wrap) — for Intel Arc / oneAPI hardware. Requires a SYCL compiler + MKL; not built by default.

Likely won't work

• Intel iGPU (SYCL unified memory): no discrete PCIe bus to cross — VITRIOL's trick is unnecessary there.
• Apple Silicon (Metal): unified memory; the entire model fits if it fits.
• Nouveau: lacks the cudaHostRegister GMMU page-table path.

Durability

• systemd units (system scope): the engine runs under /etc/systemd/system/ with a real OOMScoreAdjust=-500; Restart=always; a polkit rule lets the owner manage exactly the two vitriol units.
• oom-shield: unprivileged users cannot lower their own OOM score, so VITRIOL marks other consumers more killable (+300) — the kernel eats browsers first, never the engine.
• Persistence sidecar: startup restore, autosave with churn guard (frozen /metrics counters ⇒ nothing happened ⇒ skip), clobber protection (staged writes; an empty save can never replace a rich checkpoint), hang watchdog (~60 s health-deaf → restart), proactive bounce (clean restart before memory exhaustion wedges the box).
• Flag provenance: every launch emits a VITRIOL-FINGERPRINT: line and journals it with a per-field diff against the previous launch and the blessed operating point. Silent flag drift is a review blocker — config keys are flags too. Speed-bearing keys ([spec], [kv] score*, ts, ubatch) are provenance-bearing.

Performance

Numbers are depth-certified unless labeled shallow. Window ≠ depth.

Current operating point (2026-09)

Historical (War I — single GPU, 35B MoE, Chimera)

See docs/BENCHMARKS.md, docs/FINDINGS_2026-05-19.md, and docs/plans/COMPUTE_OPTIMIZATIONS.md.

Architecture

┌──────────────────────────────────────────────────────────────────┐
│ TENANTS     hermes-agent (slot 0, 73k)   ontic forge (slot 1, 8k) │
├──────────────────────────────────────────────────────────────────┤
│ ENGINE     llama.cpp fork ("main")                              │
│            residency rule: weights VRAM-resident by default      │
│            LULL attention-probe KV scoring + eviction + reset    │
│            TurboQuant KV tq3_0/1s/4s (3.5 bpw), per-device       │
│            MTP draft head (n_max=1)                              │
│            slot save/restore (--slot-save-path, warm resume)     │
├──────────────────────────────────────────────────────────────────┤
│ RUNTIME    scripts/vitriol launcher (profiles → argv, fingerprint)│
│            systemd units: vitriol-server + persistence sidecar   │
│            oom-shield · hang watchdog · proactive bounce         │
├──────────────────────────────────────────────────────────────────┤
│ TRUTH      libvitriol (Rust calibrator, GGUF-derived VRAM math)  │
│            certification reports: FILLED-depth benchmarks only   │
└──────────────────────────────────────────────────────────────────┘

        ┌────────────── RTX 3060 (12 GiB) ──────────────┐
        │  ts split 24: model layers + KV (tq3_0)       │
        │  compute buffers · MTP head (resident mode)   │
        └───────────────────────────────────────────────┘
                          ▲ PCIe
        ┌────────────── GTX 1070 Ti (8 GiB) ────────────┐
        │  ts split 12: model layers + KV (tq3_0)       │
        │  LRU pool (stream mode) · probe scorer        │
        └───────────────────────────────────────────────┘
                          ▲ PCIe
        ┌──────────── CPU / 16 GiB DDR3 + zram ─────────┐
        │  VITRIOL host buffer (stream mode: experts)   │
        │  slot checkpoints (slotN.bin, warm resume)    │
        └───────────────────────────────────────────────┘

Project Structure

├── vitriol                  ← CLI entry point (symlink to scripts/vitriol)
├── scripts/
│   ├── vitriol              ← launcher: config TUI + run + serve + profiles
│   ├── build-llama-server.sh
│   ├── vitriol-oom-hardening.sh
│   └── lull_slot_persist.py ← persistence + watchdog sidecar
├── libvitriol/              ← Rust calibrator (GGUF parser, VRAM estimator)
│   ├── src/{gguf,probe,estimator,main}.rs
│   ├── gguf_reader.py       ← Python fallback
│   └── sweep_controller.py  ← automated HTTP benchmark sweeps
├── profiles/                ← canonical configs (personal + examples/)
├── officina/                ← the built-in coding workshop
├── llama.cpp/               ← Git submodule (pinned, "main" branch)
│   └── ggml/src/ggml-cuda/
│       ├── vitriol-buffer.{cpp,h}          ← RAM Shot buffer type
│       ├── vitriol-cuda-integration.{cpp,h}← LRU + pin + predictor + output cache
│       ├── vitriol_copy_engine.{cpp,h}     ← copy-engine DMA (Phase 1)
│       └── tq3-native.cu, turbo-wht.cu     ← TurboQuant KV kernels
├── vitriol-daemon/          ← experimental NVMe→GPU DMA kernel module
├── systemd/                 ← unit files (system scope)
├── docs/                    ← living documentation
├── .opencode/plans/         ← agent session reports (the lab notebook)
└── EXPERIMENT_LOG.md

Ars Priori & Acknowledgements

VITRIOL stands on the shoulders of giants. Every core insight — DMA over PCIe, metapage completion signaling, async expert prefetching, extreme quantization on legacy hardware — was reverse-engineered from the following works. We document our debt explicitly.

Inference Engine

GPUDirect Storage & DMA

Async Scheduling & MoE Orchestration

🏆 VITRIOL-Implemented Techniques

Speculative Decoding

KV Cache & Context Management

CPU Offload & Ternary Compute

Extreme Quantization & Compute

Emulated Memory & Context Retrieval

Chunked Recurrence Inference

Agent Harness & Context Efficiency (Officina)

Measured result of the stack (2026-08-31): ~20K live context against ~200K cumulative offloaded in a working session (~10:1 discard ratio) — full record in .opencode/plans/officina-context-efficiency-record-2026-08-31.md.

See docs/PROVENANCE.md for the file-level citation registry, and docs/provenance/ for per-module headers. VITRIOL is licensed Apache-2.0 OR MIT (see LICENSE, LICENSE-MIT); upstream projects are mined for insight, never depended on as runtimes, and GPL sources are re-derived only.

See docs/OPTIMIZATION_PLAN.md for the full V2 roadmap with implementation phases.