plow

Packet Language for On-device Warps — an LLM inference stack from Infervisor.

plow compiles a Hugging Face checkpoint into a static packet stream, then runs it with a persistent on-device interpreter: one cooperative GPU launch that stays resident, executes ops at warp granularity, and coordinates with counters instead of per-op CPU dispatch.

Architecture

Architecture chapters: docs/arch/ — compiler pipeline, tile graph, scheduler, packet ABI, counter system, runtime, cost model, formal verification, multi-GPU. Build-system rationale: docs/BUILD.md. Every emit/build/runtime flag: docs/flags-reference.md.

Bringing up a new model: a staged playbook — operator IR → rewrite rules → formal verification → kernel tuning → single-block sweep → runtime optimization → measured campaign — lives in docs/bringup/, with per-stage LLM-agent prompt templates in docs/bringup/agents/.

Fully supported today

These paths are the ones exercised end-to-end for serving. Start here.

--gpu sets the SM/CU count from the built-in GPU registry (--n-cu overrides it for unknown or partitioned parts). PLOW_UNISEG=1 is NVIDIA-only — on AMD it collapses wave-class segments and breaks prefill. An emit is not portable across SM counts: emitting for the 188-SM 6000 Pro and running on a 170-SM 5090 (or the reverse) mis-schedules work.

Also emit-capable (not the first-run walkthrough): Qwen3, Llama-3.1; bf16 and weight-only fp8 (e4m3). Descriptors exist for other parts (H100, B200) — do not treat those as drop-in substitutes without matching interpreter objects.

Requirements

• Nix with flakes. The flake provides everything — Rust, CMake, Lean, and the CUDA and ROCm toolchains. Kernel and interpreter builds need no system toolchain: no /opt/rocm, no /usr/local/cuda.
• A GPU driver at runtime. NVIDIA: libcuda.so.1 from the driver. AMD: the amdgpu kernel driver — user-space ROCr and its libraries come from nix.
• A local Hugging Face checkpoint directory for the model you serve. The walkthrough uses google/gemma-4-12B-it at $HOME/models/gemma-4-12B-it.

plowrt does not link CUDA/HIP. Features cuda / hsa dlopen the drivers at runtime.

Quickstart

All commands run inside nix develop (or via nix develop --command …).

1. Build the host tools

cargo build --release -p plowc
cargo build --release -p plowrt --features cuda,hsa

Binaries: ./target/release/plowc, ./target/release/plowrt. Optional: cargo test --workspace and (cd lean-plow && lake build).

2. Build the interpreter objects

One command per target — the dev shell provides nvcc/hipcc from nix:

ASSETS="$HOME/plow-assets/gemma4-12b"; mkdir -p "$ASSETS"

# NVIDIA sm_120 (5090 / 6000 Pro) — cubins land next to the given path
scripts/build_sm120_cubin.sh "$ASSETS/interp_sm120.cubin" -DPLOW_NV_FA_GF_FULL=4

# AMD gfx950 (MI350X / MI355X)
scripts/build_gfx950.sh build-amd/hsaco
ln -sfn "$(pwd)/build-amd/hsaco" "$ASSETS/hsaco"

# AMD gfx942 (MI300X) — PLOW_OCC4=1 is the batch-1 occupancy profile
PLOW_OCC4=1 PLOW_L2HIER=1 bash scripts/build_gfx942.sh build-amd/hsaco/gfx942
ln -sfn "$(pwd)/build-amd/hsaco/gfx942" "$ASSETS/hsaco"

Hermetic alternative: nix build .#plow-interp-sm120a / .#plow-interp-gfx950 / .#plow-interp-gfx942 (objects in result/cubin/ and result/hsaco/<arch>/).

3. Compile the packet

One command; take the flags and env for your GPU from the support table:

CKPT="$HOME/models/gemma-4-12B-it"

# example: MI355X
./target/release/plowc --hf-dir "$CKPT" \
  --arch gfx950 --gpu mi355x --max-ctx 131072 \
  --out "$ASSETS"

This writes model.pkt + weights.json into $ASSETS and symlinks checkpoint → $CKPT and tokenizer.json. The interpreter objects from step 2 must already be in place.

Sanity check on AMD: a correct Gemma-4 dense emit reports 121 segments per prefill bucket in build.json (2·layers + 1).

On gfx942, the fp8 env selects per-channel fp8 serving (weights and activations); scripts/quantize_fp8_head.py builds the quantized lm_head shard. MX-FP4 stays gfx950-only — CDNA3 has no fp4 hardware and plowc refuses it at emit.

4. Serve and chat

./target/release/plowrt serve --assets "$ASSETS" --port 8080

curl -s http://127.0.0.1:8080/v1/models | jq .

curl -s http://127.0.0.1:8080/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{
    "model": "gemma-4-12b-it",
    "messages": [{"role":"user","content":"What is the capital of France?"}],
    "max_tokens": 64
  }' | jq .

curl -s http://127.0.0.1:8080/v1/completions \
  -H 'content-type: application/json' \
  -d '{"model":"gemma-4-12b-it","prompt":"The capital of France is","max_tokens":32}' \
  | jq .

plowrt serve after a successful load — Gemma-4 12B (~22 GiB weights) on an NVIDIA GH200, OpenAI-compatible API on TCP:

Measured campaigns and their protocols live in perf-data/plow-gfx942/. The full Kimi-K3 TP8/MI325X build and serving recipe is docs/amd/kimi-k3-mi325x-recipe.md.

Asset layout (what serve expects)

After the quickstart, $ASSETS contains at least:

model.pkt
weights.json          # "network": "gemma-4-12b-it"
tokenizer.json -> …   # symlink into the HF dir
checkpoint -> …       # symlink to the HF dir (weights mmap’d / uploaded)
# NVIDIA:
interp_sm120.cubin
interp_sm120_pf.cubin
# AMD:
hsaco/ -> …           # interp_decode.elf, interp_prefill.elf, interp_flash.elf, …

The API slug is the lowercased checkpoint directory name (e.g. gemma-4-12b-it); plowc and /v1/models both derive it this way. Confirm it before curling:

jq -r .network "$ASSETS/weights.json"
# → gemma-4-12b-it

Streaming: "stream": true. Also /healthz, /metrics. Multiple --assets DIR register more models.

Contributing

See CONTRIBUTING.md. Please follow the Code of Conduct.

Security

Report vulnerabilities privately — see SECURITY.md.

License

Copyright 2026 Infervisor.

Licensed under the Apache License, Version 2.0.

Code of Conduct

This project follows the Contributor Covenant. Report unacceptable behavior to lava@infervisor.ai.