TuiskoLLM

Exact-model inference, built all the way down for the RTX 5090.

Rust · NVFP4 · CUDA Graphs · OpenAI-compatible HTTP

TuiskoLLM specializes the full inference stack for exact checkpoint × GPU targets and fails closed on unsupported combinations. Its OpenAI-compatible server and CUDA kernels are Rust, built with cuda-oxide. Releases are single stripped executables with embedded device code: no PyTorch, Triton, JIT, model conversion, or runtime CUDA Toolkit.

Download and run

Download the Linux x86-64 binary

Requires Linux/glibc 2.34+, an NVIDIA driver, and an RTX 5090. Download the archive, then run:

tar -xzf tuiskollm-*-linux-x86_64*.tar.gz
cd tuiskollm-*-linux-x86_64*/
./tuiskollm serve unsloth/Qwen3.8-27B-NVFP4

TuiskoLLM resolves the selected model's pinned Hugging Face revision, downloading and verifying any missing files. Pass --snapshot PATH to use an explicit checkpoint directory.

After validating and loading the checkpoint, the server listens on 127.0.0.1:8000 by default:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{
    "model": "unsloth/Qwen3.8-27B-NVFP4",
    "messages": [{"role": "user", "content": "Reply with hi."}],
    "max_tokens": 32
  }'

Models and performance

Decode rates are controlled-clock target-graph medians at the measured context; see docs/performance.md for methods and checked baselines.

RTX 5090 / SM120 is the complete product target. SM89 and SM86 remain partial qualification targets, not fallback servers. Vision is not served yet.

What the server owns

• Source-native represented weights: no decode-and-requantize checkpoint conversion.
• Address-stable resident arenas and immutable whole-model CUDA Graphs.
• Exact compact scheduling for every admitted B=1..8 route.
• Paged KV ownership, retained-prefix reuse, cancellation, and retryable overload.
• Blocking and SSE POST /v1/chat/completions; native continuation scoring at POST /v1/evals/loglikelihood; scoring-only OpenAI echo logprobs at POST /v1/completions; plus GET /health, GET /ready, and GET /v1/models. See docs/evaluation.md for the exact lm-eval contracts.
• Qwen3.8 lifecycle administration: /v1/unload/load release and recreate the complete model; /v1/park/resume preserve retained-prefix KV and recurrent state in pinned host memory while releasing all target/MTP arena backing. Resume reloads represented weights from the admitted snapshot at the retained graph addresses before restoring the mirror. Exact-model park/resume device qualification remains pending.
• Content-free request logs with route-specific timing: generation reports queue-inclusive latency, TTFT, decode rate, and cache reuse; prompt scoring reports batch shape, common prefix, queue and scoring time, and scoring throughput.

Build from source

Building requires Linux, Git, rustup, CUDA Toolkit 13.3.73, Clang/libclang 21, the NVIDIA driver, and an RTX 5090. Rust is pinned by rust-toolchain.toml.

cargo run -p xtask -- bootstrap-cuda-oxide
cargo run -p xtask -- build-server

The executable is written to target/cuda-oxide-build-sm120/release/tuiskollm. A plain cargo build does not finalize the embedded device artifacts. Start it by selecting one exact model, for example:

target/cuda-oxide-build-sm120/release/tuiskollm serve \
  unsloth/Qwen3.8-27B-NVFP4

--address ADDRESS overrides the default 127.0.0.1:8000 listener. The lifecycle routes are enabled only for the exact Qwen3.8 target. A non-loopback listener also requires --admin-token-env NAME; requests then authenticate with Authorization: Bearer .... Park is accepted only after previously admitted chats drain. The first chat submitted while parked starts one resume attempt; concurrent chats pend behind that same attempt. Explicit /v1/resume remains available for administrative warmup. POST /v1/park?seconds=N instead enforces a minimum park of N whole seconds after the park completes. Inference requests received during that hold return 503 model_forced_parked with Retry-After; after it expires, the next request auto-resumes normally. An explicit /v1/resume overrides the hold.

For qualification commands and engineering constraints, see docs/performance.md and AGENTS.md. The optional tuisko-llm Python distribution exposes the admitted tokenizer and chat-template frontend; its standards-normalized wheel and sdist filenames use tuisko_llm. It is not an in-process inference API.

Status

TuiskoLLM is experimental and under active development. Performance, supported targets, and APIs may change before 1.0.

License

MIT OR Apache-2.0, at your option. See LICENSE-MIT and LICENSE-APACHE.