Qwen3.8-27B FP8 on a rented B200

A working recipe for burst-serving Qwen3.8-27B on an on-demand NVIDIA B200 rented from RunPod, joined to your Tailscale tailnet as b200, with speculative decoding and no public endpoint at all.

One command rents the GPU, joins it to the tailnet, downloads the checkpoint and serves it. One command gives it back and proves nothing is still billing.

Qwen/Qwen3.8-27B-FP8         target, 29.0 GB, MTP head ships in the checkpoint
lmsysorg/sglang:latest       engine — a MOVING tag, and that matters (see below)
NEXTN, steps 3, topk 1       in-checkpoint MTP speculative decoding
trtllm_mha                   attention backend — WRONG on the other Blackwell part

Engine pools at that config: KV 506,176 tokens, max_running_requests 48, context 262,144. Quality probe 10/10, arithmetic 12/12. Warm spin-up to serving: 2 min 55 s. Cost at N=64: about $0.00044 per 1k output tokens.

Speculation is a mode switch, not a tuning knob. It buys +82% at concurrency 1 and costs 4.4× the KV pool and 3.6× the concurrency — 48 slots instead of 175. Above ~48 requests in flight, turn it off and the same card does 5,694 tok/s at N=64 instead of 4,242. SPEC_K=0 does that.

docs/ is the source of truth — the decisions, every measured number with the arm it came from, the flags and why, and what is still unmeasured. This README is the landing page.

The design in one paragraph

RunPod is told about 22/tcp and nothing else, so it never publishes port 8000 on a public proxy hostname. The pod runs tailscaled in userspace-networking mode — RunPod containers get no /dev/net/tun — and registers as b200. Your client points at http://b200:8000/v1 and that is the only route in. There is no auth proxy to write, no token to rotate, and no endpoint to leak. The cost is that the tailnet name becomes a shared resource with a real failure mode, which is why half the code in down.sh is about freeing it. → ADR: Reach the pod only over the tailnet

Requirements

• A RunPod account with an API key, and the willingness to spend $6.79/hr.
• A Tailscale tailnet, and an auth key that is reusable, ephemeral and pre-approved. Ephemeral is not optional — see the traps below.
• This machine on that tailnet, with curl, python3 and ssh.
• A Hugging Face token. The default checkpoint is not gated, but anonymous downloads are slower and rate-limited.

cp .env.example .env       # then fill it in
./check-requirements.sh    # credentials, tools, and whether `b200` is free

check-requirements.sh runs entirely before anything bills, and separates this will not work (missing credential, name already taken) from this will work but it will cost you (no Tailscale API key, so teardown cannot prove the name came free). --pod additionally reads a live pod's own recorded versions.

Run

./up.sh                    # rent, join the tailnet, serve — billing starts here
./smoke-test.sh            # 3 probes: thinking off, thinking on, tool call
./status.sh                # state, accrued cost, engine pools, in-flight requests
./benchmark.sh             # concurrency sweep — run ON the pod, see rule 1
./chat.py                  # interactive terminal client
./down.sh                  # give it back

Point any OpenAI-compatible client at it:

export OPENAI_BASE_URL=http://b200:8000/v1     # model id: qwen3.8-27b

Common overrides:

DRY_RUN=1 ./up.sh                              # print the request, rent nothing
SPEC_K=0 ./up.sh                               # no speculation: 175 slots, 2.34M KV
MODEL_REPO=Qwen/Qwen3.8-27B SPEC_K=0 ./up.sh   # BF16 reference arm
GPU_TYPES='NVIDIA RTX PRO 6000 Blackwell Server Edition' ./up.sh   # $2.09/hr tier
NO_VOLUME=1 ./up.sh                            # rent anywhere, re-download weights
FORCE=1 ./down.sh                              # tear down with requests in flight
KEEP_VOLUME=0 ./down.sh                        # also delete the 150 GB volume

Clients pass the served name qwen3.8-27b, not the Hub repo id, so it stays stable when you change checkpoints.

Change configuration by recreating, never by restarting. Any PATCH to a RunPod pod restarts its container, and the pod reads its whole configuration from its environment — so an in-place edit restarts anyway and skips the teardown that frees the tailnet name. ./down.sh && ./up.sh is under three minutes on a warm volume.

What it costs, and what it is worth

It bills by the second and nothing stops it but you. down.sh refuses to cut in-flight requests unless FORCE=1, de-registers the tailnet node, then verifies against the RunPod API that the pod is gone — a DELETE that returns 200 and leaves the pod billing is the expensive failure on this path.

Serverless is not cheaper here, and the reason generalises: measured against a real workload, 34.9% duty cycle at concurrency 1 with a mean 4.1-second gap between requests. No flex worker scales to zero on that, so it bills continuously at a rate 27% above the pod. Serverless wins when idle stretches are longer than the ~3-minute cold start — minutes to hours, not seconds. → Which Blackwell tier to rent

The bigger lever is concurrency, not billing model. That same workload ran at concurrency 1 on a card sized for 175, yielding 39 output tok/s averaged over wall-clock from hardware that does 5,306. Pushing the client to 8–16 in flight saves far more than any billing change — but note that with SPEC_K=3 the ceiling is 48, not 175.

Five traps worth inheriting

1. The allowed attention backends differ by Blackwell part, and the two asserts contradict each other. A B200 (SM 10.0) allows triton / trtllm_mha / fa4 and bans flashinfer. An RTX PRO 6000 (SM 12.0) allows flashinfer / trtllm_mha / triton and bans fa4. Qwen3.8 is hybrid GDN, so this fires on every start. There is no flag list that works on both — read the compute capability. The pod does, and keeps a per-arch ordered list, because a wrong first pick costs a 90-second engine start on a host billed by the second, and a pod that exhausts the list sits in sleep infinity looking alive while it bills. Learning this once cost 45 minutes and about $5.

2. RunPod will hand you a dead GPU. One host reported perfectly healthy to nvidia-smi — Default compute mode, MIG disabled, 0 MiB used, no compute apps — and failed every CUDA context with CUDA-capable device(s) is/are busy or unavailable. Downstream that is indistinguishable from a backend rejection, so it burns the whole fallback list after the weight download. The pod now runs a five-second torch matmul before downloading anything, reports STAGE=BAD_GPU, and up.sh terminates and re-rolls onto a different machine.

3. A non-ephemeral Tailscale key breaks the second spin-up, not the first. A terminated pod cannot log itself out. The stale node keeps the name b200, the coordination server names the next pod b200-1, and every client URL silently points at a machine that is not there. There is no in-pod recovery worth having: tailscale set --hostname moves the hostname but not the MagicDNS record. So down.sh deletes the device record before terminating, up.sh refuses to start into a taken name, and the pod itself checks the name it got against the name it asked for.

4. Do not measure throughput over the tailnet. A server doing 5,306 tok/s against 127.0.0.1 reported 272.6 tok/s at N=64 from another tailnet host, on a 70.4-second wall. The DERP relay was the bottleneck, not the GPU. That is a 19× error that looks exactly like a result — nothing in the output says "network-bound". benchmark.sh runs the probe on the pod for this reason. → Benchmark protocol

5. lmsysorg/sglang:latest moves. Every number here was taken on sglang 0.5.17. A pod you rent today is a different engine, and on a related host an engine-layer bump alone moved prose throughput ~9%. Pin IMAGE to a digest if you need comparability with anyone, including us. Otherwise measure your own baseline first and compare only against that.

What the measurements actually say

Speculation: k=3 is the optimum, and k=4 is what proves it. One pod, one checkpoint, k as the only variable:

The mechanism lives in the acceptance rate, not the length: flat at 0.975 through k=2, then 0.958, then 0.906. Once per-token acceptance decays, each further draft position costs more verify compute than it returns. Reading only the length — which rises the whole way — would have picked k=4.

Quantization buys 1.6×, not the 2.25× the weight sizes predict. One pod, one image, one flag set, two checkpoints:

Decode reads 51.05 GB of BF16 weights against 22.67 GB quantized, which alone predicts 2.25×. It does not arrive, because the hybrid GDN state work is not quantized — the mamba cache is 50.78 GB on BF16 and 64.21 GB quantized, and that traffic is BF16 in both arms. Roughly a third of decode traffic is untouched by the weight format, which caps the return on going lower-precision on this architecture. → BF16 versus quantized on one B200

What is not measured, and we say so

The default is Qwen's own FP8 checkpoint. Three of the four reasons for that are solid: on a 178 GiB card the 4-bit weight saving buys no slots that bind (both land at 48 under speculation); "NVFP4" is a label rather than a specification, with builds differing by up to 52% on prose because they disagree about which modules they quantize; and Qwen's FP8 declares its MTP head to the quantizer, where checkpoints that do not make the loader die on a fused name.

The fourth reason — that 8-bit weights sit closer to BF16 than 4-bit weights and activations — is a prior, and it has not been tested here. FP8 scores 10/10 on the ten-task probe against 4-bit's 9/10, but that same probe scored BF16 and 4-bit identically while their greedy prose diverged at word 33 and shared 24.4% of its words. That is noise, not evidence.

Do not cite this repo as evidence that FP8 is more faithful than NVFP4. The instrument that would settle it — KL divergence and top-1 agreement against a BF16 reference — ships here as eval/kl-eval.py and has not been run on these arms. It would take about 40 minutes and $5. → Choosing FP8 on a B200

Why the model behaves oddly here

Qwen3.8-27B is 64 layers in 16 blocks of (3 × Gated DeltaNet → FFN) → (1 × Gated Attention → FFN): 48 linear-attention layers carrying recurrent state, and 16 full-attention layers (24 Q heads, 4 KV heads, head_dim 256). Only those 16 consume KV cache, which is why a 262,144-token window fits at all.

That split explains most of this host's behaviour:

• It is hybrid GDN, which is what triggers the contradictory per-part attention-backend asserts in trap 1.
• The state cache is not a rounding error. At mem-fraction-static 0.90 with no speculation, the mamba cache is 64.2 GB against 71.4 GB of KV cache. GDN state scales with concurrency, not context length, so capping the window does not free it — and it is why speculation, which needs its own per-slot intermediate state caches, costs 4.4× the KV pool.
• Long-context recall degrades by depth, not wholesale. An early needle must survive the most state rewrites, so recurrent drift shows first at depth 0.10 and last at 0.90. That is a different failure from a truncating tokenizer, and no smoke test sees either. eval/needle-check.py and eval/truncation-check.py see one each.

Documentation

A visual walkthrough of the benchmark data — the k sweep, the crossover, what speculation costs, and the negative results — is published at salient-data.github.io/b200-qwen38-27b-fp8.

index.html is generated from benchmark-report.html by ./build-report.sh; edit the report and re-run it, never the generated file. ./build-report.sh --check fails if the two have drifted.

The docs/ bundle is an OKF corpus driven by the lore CLI: every page carries typed front-matter and a schema, so the decisions, the measurements and the procedures cross-link rather than drift. Machine-readable results are in results/.

Portability

The flag set does not transfer between Blackwell parts. flashinfer is banned on SM 10.0 and required on SM 12.0; fa4 is the exact reverse; trtllm_mha is SM100-only. bootstrap.sh reads the compute capability and picks. Do not copy an attention-backend flag from anywhere, including here.

--mem-fraction-static does not transfer across card sizes either. 0.90 is fine on a B200's 178 GiB and contributed to an OOM in graph capture on a 96 GiB card, which is why the pod picks 0.88 below 120 GB.

Nothing here assumes a B200 specifically. The scripts ask for a B200 first and take an RTX PRO 6000 when Blackwell stock is dry, and the pod configures itself for whichever it got. The tailnet name stays b200 either way, so read ./status.sh for the real hourly rate.

Related

A companion recipe for the same model on a DGX Spark (GB10, SM 12.1, 121 GB unified) — a different quantization, a different drafter, and a completely different set of constraints — is at spark-qwen38-27b-nvfp4. Numbers from the two repos are not comparable: different silicon, different engine build, different checkpoint.

References

• Qwen/Qwen3.8-27B — architecture, context, sampling recommendations
• Qwen/Qwen3.8-27B-FP8 — the default target
• RunPod REST API — pod and network-volume endpoints
• Tailscale userspace networking — why the pod needs no /dev/net/tun
• SGLang — the serving engine

License

MIT — see LICENSE.