Qwen3.8-Flash-Next NVIDIA NVFP4 — Single GB10 SGLang

Native NVIDIA Qwen3.8-Flash-Next-NVFP4 (checkpoint revision fc694b54fb0174e0913e6adf86691ef85a4ead47, unmodified) served by SGLang on one NVIDIA GB10 (SM121, Linux ARM64, 128 GB unified memory): native NVFP4 W4A4 routed experts (FlashInfer CUTLASS), FP8 block-scaled integrated MTP (NEXTN, steps=3), BF16 full vision (image / multi-image / video), bounded file-backed PLE embedding offload, full decode CUDA graphs, 262,144-token context and shared physical KV pool on one device.

Machine-readable verdict: releases/recovery-stable-20260909/qualification-summary.json — status RUNTIME_Q200_AND_RETRIEVAL_VERIFIED.

Click-run

Prerequisites: a GB10-class machine (SM121 / DGX Spark, Linux ARM64), NVIDIA Docker, ~200 GB free NVMe for image + checkpoint + PLE + caches, no other workload on the GPU. Model weights are not included (NVIDIA / Qwen license terms); download them once from Hugging Face:

# 1. Image (public, anonymous pull verified; 33.9 GB compressed)
docker pull ghcr.io/r0b0tlab/qwen38-flash-next-nvidia-nvfp4-sm121-sglang:v1.0.0-sm121-nextn

# 2. Checkpoint (~132.7 GB, 25 files)
pip install -U "huggingface_hub[cli]"
hf download nvidia/Qwen3.8-Flash-Next-NVFP4 \
  --revision fc694b54fb0174e0913e6adf86691ef85a4ead47 \
  --local-dir /absolute/path/Qwen3.8-Flash-Next-NVFP4

# 3. Full-byte checkpoint verification (several minutes; produces the receipt
#    the launcher demands). Run from a clone of this repo:
python3 scripts/verify_files.py \
  /absolute/path/Qwen3.8-Flash-Next-NVFP4 locks/sources.json \
  --receipt /absolute/path/receipt.json

# 4. Launch (owned lifecycle: preflight, receipt gate, watchdog, scoped stop)
sh scripts/run.sh \
  --image sha256:2ee545cf877ae8497c123637e061b6e6313c624e30018f975969b1d554e27f56 \
  --profile profiles/nextn-262k-c2-s3.json \
  --sources locks/sources.json \
  --model-root /absolute/path/Qwen3.8-Flash-Next-NVFP4 \
  --cache-dir /absolute/path/cache \
  --state-dir /absolute/path/attempt-state \
  --receipt /absolute/path/receipt.json \
  --receipt-sha256 <SHA256-PRINTED-BY-STEP-3>

# 5. Wait for readiness (~9–10 min: weight load ~9 min, graph capture seconds),
#    then verify identity:
curl -s http://127.0.0.1:30080/v1/models
#   expected: "id":"nvidia/Qwen3.8-Flash-Next-NVFP4","max_model_len":262144

# 6. Chat (OpenAI-compatible; thinking is on by default for this model)
curl -s http://127.0.0.1:30080/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "nvidia/Qwen3.8-Flash-Next-NVFP4",
  "messages": [{"role": "user", "content": "What is 17*23? Reply with the number only."}],
  "max_tokens": 256, "temperature": 0}'

# 7. Stop exactly what you started:
python3 scripts/guard_stop.py --record /absolute/path/attempt-state/launch-record.json

Alternative distribution: the exact same image as a 14.6 GB zstd tar in the v1.0.0-sm121-nextn release (container.tar.zst + container.tar.zst.sha256; zstd -dc container.tar.zst | docker load). Rebuild-from-source inputs (repro-source.tar, pinned wheels) ship with the release.

Smoke expectation: the canary above returns 391; a first-token sanity reply returns cleanly with finish_reason: stop.

Identity chain (what "qualified" binds to)

Serving profile (exact resolved server configuration)

Measured host/GPU memory & startup (post-suite server telemetry)

Qualification results (this exact image + profile)

Long-context retrieval — NIAH 9/9 PASS (exact-token transport)

All at temperature 0, max_tokens 4096, generous 43,200 s timeout, prompts POSTed as exact token IDs against the real checkpoint tokenizer. One full-window case per serve epoch (see "Known limits"): a second consecutive 258k-token prefill on the same epoch legitimately walks host memory toward the watchdog floor.

Pass = final answer is exactly the ordered pass-code list after thinking; server-echoed prompt_tokens matches the constructed count exactly.

Quality — Q200-v2: 185/200 = 92.5% (SCORED_WITH_DISCLOSURES)

Frozen text-180 corpus + official BFCL v4 multi_turn_base structural-hard20 (graded by bfcl-eval==2025.12.17). All 200 transported and graded, zero grader errors.

Disclosures: (1) two model nonterminations (ifeval-023, hard-01) burned the full 8,192-token output ceiling on reasoning and emitted an empty final answer; scored incorrect, not transport failures. (2) Hard-reasoning answers were independently adjudicated; a known erroneous reference answer was not allowed to mark correct math wrong. (3) Single-GPU C2 server, mixed quality cohort — no throughput claim from this run.

Vision — 11/11 PASS (semantic-checked, not just HTTP success)

8 image (color/geometry/OCR/counting), 1 multi-image, 2 video order-tracking; median TTFT 2.88 s. Input/media hashes validated.

Vision benchmark — r0b0bench-vision v1.0 (4,703 rows, 4 suites)

Frozen public contract (r0b0tlab/r0b0bench scripts/vision/; pinned dataset revisions, deterministic graders, thinking-off, temperature 0, single image per request). Two result sets:

Protocol notes: --workers 2 (matches max_running_requests 2; the contract's default 4 assumes ≥4), MMVP's paired metric is directional (±8 pp at p≈0.5), RealWorldQA is CC-BY-ND (aggregates only, no images redistributed). The v1.0.0 numbers were assembled across three serve epochs with identical protocol/loaders/graders — full disclosure in the summary JSON's assembly and defects blocks (results/vision/).

v1.0.1 numbers: same frozen protocol on the rebuilt image (sha256:47219fecc31b…), assembled across two serve epochs of that one image/profile (4,371 rows + 332-row ocrbench resume) because the late-ocrbench retainer below still trips a 4 GiB safety floor after ~670 consecutive ocrbench rows. Within-Wilson agreement with the v1.0.0 result on every suite.

v1.0.1 vision-serving fixes (why the image was rebuilt): upstream ViTCudaGraphRunner retained an unbounded per-image-shape graph cache; real benchmark traffic (185 distinct image sizes in the first 400 cvbench rows) drove ~48 MiB/row of permanent unified-memory growth until the safety watchdog stopped the serve. The shipped patch adds an LRU budget (SGLANG_VIT_MAX_GRAPHS, default 16) with an in-flight replay refcount so a live replay is never evicted. With graphs ON the budget now holds — but each retained large-image graph keeps a ~1 GiB private pool, so bounded-16 still costs ~20 GiB on a 128 GiB unified node; sustained mixed-shape vision work therefore ships graphs OFF (profiles/vision-novitgraph.json, byte-identical to the production profile except vision.cuda_graph). The 4 GiB MultiModalStaticCache behaves as a configured bound, not a leak.

Throughput & speculative decoding

Identical frozen method on both sides (16 prompts, thinking off, temp 0, streaming). Read honestly: this runtime is 1.57–1.66× slower than the 2×GB10 W4A16 Marlin derivative at identical acceptance — the entire gap is per-step cost, and the often-quoted "3×" was a SHORT-lane method artifact. Short-lane request-E2E screen (separate measurement, incl. prefill/TTFT): C1 SHORT/MED/PROSE 21.35 / 23.83 / 26.43 tok/s per request; C2 batch aggregate 42.09 / 26.32 / 36.31 tok/s. What this runtime buys: the whole 262k pool, vision, and FP8-native MTP on one device.

Test suites

Packaged source tests at build: 558 passed / 4 skipped. Repo test suite at publication SHA: 671 passed / 5 skipped (CPU-only — fake Docker/probe boundaries; explicitly not GPU qualification). CI runs the suite on every push.

Known limits (disclosed, not hidden)

• One 258k-token prefill per serve epoch. During full-window prefill the host dips to ~7 GiB MemAvailable; a second consecutive full-window prefill walks it ~2 GiB lower and the owned watchdog will stop the server (guard rc=8). The watchdog floor is operator-tunable via --mem-available-floor-gib (default 8, clamp [4, 16]); any override is written into launch-record.json.
• Marlin FP4 model rollout failed closed on this checkpoint (oracle image): not used, not benchmarked, not shipped.
• NVFP4 KV / QSA: primitive + graph replay proven on SM121, but the checkpoint ships no KV quantization scales and QSA integration is not enabled — KV stays BF16.
• Not claimed: multi-node serving, globally optimal configuration, long-context reasoning quality (NIAH is functional retrieval evidence), redundancy/HA.
• No Windows/macOS; no SM120 consumer-GPU recipe for this NVFP4 checkpoint (the W4A16 sibling project covers dual-RTX PRO 6000).

Repository layout

• runtime/ — strict versioned launch-profile contract (unknown-field, enum- and range-checked) and the exact server argv compiler; no shell commands are ever generated.
• profiles/ — frozen launch profiles (nextn-262k-c2-s3.json is production).
• scripts/guard.py + run.sh — owned launcher: preflight, receipt gate, hardened spawn (read-only mounts, caps dropped, UID 1001:1001), watchdog with claim-bound memory floor, scoped cleanup.
• scripts/verify_files.py — streamed full-byte checkpoint verification → CHECKPOINT_VERIFIED receipt; scripts/audit_checkpoint.py — structural header admission (explicitly not integrity verification).
• scripts/preflight.py — fail-closed host admission (idle GB10, ≥104 GiB host RAM available, per-filesystem disk budgets).
• scripts/niah.py — exact-token NIAH harness (fail-closed verdicts, bounded infra-only retries, per-attempt rows).
• releases/recovery-stable-20260909/ — the qualified package: summary, evidence (performance/vision/q200 JSON), launch helper, checksums.
• tests/ + .github/workflows/ci.yml — CPU-only pytest suite in CI.

Launcher contract (short version)

sh scripts/run.sh requires --image sha256:<config-id>, --profile, --sources, --model-root, --cache-dir, --state-dir, --receipt, --receipt-sha256 (last = SHA256 of the receipt from step 3 of click-run). Checkpoint, cache and state dirs must be disjoint absolute canonical paths. --print/--dryrun plans without touching the host. Exit classes: 2 usage, 3 preflight, 4 integrity/binding, 5 spawn, 7 cleanup, 8 watchdog, 9 lock; server exit codes are preserved; TERM/INT → 128+signal. Cleanup retains the CID on failure for scripts/guard_stop.py --record <launch-record.json>. No automatic retry; no foreign-container deletion.

Licenses

Code: MIT (see LICENSE). Upstream SGLang (Apache-2.0) attribution in third_party/ and NOTICE. The checkpoint remains subject to the NVIDIA Open Model License and applicable Qwen terms; weights are not redistributed here.