Eider

Eider is a native inference server for the NVIDIA DGX Spark. It runs NVFP4 and mixed-precision models directly on the GB10 sm_121 GPU.

The runtime uses Rust, CUDA, and cuBLASLt. It does not depend on PyTorch, llama.cpp, vLLM, or another tensor runtime.

Eider has three priorities:

• It starts models quickly without the large software stack that vLLM uses.
• It supports fast iteration on GB10-specific kernels, especially for NVFP4 paths that llama.cpp does not support well.
• It provides a practical platform for experiments and for learning how inference systems work.

Qwen3.8 Flash Next

Eider served Qwen3.8 Flash Next on its release day, August 26, 2026. A live Pi session reaches about 200 tokens/sec for cold prefill and 12–14 tokens/sec for MTP-assisted decode.

The server runs the Inferact NVFP4 checkpoint. Eider will evaluate other NVFP4 checkpoints when they become available.

This path only serves the complete text model. It includes native Qwen Sparse Attention (QSA), Gated DeltaNet (GDN), hyperconnections, all 512 MoE experts, and the 51B-parameter BF16 PLE n-gram table. Eider pages the selected n-gram rows directly from NVMe. The model has a native 262,144-token context. Eider does not serve the vision tower yet.

The pinned checkpoint contains about 170 GiB of safetensors. Fetch it before you use the offline launcher:

hf download Inferact/Qwen3.8-Flash-Next-NVFP4 \
  --revision 129972269565f7f4f664fdf8dd42268d3bbda9fd
scripts/run-eider-qwen38-flash-next.sh

In a second terminal, start Pi:

scripts/run-pi-eider-qwen38-flash-next.sh

Performance

These rounded results come from Eider telemetry during a live Pi tool-use session. The server used the native 262K context profile.

The prefill path batches QSA projections and uses BF16 tensor cores for hyperconnection projections. It processes up to 512 prompt tokens per prefill iteration to reuse routed expert weights.

The MTP prompt path batches its input and QSA projections. It appends only the index and attention cache state that future drafts need.

Eider serves one speculative draft by default from the checkpoint's QSA and MoE MTP block. Set EIDER_SPECULATIVE_DRAFTS=0 for target-only decode.

The MTP path uses an exact two-row target verifier with transactional partial acceptance. The fixed greedy probe matched every canonical target decision and preserved recurrent state bitwise.

The shared radix cache retains the target prefix and the MTP QSA prefix. Pi follow-up turns keep MTP speculation after a prefix-cache hit.

These values are server telemetry, not isolated kernel rates. The PLE n-gram table remains on NVMe, while the complete neural body stays resident in unified memory.

Send a Responses API request directly:

curl -fsS http://127.0.0.1:8080/v1/responses \
  -H 'Content-Type: application/json' \
  -d '{"model":"eider-qwen3.8-flash-next","input":"What is 2+2?","max_output_tokens":64}'

Qwen3.8 27B

Qwen3.8 27B is the primary dense model in Eider. The standard launcher uses the Unsloth Dynamic NVFP4 checkpoint and its z-lab/Qwen3.8-27B-DFlash2 companion.

The checkpoint uses NVFP4 for most MLP weights. It keeps attention, the LM head, and the final eight MLP blocks in FP8.

You need a DGX Spark, stable Rust, and CUDA 13.x. The CUDA installation must include nvcc and cuBLASLt.

scripts/run-eider-qwen38.sh

The target checkpoint uses about 24.6 GiB of disk space. The first start also resolves the DFlash2 checkpoint. Eider listens on 127.0.0.1:8080.

In a second terminal, start Pi:

scripts/run-pi-eider-qwen38.sh

The launcher enables two DFlash2 drafts and the native 262,144-token context window. It uses one active sequence to keep the full-attention cache within the Spark memory budget.

Current performance

These Qwen3.8 results come from a live Pi tool-use session on one DGX Spark. The session used a release build on August 22, 2026.

The three completed turns had 5.8K to 10.3K prompt tokens. The shared prefix cache restored state from each previous turn.

A separate correctness-gated 4K benchmark measures about 13 tokens/sec for the target alone. DFlash2 reaches about 26 effective tokens/sec on the same synthetic sequence.

These numbers are API results, not isolated kernel rates. Context length, sampling, tool grammar, prefix reuse, and draft acceptance change the result.

API request

Send a Responses API request:

curl -fsS http://127.0.0.1:8080/v1/responses \
  -H 'Content-Type: application/json' \
  -d '{"model":"eider-qwen3.8","input":"What is 2+2?","max_output_tokens":64}'

Use --offline after Eider caches the checkpoint and its derived artifacts:

scripts/run-eider-qwen38.sh --offline

Runtime details

The Qwen3.8 target has 64 dense layers. Its layer schedule repeats three linear-attention layers and one full-attention layer.

The DFlash2 companion proposes tokens for target verification. Eider commits only the proposal prefix that matches the target output.

The shared radix cache retains full-attention prompt pages. A separate cache retains exact DFlash2 state for the same prompt prefix.

Set --speculative-drafts 0 to measure target-only decoding:

scripts/run-eider-qwen38.sh --speculative-drafts 0

For local checkpoints, supply both paths:

eider-serve --model-dir ./models/qwen3.8-27b-nvfp4 \
  --dflash2-dir ./models/qwen3.8-27b-dflash2 \
  --speculative-drafts 2

Supported models

Catalogue IDs start the server. API clients use the served model name in the second column.

DeepSeek V4 Flash uses a local preparation path. The complete source checkpoint is larger than the Spark unified memory. Read the expert-storage notes for the storage layout.

The generic launcher starts Gemma 4 by default. Select another catalogue model with EIDER_MODEL:

scripts/run-eider
EIDER_MODEL=muse-glimmer-30b-nvfp4 scripts/run-eider

Other model performance

The table contains representative single-session API results on one GB10. These values are not peak kernel rates.

Use the telemetry from eider-serve for comparisons. It reports prompt cache hits, prefill rate, time to first token, decode rate, and speculative acceptance.

Tomatoes smoke test

scripts/tomatoes.sh sends one fixed Chat Completions request to Eider. The prompt asks for a detailed twenty-point guide to growing tomatoes.

The script needs curl, jq, and a running Eider server.

Start the server first. Then run the script in a second terminal:

scripts/tomatoes.sh

With no argument, the script reads /v1/models and uses the served model. An explicit argument requires the server to report the same model:

scripts/tomatoes.sh eider-qwen3.8
scripts/tomatoes.sh qwen3.8-27b

The script uses a 4,000-token output limit by default. If you need a shorter smoke test, set EIDER_TOMATO_MAX_TOKENS:

EIDER_TOMATO_MAX_TOKENS=256 scripts/tomatoes.sh

Use these environment variables for other server configurations:

The final report includes prompt tokens, cached tokens, prefill rate, decode rate, time to first token, and wall time. The answer goes to standard output. Progress and performance data go to standard error.

Serving

List the pinned catalogue without a model download:

cargo run --release -p eider-api --bin eider -- model list

Start a catalogue model directly:

eider-serve qwen3.8-flash-next --offline
eider-serve qwen3.8-27b
eider-serve qwen3.6-35b-a3b
eider-serve ornith-1.5-35b-a3b
eider-serve muse-glimmer-30b-nvfp4
eider-serve step-3.7-flash --offline

The direct Qwen3.8 command uses conservative server capacities. The dedicated launcher selects the 262K context profile for a single Pi session.

The first online start resolves an immutable Hugging Face revision. Eider puts derived artifacts below $XDG_CACHE_HOME/eider/models/.

Use --model-dir only for local development checkpoints. Catalogue IDs select pinned deployments.

Both API adapters use the same scheduler and model runtime:

• POST /v1/responses
• POST /v1/chat/completions
• GET /healthz
• GET /metrics

The server supports streaming, tool history, sampling, cancellation, concurrent requests, and prompt-prefix caching. Run eider-serve --help for the complete command reference.

Set EIDER_API_KEY to require bearer authentication.

Pi

The Pi launchers use pi/agent/models.json from this repository. They do not change the global Pi configuration.

scripts/run-pi-eider-qwen38-flash-next.sh
scripts/run-pi-eider-qwen38.sh
scripts/run-pi-eider-qwen.sh
scripts/run-pi-eider-ornith.sh
scripts/run-pi-eider-agents-a1.sh
scripts/run-pi-eider-stepfun.sh
scripts/run-pi-eider-laguna.sh
scripts/run-pi-eider-muse.sh
scripts/run-pi-eider-gemma4.sh
scripts/run-pi-eider-nemotron3-super.sh
scripts/run-pi-eider-deepseek4.sh

The Qwen3.8 Pi launcher uses medium reasoning by default. Set PI_EIDER_THINKING=xhigh for tasks that need more reasoning tokens.

The launchers use the Responses API by default. Set PI_EIDER_PROVIDER=eider-chat to use Chat Completions.

Codex

model = "eider-qwen3.8"
model_provider = "eider"

[model_providers.eider]
name = "Eider"
base_url = "http://127.0.0.1:8080/v1"
env_key = "EIDER_API_KEY"
wire_api = "responses"

Why Eider exists

The DGX Spark is not a data-centre GPU. Its SM12x Blackwell GPU does not use the B200 programming model.

Eider treats the 128 GB unified memory as one budget. Weights, active sequence state, retained prefixes, CUDA workspaces, and host processes share this budget.

The runtime uses bounded page pools and explicit cache budgets. Large MoE models can keep selected experts resident and page the remaining experts from disk.

Each model keeps its native sequence state. Nemotron Mamba state does not use a KV-cache abstraction. Flash Next keeps GDN, QSA, PLE, and hyperconnection state in one exact sequence snapshot. Qwen3.8 27B target and DFlash2 state advance together.

This narrow hardware target keeps performance decisions visible. It also makes the runtime useful as a place to study modern inference systems.

Development

The workspace has three main crates:

• nvfp4 contains CUDA kernels, cuBLASLt plans, device storage, checkpoint formats, and GPU benchmarks.
• infer contains model execution, sampling, scheduling, sequence caches, and model benchmarks.
• eider-api contains catalogue deployment, the inference actor, HTTP APIs, streaming, and telemetry.

Build and test the workspace:

cargo build --workspace
cargo build --release -p eider-api --bin eider-serve
cargo test --workspace
cargo clippy --workspace --all-targets -- -D warnings

Configure the repository-local CUTLASS tree for SM121:

scripts/setup-cutlass-sm12x.sh
source .deps/cutlass-sm12x.env
scripts/probe-cutlass-sm12x.sh

Focused benchmarks use micromeasure. Each benchmark has a correctness gate before it records timing data.

cargo bench -p nvfp4 --bench qwen36_routed_moe_decode
cargo bench -p nvfp4 --bench qwen38_grouped_moe_prefill
cargo bench -p infer --bench qwen36_prefill
cargo bench -p infer --bench step37_prefill
cargo bench -p infer --bench laguna_prefill

A kernel result is not sufficient evidence for a server improvement. Measure the complete model path before you keep an optimization.

Troubleshooting

If CUDA cannot target sm_121, use one CUDA 13.x installation for all CUDA components. Then run:

scripts/probe-cutlass-sm12x.sh

If an offline catalogue start fails, fetch the model once online:

cargo run --release -p eider-api --bin eider -- model fetch MODEL_ID

If a CUDA allocation fails, stop other GPU and memory-heavy processes. GPU allocations and host processes share the Spark unified memory.

Reduce the context, the prefill capacity, or the number of active sequences to reduce the memory footprint.

Authorship

The project began as a hand-written study of the GB10 GPU and its software stack. AI assistance now contributes to implementation and performance work, especially for boilerplate, CUDA wrappers, and benchmark infrastructure.

This boundary stays visible. The project treats measurable behaviour and correctness checks as more important than claims about how code was produced.

Further reading

• Model deployment
• Qwen3.6 batching and scheduling
• Step-3.7 expert paging
• DeepSeek V4 expert storage
• SM12x CUTLASS and NVFP4