Kimi K3 DSpark speculator

Overview

A long-context DSpark speculator for Kimi K3. It supports context lengths of up to 1 million tokens.

A DSpark speculator for the Kimi K3 target, enabling faster inference through speculative decoding. DSpark extends the DFlash parallel-draft backbone with a Markov logit-bias head and a per-position confidence head. This checkpoint was trained with SpecForge using hidden states from a live SGLang target engine.

Model Specifications

• Base model: moonshotai/Kimi-K3
• Format: Safetensors (single-file BF16, 2,249,289,601 parameters)
• Draft: 5 full-attention Qwen3-style GQA layers, hidden size 7168, 64 query heads / 16 KV heads, and block_size=7
• Verification width: 1 current token + 7 draft tokens
• Auxiliary target layers: [7, 23, 51, 67, 83]
• Trained context: 65,536 tokens
• Target weights: embedding and unembedding weights are not included

Evaluation Results

acc_len is SGLang's histogram-native request acceptance length, averaged within each question and then equally across questions.

RULER V2 uses the 1M input configuration. Actual prompts span 1,000,432–1,047,925 tokens; partition acc_len is 4.4658 for MK, 4.3081 for MV, and 3.9919 for QA.

AIME26 acc_len by output length

Serving with SGLang

SGLang Cookbook provides Kimi K3 deployment recipes.

sglang serve \
  --trust-remote-code \
  --model-path moonshotai/Kimi-K3 \
  --tp-size 8 \
  --dcp-size 8 \
  --mem-fraction-static 0.85 \
  --max-mamba-cache-size 160 \
  --max-running-requests 32 \
  --cuda-graph-max-bs-decode 32 \
  --reasoning-parser kimi_k3 \
  --tool-call-parser kimi_k3 \
  --host 0.0.0.0 \
  --port 30000 \
  --speculative-algorithm DSPARK \
  --speculative-draft-model-path RadixArk/Kimi-K3-DSpark \
  --speculative-dspark-block-size 7 \
  --speculative-draft-attention-backend trtllm_mha \
  --enable-linear-replayssm-spec \
  --context-length 1048576 \
  --chunked-prefill-size 16384

YaRN-16 is enabled in the published draft config by default with original_max_position_embeddings=65536 and max_position_embeddings=1048576; no separate draft config override is required.

Training Details

• Framework: SpecForge online distillation, with hidden states captured from a frozen Kimi K3 target served by a live SGLang engine. Draft trained from random initialization.
• Loss: 0.1 CE + 0.9 L1 distillation + 1.0 confidence BCE, decay gamma 4.0, with 512 sampled anchors per sequence and block_size=7.
• Topology: 4 nodes × 4 GB300 (16 ranks) — 2 × TP8 target replicas, DP2 sampler, FSDP16 SHARD_GRAD_OP on the draft, TP-batch scatter. Batch 8 per replica × 32 accumulation steps × 2 replicas = global batch 512.