Kimi-K3 DFlash2 draft

Draft model for speculative decoding on moonshotai/Kimi-K3, served with TokenSpeed (--speculative-algorithm DFLASH).

Architecture

5-layer, lookahead-8, MLA + SWA

• DFlash2DraftModel, 5 layers, hidden size 7168, bfloat16
• block_size: 8 (lookahead-8), attention_mode: mla
• target_layer_ids: [19, 37, 66, 78, 90] - five auxiliary hidden-state taps
• layer_types: 4x sliding_attention (window 4096) + 1x full_attention
• yarn RoPE, rope_theta: 50000.0, factor: 32.0
• 3.69 B parameters, 7.39 GB of weights

Benchmarks

Acceptance length acc_len = 1 + accepted/steps at 7 speculative proposals, served with TokenSpeed DFLASH on two GB300 nodes (8 GPUs, TP8), MLA target and drafter. Target is moonshotai/Kimi-K3 with the serving flags shown at the bottom of this card. Production sampling only: temperature=1.0, top_p=0.95, reasoning effort max, no top_k.

Ten-benchmark matrix

Long context

RULER multi-key needle retrieval over natural-text paragraphs, with only the input length varying. retrieval is exact match on the needle value, recorded so that acceptance cannot be credited to a model that has stopped reading its context.

Serving

python3 -m tokenspeed.cli serve <Kimi-K3> \
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path <this repo> \
  --speculative-num-steps 7 --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 8 \
  --speculative-draft-model-quantization unquant \
  --attention-backend mla --drafter-attention-backend mla \
  --sampling-backend flashinfer --trust-remote-code \
  --tensor-parallel-size 8 --moe-backend flashinfer_trtllm \
  --max-model-len 32768 --gpu-memory-utilization 0.90 \
  --max-num-seqs 4 --max-cudagraph-capture-size 4 \
  --cudagraph-capture-sizes 1 2 3 4 \
  --disable-prefill-graph --disable-kvstore

At startup the engine reports the auxiliary taps it captured; confirm they match (19, 37, 66, 78, 90).