DeepSeek V4 Flash 0731 Spark Vision

DeepSeek-V4-Flash-0731 with working image input on one DGX Spark. The backbone is REAP-pruned to 216 of 256 routed experts and quantized with ExLlamaV3, which brings a 167 GB model down to 93 GiB resident. A DeepEncoderV2 tower and projector splice image embeddings in at token 129279.

The point of this repo is that the combination runs. Vision on the pruned, quantized, single-Spark build failed at startup before the patch described below. The earlier experimental release reported that it struggled with photographs; this build scores 40.8% content-word overlap against human COCO captions. Those two numbers come from different test setups, so treat the comparison as directional.

What was fixed

Serving died during startup with:

RuntimeError: swa_k_cache page stride 37376 is smaller than DSV4 page width 37440

The number never moves no matter what you set. KV_CACHE_DTYPE, MAX_MODEL_LEN, GPU_MEMORY_UTILIZATION and VLLM_DSV4_PADDED_NVFP4 all leave it at 37376, because none of them apply to the tensor actually being rejected.

During memory profiling vLLM allocates a one-page dummy KV cache before the packed pool exists. That dummy is contiguous at the raw record size, 64 blocks x 584 bytes = 37376. The b12x kernel reads pages rounded up to the 576-byte MLA boundary, so it wants ceil(37376/576) x 576 = 37440, sees a shorter stride, and aborts. The text-only build never trips this because its smallest profiling pass is already a four-page packed view with a stride of 1,078,464.

The fix, in vllm/models/deepseek_v4/nvidia/b12x.py, accepts the profiling tensor instead of aborting on it:

# A profiling/dummy cache is allocated contiguously at the unpadded record size
# before the packed pool exists. It is never read for real data, so flatten it to
# the [pages, page_bytes] contract; real packed views are strided and always
# exceed page_nbytes.
if byte_cache.ndim == 3 and byte_cache.is_contiguous() and page_stride_nbytes < page_nbytes:
    flat = byte_cache.reshape(int(byte_cache.shape[0]), -1)
    if int(flat.shape[1]) < page_nbytes:
        pad = torch.zeros((int(flat.shape[0]), page_nbytes - int(flat.shape[1])),
                          dtype=flat.dtype, device=flat.device)
        flat = torch.cat([flat, pad], dim=1)
    return flat

Real packed views are strided and never reach that branch, so live inference is untouched.

Do not "fix" this by setting VLLM_DSV4_PADDED_NVFP4=0. That silences the check by changing the cache record format, the server starts cleanly, and every reply is fluent nonsense. Ask the model a text-only question before you trust an image answer.

Two other traps in this stack. The entrypoint sets KV_CACHE_DTYPE and VLLM_DSV4_PADDED_NVFP4 with a bare export, not ${VAR:-default}, so anything you put in compose for those two is silently overwritten. And the server binds port 8000 regardless of VLLM_PORT.

Running it

environment:
  MODEL_PATH: /models/tp1-vision
  MAX_MODEL_LEN: "32768"
  MAX_NUM_SEQS: "1"
  GPU_MEMORY_UTILIZATION: "0.92"
  KV_CACHE_DTYPE: nvfp4_ds_mla
  VLLM_DSV4_PADDED_NVFP4: "1"
  DSV4_VISION_TOWER: /vision/tower/deepencoder_v2_tower.safetensors
  DSV4_VISION_ADAPTER: /vision/adapter/merged-004800-5af0c5.pt
  DSV4_VISION_TILES: "2"

93.01 GiB of weights, 55,614 tokens of KV cache, one sequence at a time. Text-only serving of the same backbone reaches 262,144 context and 265,066 tokens of KV; vision costs you that headroom because the tower and projector share the 128 GB pool.

import base64, json, urllib.request

b64 = base64.b64encode(open("photo.jpg", "rb").read()).decode()
body = json.dumps({
    "model": "deepseek-v4-flash-vision",
    "max_tokens": 60,
    "messages": [{"role": "user", "content": [
        {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64," + b64}},
        {"type": "text", "text": "Describe this image in one short sentence."}]}],
}).encode()
r = json.load(urllib.request.urlopen(urllib.request.Request(
    "http://localhost:8000/v1/chat/completions", body,
    {"Content-Type": "application/json"}), timeout=280))
print(r["choices"][0]["message"]["content"])

Measured

20 images from COCO val2017, greedy decoding, scored on content-word overlap against the best of five human reference captions:

Real output, unedited:

laptop on a desk        A laptop computer sits on a desk with a mouse and keyboard.   100%
kitchen                 A kitchen with a white refrigerator and a wooden cabinet.      75%
phone case              A woman with a Hello Kitty phone case.                         71%
tennis court            A group of people are standing on a tennis court.              67%
skier                   A man in a red jacket is skiing down a snowy hill.             60%
cat on laptop           A black cat is sitting on a laptop computer.                   60%
street sign             A red and white sign that says "DOLS" is on a pole.            33%
two runners             A man in a green shirt is running with a white shirt man.       0%

That is the honest spread. It gets the scene and the main objects, and it reads sign shape, colour and placement while garbling the letters. Fine-grained OCR is not there.

Limitations

The projector is a step-4800 development checkpoint trained against the unpruned 0731 model, not this pruned build. Vision quality is capped by that mismatch rather than by the tower.

Context drops from 262,144 text-only to 32,768 with vision on a single Spark. One sequence at a time; concurrency past that is untested.

Counting, spatial relations and small text are unreliable. Two of twenty images scored zero. Do not put image output anywhere a wrong answer costs something.

Credits

The language model is 0xSero/deepseek-v4-flash-0731-spark, REAP-pruned and EXL3-quantized from deepseek-ai/DeepSeek-V4-Flash-0731. The DeepEncoderV2 tower and projector are third-party vision weights for the 0731 family and were not trained here. This repo contributes the serving fix and the measurements.