DeepSeek-V4 tiny corrected-native-primitives CPU text fixture

Complete randomly initialized, untrained QFSDeepseekV4ForCausalLM text class using Transformers5.16.1 native primitives and a reviewed RMSNorm arithmetic correction. No upstream weights, paid GPU/cloud compute or useful-model claim. This is not unmodified native Transformers or the complete production release.

Architecture and scope

Text lineage: deepseek-ai/DeepSeek-V4-Flash-0731@7872f01b1d1fe23eabc4c98b48bffcef5a386062. Vision coverage research: deepseek-ai/DeepSeek-V4-Flash-Vision-Exp@6821d6ad3681a4b137b066b76094fa82ebd0a380.

Complete native-primitives text class with explicit reviewed per-instance RMSNorm arithmetic correction: weighted normalization in FP32 followed by final cast to input dtype. This is not bit-identical native Transformers execution or the complete upstream release. No vision encoder/aligner, DSpark, MTP, original FP8/FP4 storage execution, full-size numerical alignment, or million-token claim. HCA rate is intentionally 16 rather than production 128; CSA rate remains 4 and sliding window is 8 rather than 128. Random weights are not a quality benchmark. The unmodified native RMSNorm multiplies a BF16 normalized activation by its required FP32 weight, promoting output to FP32 and causing the next BF16 linear projection to fail. The wrapper computes weighted normalization in FP32 and casts the complete result to the input activation dtype. Only this model instance's RMSNorm modules are replaced; no global/installed-library patches, autocast, forward-hook corrections or weight changes. This moves an arithmetic rounding boundary and is explicitly not native bit-equivalence. deepseek_v4_qfs.py and its Apache-2.0 license are retained in the immutable model pin. Observed replacements and source hashes: {"change": "RMSNorm weighted normalization in FP32 with final output cast to input dtype; unchanged tensor names/values/dtypes and all other native primitives.", "code_file": "deepseek_v4_qfs.py", "dtype_boundary_audit": "Native HyperConnection/HyperHead explicitly cast final stream outputs; attention sink promotes score math then casts to value dtype; compressor position bias promotes gates then softmax casts to KV dtype; router correction bias affects selection only. Only weighted RMSNorm lacks final activation dtype restoration.", "license": "Apache-2.0", "replaced_modules": ["model.layers.0.self_attn.q_a_norm", "model.layers.0.self_attn.kv_norm", "model.layers.0.input_layernorm", "model.layers.0.post_attention_layernorm", "model.layers.1.self_attn.q_a_norm", "model.layers.1.self_attn.kv_norm", "model.layers.1.input_layernorm", "model.layers.1.post_attention_layernorm", "model.layers.2.self_attn.q_a_norm", "model.layers.2.self_attn.kv_norm", "model.layers.2.self_attn.compressor.kv_norm", "model.layers.2.self_attn.compressor.indexer.kv_norm", "model.layers.2.input_layernorm", "model.layers.2.post_attention_layernorm", "model.layers.3.self_attn.q_a_norm", "model.layers.3.self_attn.kv_norm", "model.layers.3.self_attn.compressor.kv_norm", "model.layers.3.input_layernorm", "model.layers.3.post_attention_layernorm", "model.norm"], "sha256": "82c9af18f8a68dac3a6fee763a55ee6c9a33a49d6f9dfdc196155a34d63d862e", "upstream_failure": "Native retained FP32 norm weights promote BF16 activations to FP32 before BF16 linear projection."}.

Four layers use explicit native legacy schedule selectors compress_ratios=[0,0,4,128], normalized to two sliding-only, one CSA, one HCA layer. Actual rates are CSA4/HCA16. All layers retain a local window8, shared K=V multi-query attention, learned sinks, low-rank query projection and grouped low-rank output (Q4/KV1, head16, Q-rank16, two output groups of rank8). Hidden64, expert intermediate32, eight routed experts, top2 plus one shared expert; three seeded frozen hash routers then one learned router use sqrt-softplus scores. mHC4 and20 Sinkhorn iterations are preserved. Native gate/up clipping limit10 is preserved. Partial interleaved RoPE uses dimension4, with separate main/compressor bases. No production YaRN/long-context scaling claim. The small real tokenizer is256 byte-level BPE tokens without merges plus16 special/reserved entries (272 total; IDs0/1/2/3 pad/BOS/EOS/unknown). The untied head is complete.

The upstream flat vision_* fields describe a32-layer vision encoder plus aligner, not instantiated by this native text class. Upstream dspark_* fields and MTP are not executed. A declared DeepseekV4ForCausalLM architecture in Vision-Exp metadata does not establish native multimodal coverage. The upstream configs/licenses are retained.

Actual measured evidence

• Parameters: 339,037; semantic tensors: 112; seed: 20260907.
• Two independent generator subprocesses matched complete byte inventories.
• Native FP32 initialization is converted by the actual BF16 loader; mandatory native FP32 retention is preserved before rounding. Exact values and dtypes pass corrected-class final reload and strict pinned QFS loading. Retained tensors: ["model.layers.0.self_attn.sinks", "model.layers.0.self_attn.q_a_norm.weight", "model.layers.0.self_attn.kv_norm.weight", "model.layers.0.input_layernorm.weight", "model.layers.0.post_attention_layernorm.weight", "model.layers.0.attn_hc.fn", "model.layers.0.attn_hc.base", "model.layers.0.attn_hc.scale", "model.layers.0.ffn_hc.fn", "model.layers.0.ffn_hc.base", "model.layers.0.ffn_hc.scale", "model.layers.1.self_attn.sinks", "model.layers.1.self_attn.q_a_norm.weight", "model.layers.1.self_attn.kv_norm.weight", "model.layers.1.input_layernorm.weight", "model.layers.1.post_attention_layernorm.weight", "model.layers.1.attn_hc.fn", "model.layers.1.attn_hc.base", "model.layers.1.attn_hc.scale", "model.layers.1.ffn_hc.fn", "model.layers.1.ffn_hc.base", "model.layers.1.ffn_hc.scale", "model.layers.2.self_attn.sinks", "model.layers.2.self_attn.q_a_norm.weight", "model.layers.2.self_attn.kv_norm.weight", "model.layers.2.self_attn.compressor.position_bias", "model.layers.2.self_attn.compressor.kv_norm.weight", "model.layers.2.self_attn.compressor.indexer.position_bias", "model.layers.2.self_attn.compressor.indexer.kv_norm.weight", "model.layers.2.input_layernorm.weight", "model.layers.2.post_attention_layernorm.weight", "model.layers.2.attn_hc.fn", "model.layers.2.attn_hc.base", "model.layers.2.attn_hc.scale", "model.layers.2.ffn_hc.fn", "model.layers.2.ffn_hc.base", "model.layers.2.ffn_hc.scale", "model.layers.3.self_attn.sinks", "model.layers.3.self_attn.q_a_norm.weight", "model.layers.3.self_attn.kv_norm.weight", "model.layers.3.self_attn.compressor.position_bias", "model.layers.3.self_attn.compressor.kv_norm.weight", "model.layers.3.mlp.gate.e_score_correction_bias", "model.layers.3.input_layernorm.weight", "model.layers.3.post_attention_layernorm.weight", "model.layers.3.attn_hc.fn", "model.layers.3.attn_hc.base", "model.layers.3.attn_hc.scale", "model.layers.3.ffn_hc.fn", "model.layers.3.ffn_hc.base", "model.layers.3.ffn_hc.scale", "model.norm.weight", "model.hc_head.hc_fn", "model.hc_head.hc_base", "model.hc_head.hc_scale"].
• Native cache probes exercise CSA boundaries4, sliding eviction8, HCA16, and63/64; single- and multi-token continuation runs have exact same-schedule repeats. Cached-vs-cold differences are recorded, not asserted zero. Real cache remainder, entry count and CSA overlap states are checked. Actual hooks observe mHC row/column sums, hash/learned routes, grouped output, and CSA future-entry exclusion. Suffix perturbation checks causal prefix independence; beyond-limit activation inputs exercise the real clipped expert function.
• Two fresh CPU processes, two threads, BF16, honest QFS lane other, sealed four64-token synthetic windows,252 shifted prediction positions; no quality benchmark/registry submission.
• Strict tensor verification and forced numerical replay: metric 0.0, KL {"max": 0.0, "mean": 0.0, "median": 0.0, "p95": 0.0, "p99": 0.0, "p99_9": 0.0}, top1 agreement 1.0. The actual tokenwise artifact is252 positive float64 zeros, not just a summary claim.
• Captured immutable private-stage checkpoint: malaiwah/deepseek-v4-tiny-random-bf16@4f9dcfcbabdbe336f5f83e836dd8fac7138dda23. QFS source revision c8c354e54217c47fc7d04c4f0308baa5da5553d1; executed file hashes accompany the evidence. Code repository is the same model repository at the same immutable pin, explicitly selected with --trust-remote-code --code-repository --code-revision. Capture weights/tokenizer remain local; HF network is permitted for pinned code verification. Captures are not entirely offline.

See native-verification.json, evidence-summary.json, build-manifest.json, support.json, and the evidence repository. The evidence repository root is an artifact bundle, not a single root-format QFS dataset; first/ and repeat/ are sealed roots. QFS retains its hash-proof reproduction-confirmation semantics while --force-compute also runs numerical replay. This proves same-stack/machine repeatability only.

Reproduction and capture

Python3.12, isolated CPU environment: pip install -r requirements-cpu.txt. With the recorded QFS checkout at $QFS, outputs must be new:

python build_fixture.py --out /tmp/deepseek-v4-rebuilt
python verify_native.py --model-dir /tmp/deepseek-v4-rebuilt --code-repository malaiwah/deepseek-v4-tiny-random-bf16 --code-revision 4f9dcfcbabdbe336f5f83e836dd8fac7138dda23 --qfs-root "$QFS" --out /tmp/deepseek-v4-native.json
python build_panel.py --model-dir /tmp/deepseek-v4-rebuilt --model-repository malaiwah/deepseek-v4-tiny-random-bf16 --model-revision 4f9dcfcbabdbe336f5f83e836dd8fac7138dda23 --qfs-root "$QFS" --out /tmp/deepseek-v4-panel
python verify_cpu.py --model-dir /tmp/deepseek-v4-rebuilt --model-repository malaiwah/deepseek-v4-tiny-random-bf16 --model-revision 4f9dcfcbabdbe336f5f83e836dd8fac7138dda23 --code-repository malaiwah/deepseek-v4-tiny-random-bf16 --code-revision 4f9dcfcbabdbe336f5f83e836dd8fac7138dda23 --panel-dir /tmp/deepseek-v4-panel --qfs-root "$QFS" --out /tmp/deepseek-v4-repro

support.json gives the real QFS capture argv for third-party authors with an explicit --author, including the corrected native-primitives BF16 text checkpoint path. Original releases declare dynamic FP8 e4m3/ue8m0 block128 storage plus expert_dtype=fp4. This BF16 fixture does not prove those kernels/conversions or the complete production model. --drop-parallel-plan addresses the HF quantizer parallel-plan rewrite bug, not unsupported packed expert formats. Never remove quantization_config from real upstream weights to make them load. Do not bypass missing/unexpected/mismatched/conversion-error guards. Full original-format qualification needs actual supported hardware/runtime and clean full-checkpoint loading; vision and DSpark additionally need their real implementations and modality/speculation probes. For an unchanged original config without auto_map, explicit --code-repository malaiwah/deepseek-v4-tiny-random-bf16 and --code-revision 4f9dcfcbabdbe336f5f83e836dd8fac7138dda23 use only that pinned repository's verified auto_map as runtime class mapping. Wrapper config construction still uses all original checkpoint fields; no dimensions, quantization metadata or weights are replaced. This makes the runtime override explicit; it does not qualify untested original packed formats or ignored vision/DSpark modules.

Publisher workflow: release.py --stage-only --out NEW_DIR --qfs-root "$QFS" requires authorization for the two fixed repository IDs, refuses collisions, stages privately and captures an immutable pin. Only explicit release.py --publish --out SAME_DIR releases an unchanged successful stage and verifies every anonymous download. It cannot overwrite a published repository.

Licensing

Independent fixture weights, authored synthetic text, and workflow: MIT (LICENSE). Upstream DeepSeek text and vision repository licenses are MIT and retained as lineage-LICENSE / vision-lineage-LICENSE with pinned configs. Native execution uses Transformers5.16.1 Apache-2.0 source as a dependency. The reviewed derived runtime deepseek_v4_qfs.py remains Apache-2.0 (LICENSE-APACHE-2.0), not MIT; its prominent modification notice and installed source SHA256 values are retained.