README
DeepSeek-ViT
This repository packages DeepSeek-ViT and the two-layer MLP projector from DeepSeek-V4.1-Flash.
Contents
| File | Tensors | What it holds |
|---|---|---|
model.safetensors | 259 | DeepSeek-ViT Tower, extracted from source shard 1 |
projector.safetensors | 4 | Two-layer MLP Aligner |
config.json | Vision-only DeepSeekV41ViT | |
vision.py | Standalone ViT and Aligner used by examples/inference.py |
This repository does not include Transformers modeling files or a preprocessor
config. AutoModel.from_pretrained will not work.
Architecture
| Component | Details |
|---|---|
| Tower | 32 layers, 1024 hidden, 16 heads, 2816 intermediate, patch size 14, 2D-RoPE |
| Token compression | 3x3 unfold / pixel-unshuffle with padding |
| Aligner | Linear(9216, 5120), GELU, Linear(5120, 5120), both with bias |
Usage
See examples/inference.py for image feature extraction.
AutoModel.from_pretrained will not load this repository. The example uses
vision.py in this repo.
Validation
The parity script
compares all 259 Tower tensors and 4 Aligner tensors with the pinned parent checkpoint
using torch.equal.
Reproduction
The export script
reads vision.* and aligner.* from shard 1 of deepseek-ai/DeepSeek-V4.1-Flash,
removes the prefixes, and writes the original BF16 tensors.
Credits
DeepSeek released the DeepSeek-V4.1-Flash weights. DeepSeek-ViT is described in DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression.
License
MIT License, the same license as the source model.
Contributors
2 commits
Similar repos
Linked in READMEs
README
DeepSeek-ViT
This repository packages DeepSeek-ViT and the two-layer MLP projector from DeepSeek-V4.1-Flash.
Contents
| File | Tensors | What it holds |
|---|---|---|
model.safetensors | 259 | DeepSeek-ViT Tower, extracted from source shard 1 |
projector.safetensors | 4 | Two-layer MLP Aligner |
config.json | Vision-only DeepSeekV41ViT | |
vision.py | Standalone ViT and Aligner used by examples/inference.py |
This repository does not include Transformers modeling files or a preprocessor
config. AutoModel.from_pretrained will not work.
Architecture
| Component | Details |
|---|---|
| Tower | 32 layers, 1024 hidden, 16 heads, 2816 intermediate, patch size 14, 2D-RoPE |
| Token compression | 3x3 unfold / pixel-unshuffle with padding |
| Aligner | Linear(9216, 5120), GELU, Linear(5120, 5120), both with bias |
Usage
See examples/inference.py for image feature extraction.
AutoModel.from_pretrained will not load this repository. The example uses
vision.py in this repo.
Validation
The parity script
compares all 259 Tower tensors and 4 Aligner tensors with the pinned parent checkpoint
using torch.equal.
Reproduction
The export script
reads vision.* and aligner.* from shard 1 of deepseek-ai/DeepSeek-V4.1-Flash,
removes the prefixes, and writes the original BF16 tensors.
Credits
DeepSeek released the DeepSeek-V4.1-Flash weights. DeepSeek-ViT is described in DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression.
License
MIT License, the same license as the source model.
Similar repos
Linked in READMEs
Contributors
2 commits