Vision Tower Bench

Adding vision to a text-only LLM means picking a Tower, a layer, and usually a Projector, which full-model VLM benchmarks never isolate. This bench probes eleven frozen Towers. The published report covers six of them: 15 of 16 paired intervals across two geometry tasks and two probe capacities went against the idea that the Projector tosses away spatial information.

Report · Towers · Paper (coming soon)

What it measures

Four capability axes:

Roster

The multimodal Towers are standalone extracts of their parents. The model cards record counts and tolerances.

Results

Source files live in results/:

Reproducing the figures

uv sync --group figures
uv run --group figures python scripts/figures.py

Evaluating a new Tower

1. Wrap it. Return patch tokens at eight even relative depths, plus merged and projected if you have them. Include your own preprocessing.

2. Meet three constraints. Patch tokens only, no class token. Raster order, so undo any merge-block or window grouping before caching. Square token grid, the pooling and dense decoders assume one.

3. Prove it matches. A Tower pulled from a larger checkpoint must match that checkpoint within BF16 tolerance on fixed images.

4. Extract once. Cache features per model, Stage, depth and resolution. Probes read the cache after that.

5. Run the matrices. Same learning-rate grid, three seeds, validation picks the rate.

Layout

vtb/          # adapters, feature cache, probes, geometry and correspondence scoring
scripts/      # export, extraction matrices, bootstraps, figures
tests/        # parity, preprocessing, probe and merge-layout tests
results/      # published measurements
hf/           # model cards and configs for the published towers
CONTEXT.md    # project vocabulary

Tests

uv run pytest -q                      # full suite, downloads Tower weights
VTB_SKIP_WEIGHTS=1 uv run pytest -q   # 73 tests, no downloads

License

MIT covers code and results here, not weights. Published Towers keep parent terms. Apache-2.0 for Qwen3.8, Muse Glimmer, and Gemma 4. MIT for GLM-5.3 Flash and DeepSeek-ViT. Kimi License for both Kimi Towers. NVIDIA Open Model Agreement for C-RADIOv4-H. MiniMax Community License for MiniMax M3. Each release ships its license.