README
Gemma 4 12B IT — OSCAR INT2 KV cache (GGUF)
Gemma-4-12B-it in GGUF, packaged for the OSCAR ~2-bit (INT2) KV cache fork of llama.cpp. Two independent axes are combined here:
- weights:
bf16(full precision) orQ4_K_M(~4.5-bit) — affects model size/quality, not the KV cache. - OSCAR rotation:
rot-kv= the per-layer calibrated rotation is baked into the GGUF, which is what makes the 2-bit KV cache keep near-f16 quality.base= plain model, no rotation.
The -rot-kv models need the OSCAR fork (-fa on + --cache-type-k/v q2_0 + the env vars below).
The base models are standard GGUFs that run on stock llama.cpp with a normal f16 KV cache.
Folders
| folder | weights | OSCAR rotation | size | what it's for |
|---|---|---|---|---|
q4km-rot-kv/ ⭐ | Q4_K_M | ✅ baked in | 6.9 GB | Recommended. Small weights + OSCAR INT2 KV — the practical deployment model. |
bf16-rot-kv/ | bf16 | ✅ baked in | 22 GB | Full-precision weights + OSCAR INT2 KV. Use to isolate the KV-cache effect (identical weights, compare f16-KV vs INT2-KV) or for max quality. |
base-q4km/ | Q4_K_M | ❌ none | 6.9 GB | Plain Gemma 4 12B, Q4_K_M weights. Runs on stock llama.cpp (f16 KV). |
base-bf16/ | bf16 | ❌ none | 22 GB | Plain Gemma 4 12B, full precision. Stock llama.cpp, or use as the base to bake your own rotation. |
rotation/ | — | the matrices | 2×18 MB | Raw OSCAR calibrated rotation (k_/v_rotation_*.pt, per-layer 512×512). Bake onto any Gemma 4 12B base GGUF. |
"OSCAR rotation" is an in-graph, post-RoPE orthogonal transform applied to Q/K (and V) so that the KV cache quantizes well at 2-bit. The
-rot-kvGGUFs already contain it asblk.{i}.attn_{k,v}_rot.weighttensors; the base weights are copied through unchanged.
Run (the -rot-kv models)
Needs the OSCAR fork built with Metal. Gemma 4 also needs its chat template.
LLAMA_KV_FUSED_FA=1 LLAMA_KV_NO_HADAMARD=1 LLAMA_KV_CLIP_RATIO=0.96 \
LLAMA_KV_HP_SINK=512 LLAMA_KV_HP_RECENT=2048 \
./build/bin/llama-server -m q4km-rot-kv/gemma-4-12b-it-rot-kv.gguf \
-fa on -ngl 99 -c 16384 \
--cache-type-k q2_0 --cache-type-v q2_0 \
--chat-template-file models/templates/google-gemma-4-31B-it.jinja \
--host 127.0.0.1 --port 8080
| env var | meaning | value |
|---|---|---|
LLAMA_KV_FUSED_FA | fused INT2+f16 flash-attention kernels (fast Metal path) | 1 |
LLAMA_KV_NO_HADAMARD | rotation is in-graph, so skip the in-quant Hadamard | 1 |
LLAMA_KV_CLIP_RATIO | per-row outlier clip before quant | 0.96 |
LLAMA_KV_HP_SINK / LLAMA_KV_HP_RECENT | tokens kept high-precision (first/last) | 512 / 2048 |
--cache-type-v f16 keeps V high-precision (a bit more quality, more memory). A base (non-rotated)
model with INT2 flags falls back to degraded data-free INT2 — use a -rot-kv model for INT2.
Bake your own rotation
python3 oscar-rotation/export_rot_kv_gguf.py \
--base base-bf16/gemma-4-12b-it-bf16.gguf \
--rot-dir rotation/ \
--out gemma-4-12b-it-bf16-rot-kv.gguf
Contributors
2 commits
Similar repos
README
Gemma 4 12B IT — OSCAR INT2 KV cache (GGUF)
Gemma-4-12B-it in GGUF, packaged for the OSCAR ~2-bit (INT2) KV cache fork of llama.cpp. Two independent axes are combined here:
- weights:
bf16(full precision) orQ4_K_M(~4.5-bit) — affects model size/quality, not the KV cache. - OSCAR rotation:
rot-kv= the per-layer calibrated rotation is baked into the GGUF, which is what makes the 2-bit KV cache keep near-f16 quality.base= plain model, no rotation.
The -rot-kv models need the OSCAR fork (-fa on + --cache-type-k/v q2_0 + the env vars below).
The base models are standard GGUFs that run on stock llama.cpp with a normal f16 KV cache.
Folders
| folder | weights | OSCAR rotation | size | what it's for |
|---|---|---|---|---|
q4km-rot-kv/ ⭐ | Q4_K_M | ✅ baked in | 6.9 GB | Recommended. Small weights + OSCAR INT2 KV — the practical deployment model. |
bf16-rot-kv/ | bf16 | ✅ baked in | 22 GB | Full-precision weights + OSCAR INT2 KV. Use to isolate the KV-cache effect (identical weights, compare f16-KV vs INT2-KV) or for max quality. |
base-q4km/ | Q4_K_M | ❌ none | 6.9 GB | Plain Gemma 4 12B, Q4_K_M weights. Runs on stock llama.cpp (f16 KV). |
base-bf16/ | bf16 | ❌ none | 22 GB | Plain Gemma 4 12B, full precision. Stock llama.cpp, or use as the base to bake your own rotation. |
rotation/ | — | the matrices | 2×18 MB | Raw OSCAR calibrated rotation (k_/v_rotation_*.pt, per-layer 512×512). Bake onto any Gemma 4 12B base GGUF. |
"OSCAR rotation" is an in-graph, post-RoPE orthogonal transform applied to Q/K (and V) so that the KV cache quantizes well at 2-bit. The
-rot-kvGGUFs already contain it asblk.{i}.attn_{k,v}_rot.weighttensors; the base weights are copied through unchanged.
Run (the -rot-kv models)
Needs the OSCAR fork built with Metal. Gemma 4 also needs its chat template.
LLAMA_KV_FUSED_FA=1 LLAMA_KV_NO_HADAMARD=1 LLAMA_KV_CLIP_RATIO=0.96 \
LLAMA_KV_HP_SINK=512 LLAMA_KV_HP_RECENT=2048 \
./build/bin/llama-server -m q4km-rot-kv/gemma-4-12b-it-rot-kv.gguf \
-fa on -ngl 99 -c 16384 \
--cache-type-k q2_0 --cache-type-v q2_0 \
--chat-template-file models/templates/google-gemma-4-31B-it.jinja \
--host 127.0.0.1 --port 8080
| env var | meaning | value |
|---|---|---|
LLAMA_KV_FUSED_FA | fused INT2+f16 flash-attention kernels (fast Metal path) | 1 |
LLAMA_KV_NO_HADAMARD | rotation is in-graph, so skip the in-quant Hadamard | 1 |
LLAMA_KV_CLIP_RATIO | per-row outlier clip before quant | 0.96 |
LLAMA_KV_HP_SINK / LLAMA_KV_HP_RECENT | tokens kept high-precision (first/last) | 512 / 2048 |
--cache-type-v f16 keeps V high-precision (a bit more quality, more memory). A base (non-rotated)
model with INT2 flags falls back to degraded data-free INT2 — use a -rot-kv model for INT2.
Bake your own rotation
python3 oscar-rotation/export_rot_kv_gguf.py \
--base base-bf16/gemma-4-12b-it-bf16.gguf \
--rot-dir rotation/ \
--out gemma-4-12b-it-bf16-rot-kv.gguf
Similar repos
Linked in READMEs
Contributors
2 commits