Gemma 4 Abliteration

Abliterated (uncensored) versions of Google's Gemma 4 model family using norm-preserving biprojected abliteration and Expert-Granular Abliteration (EGA) for MoE models.

Models

Refusal rates measured across 686 prompts from 4 independent datasets (JailbreakBench, tulu-harmbench, NousResearch, mlabonne). Every flagged refusal was manually audited — most are refusal-then-comply false positives.

Method

Dense models (E2B, E4B, 31B): Norm-preserving biprojected abliteration. Per-layer refusal directions are computed from 800 harmful/harmless prompt residuals, orthogonalized against harmless means, and projected out of o_proj + mlp.down_proj weights while preserving row norms.

Unified model (12B): Same dense biprojection on the text decoder. gemma-4-12B-it is the encoder-free Gemma4Unified arch (added 2026-06-03); its refusal signal sits in the upper layers (L15-47), so only the top 70% are abliterated. Two arch quirks: it needs transformers >= 5.10.1, and it emits -inf logits for reserved vocab tokens that NaN a naive KL — the eval masks those. See version requirements.

MoE model (26B-A4B): Same as above on the dense pathway, plus Expert-Granular Abliteration (EGA) — hooks MoE routers to compute per-expert routing weights, then applies the same projection to each of the 128 expert down_proj slices per layer. Dense-only abliteration leaves 29/100 refusals; adding EGA drops it to 3/100.

Built on heretic. EGA concept from OBLITERATUS. Biprojection from grimjim.

Quick Start

# Install heretic
uv tool install 'heretic-llm @ git+https://github.com/p-e-w/heretic' --with protobuf

# Abliterate a dense model
HF_DATASETS_CACHE=/tmp/hf_datasets_cache \
  heretic-python scripts/abliterate.py biprojection \
  --model google/gemma-4-E4B-it \
  --top-pct 100 --strip-topic-markers --skip-prefix --batch-size 4 \
  --auto-save models/output-dir

# Abliterate the 12B Unified model (needs transformers >= 5.10.1; 70% layers)
HF_DATASETS_CACHE=/tmp/hf_datasets_cache \
  heretic-python scripts/abliterate.py biprojection \
  --model google/gemma-4-12B-it \
  --top-pct 70 --strip-topic-markers --skip-prefix --batch-size 4 \
  --auto-save models/output-dir

# Abliterate the MoE model (EGA)
HF_DATASETS_CACHE=/tmp/hf_datasets_cache \
  heretic-python scripts/ega.py \
  --model google/gemma-4-26B-A4B-it \
  --strip-topic-markers --skip-prefix --no-eval --batch-size 4 \
  --save models/output-dir

# Memory-efficient 31B (4-bit directions + bf16 shard-by-shard save)
heretic-python scripts/export_31b.py

Requirements

The original four models (E2B/E4B/26B-A4B/31B) were produced on transformers 5.5.0 and the pre-refactor llama.cpp converter. Only the 12B Unified model needs the newer toolchain above.

Scripts

Key Findings

1. Biprojection + EGA covers dense, MoE, and Unified — sub-1% refusal rate across all models
2. Larger models absorb abliteration better — E4B KL=0.07 vs E2B KL=0.35; 12B is cleanest at KL=0.056
3. MoE experts carry most refusal signal — dense-only gets 29/100, EGA gets 3/100
4. Default refusal markers are broken — "illegal", "harmful" etc. match disclaimers, not refusals
5. Task vector negation doesn't work — sharp phase transition, no usable middle ground
6. 4-bit directions ≈ bf16 directions — mean cosine similarity 0.935, fine for abliteration
7. The Unified arch needs care — refusal signal in upper layers only (abliterate 70%, not 100%); reserved-token -inf logits NaN a naive KL (mask them); needs transformers >= 5.10.1 and a recent llama.cpp for GGUF

Repo Structure

├── ABLITERATION.md    # Full research doc with tables and references
├── STATE.md           # Current best results per model
├── IDEAS.md           # Prioritized technique backlog
├── REDTEAM.md         # Red team benchmark survey
├── scripts/           # All experiment and export scripts
├── experiments/       # JSON results from every experiment
├── prompts/           # Eval datasets (150 iteration + 686 full)
├── models/            # Saved weights (gitignored)
└── checkpoints/       # Heretic Optuna checkpoints (gitignored)

License

Apache 2.0 (same as base Gemma 4 models).