It’s just semantics: what are we measuring with evaluation-awareness steering vectors?

A project for BlueDot Impact's Technical AI Safety Project Sprint

Hua et al. (2026) developed an evaluation-aware model organism to use type hints in code during evaluation, but not deployment, contexts. They used contrastive prompts to extract an evaluation vector that they then added to the residual stream to suppress type hints and other evaluation behaviours. This project replicated the main finding, looked at the geometry of the internals, and tested a semantically similar control vector.

Main findings:

• Reproduced primary finding of evaluation suppression. Injecting a contrastive evaluation vector into the residual stream suppressed type hint usage and evaluation-aware CoT, while reversing the sign amplified both.

• Model internals shift toward deployment state when the steering vector is applied during evaluation. Residual stream activity during evaluation suppression steering resembled the deployment condition, suggesting that the effect is not purely surface-level.

• Evaluation suppression may not be specific to an explicit evaluation vector. A formal versus informal language vector partially replicated the effect and shared representational overlap with the evaluation vector in the residual stream. This suggests that either the evaluation vector is capturing something more general than evaluation-awareness, or that the model organism is broadly sensitive to residual stream perturbations.

Full write-up: jimmiegmaz.com/steering

Reference repo: tim-hua-01/steering-eval-awareness-public

---

Project structure

src/
  steer_core/
    nnsight_extract.py     extract_difference_vectors() — difference-of-means across all layers
    nnsight_steer.py       steer_and_generate() — batched steering with hidden state capture
    format_simple.py       process_contrastive_prompts_simple() — tokenise contrastive pairs
  grading/
    hint_grader.py         AST-based Python type hint detection

scripts/
  01_extract_steering_vector.py   extract difference vectors → results/steering_vectors.pt
  02_generate_with_hidden_states.py  generate responses + capture hidden states (all conditions)
  03_behaviour_plots.py           hint-rate + eval-aware CoT bar charts (Figs 1, 2, 4, 5)
  04_internals_analysis.py        PCA cumvar, cross-window variance/angles, PC cosine similarity
  05_geometry.py                  dPCA geometry — eval vector ±0.6 (Fig 3) and control vectors at 0.3 (Fig 6)

data/
  contrastive_prompts/
    eval_vs_deploy.json            16 contrastive pairs from paper Table 12
    control_formal_informal.json   16 contrastive pairs for formal/informal control vector
  system_prompts.json              D1-D5, E1-E5, W1-W5 from paper Tables 5-7
  user_prompts.json                UD1-6, UN1-3, UE1-5 from paper Tables 8-10
  coding_prompts.csv               64 Python coding questions

---

Quickstart

Requires a cloud GPU. Needs ~100 GB VRAM.

# On the GPU pod
git clone https://github.com/jgmaz/steering.git && cd steering
pip install uv && uv sync
echo "HF_TOKEN=hf_..." > .env

# Download model weights (requires HuggingFace token with access to Nemotron 49B)
huggingface-cli download nvidia/Llama-3.3-Nemotron-Super-49B-v1 \
  --local-dir /workspace/models/nemotron-49b
huggingface-cli download timhua/wood_v2_sftr4_filt \
  --local-dir /workspace/models/adapter

# Patch the model source file (once per pod)
# Nemotron's bundled modeling_decilm.py imports NEED_SETUP_CACHE_CLASSES_MAPPING,
# which was removed in newer transformers. This patches it out before loading.
python3 -c "
p='/workspace/models/nemotron-49b/modeling_decilm.py'
c=open(p).read()
c=c.replace(
    'from transformers.generation.utils import NEED_SETUP_CACHE_CLASSES_MAPPING, GenerationMixin, GenerateOutput',
    'from transformers.generation.utils import GenerationMixin, GenerateOutput\nNEED_SETUP_CACHE_CLASSES_MAPPING = {}'
)
open(p,'w').write(c)
print('Patched OK')
"

# 1. Extract steering vectors
# Evaluation steering vector (evaluation vs. deployment prompts)
PYTHONPATH=. uv run python scripts/01_extract_steering_vector.py \
  --base_model /workspace/models/nemotron-49b \
  --lora_adapter /workspace/models/adapter \
  --tokenizer nvidia/Llama-3.3-Nemotron-Super-49B-v1 \
  --no_quantize

# Control steering vector (formal vs. informal language)
PYTHONPATH=. uv run python scripts/01_extract_steering_vector.py \
  --base_model /workspace/models/nemotron-49b \
  --lora_adapter /workspace/models/adapter \
  --tokenizer nvidia/Llama-3.3-Nemotron-Super-49B-v1 \
  --contrastive_file data/contrastive_prompts/control_formal_informal.json \
  --output results/control_vector_formal.pt \
  --no_quantize

# 2. Generate responses + hidden states
# Unsteered and evaluation-steered conditions
PYTHONPATH=. uv run python scripts/02_generate_with_hidden_states.py \
  --base_model /workspace/models/nemotron-49b \
  --lora_adapter /workspace/models/adapter \
  --tokenizer nvidia/Llama-3.3-Nemotron-Super-49B-v1 \
  --condition all --strength 0.6 0.3 --n_prompts 8 --no_quantize

# Control-steered conditions
PYTHONPATH=. uv run python scripts/02_generate_with_hidden_states.py \
  --base_model /workspace/models/nemotron-49b \
  --lora_adapter /workspace/models/adapter \
  --tokenizer nvidia/Llama-3.3-Nemotron-Super-49B-v1 \
  --vectors_path results/control_vector_formal.pt \
  --condition control-steered-eval control-steered-eval-reversed \
              control-steered-deploy control-steered-deploy-reversed \
  --strength 0.6 0.3 --n_prompts 8 --no_quantize

# 3. Behaviour bar charts (hint rate + eval-aware CoT, Figures 1, 2, 4, 5)
PYTHONPATH=. uv run python scripts/03_behaviour_plots.py

# 4. Hidden-state geometry analyses (PCA, cross-window variance/angles, PC cosine similarity)
PYTHONPATH=. uv run python scripts/04_internals_analysis.py

# 5. dPCA geometry scatter plots (Figures 3, 6)
PYTHONPATH=. uv run python scripts/05_geometry.py