Unexploitable Search

Codebase for experiments on unexploitable search. This repo studies a new generative adversarial RL method which works without demonstrations. The goal is to enforce diversity at a semantic level.

The core method is a two-player game (GAP — Game-based Adversarial Properties): a proposer (Alice) solves the task while an adversary (Charlie) identifies non-essential properties of Alice's solutions and proposes alternatives without them. Alice is rewarded for solving the task in ways the adversary cannot pin down to a single exploitable pattern — pushing the policy toward a semantically diverse set of solutions. The game needs no human demonstrations of "diverse" behaviour: the adversary generates the contrast set on the fly.

---

Setup

bash setup.sh          # installs uv, runs uv sync

Requires:

• Python 3.13+
• podman-hpc (or Docker) for sandboxed code execution (APPS domain)
• GPU with CUDA 12.8 for training
• W&B account (offline mode is the default)

Environment variables:

• PROJECT_SHARED — root for multirun/output dirs (defaults to .)
• WANDB_MODE — offline (default) or online
• PYTHON_TMPDIR — temp directory for model downloads

---

Repository Structure

unexploitable-search/
├── conf/                       # Hydra configs (YAML)
│   ├── _shared/                #   Shared config fragments
│   │   ├── adapter/lora.yaml   #     LoRA: r=32, alpha=16
│   │   ├── it_model/           #     Base models (qwen3_8b, llama3_8b, gemma3_12b, …)
│   │   ├── model_organism/     #     Starting-model loaders (prompted, trained, trained_hf)
│   │   └── setting/            #     Task domains (apps, alpaca, toy)
│   ├── mitigation_strats/      #   RL training config
│   │   ├── config.yaml         #     Main config
│   │   └── strat/              #     Strategy selection (none, game, hard_game,
│   │       │                   #       entropy_bonus, kl_from_base_penalty, length_penalty)
│   │       └── trainer_config/ #     GRPO trainer hyperparams
│   └── evals/                  #   Evaluation configs (benign task-correctness)
│
├── scripts/                    # User-facing entry points
│   ├── train/
│   │   ├── train_with_mitigation_strat.py
│   │   └── gpu_queue_runner.py     # Multi-GPU job queue
│   ├── eval/
│   │   ├── run_benign_eval.py        # Task-correctness eval
│   │   ├── run_toy_baseline.py     # Toy: baseline characterisation
│   │   ├── run_toy_diversity_eval.py # Toy: diversity eval on checkpoints
│   │   └── generate_bon_reference.py # BoN reference generation (toy)
│   ├── experiments/
│   │   ├── toy_0.6b/               # Toy 0.6B experiment batch scripts
│   │   ├── toy_1.7b/               # Toy 1.7B experiment (Qwen3-1.7B)
│   │   └── qwen3_8b_base/          # Q8B-Base diversity benchmark
│   ├── slurm/                    # Slurm job helpers
│   │   ├── auto_retry.sh         #   Failure-mode-aware retry with checkpoint resume
│   │   └── podman_storage.sh     #   Fix podman-hpc storage on unreliable nodes
│   ├── lib/                      # Shared analysis utilities
│   │   └── experiment_discovery.py # Auto-discover experiment arms & trajectories
│   ├── debug/                    # Profiling & competence checks
│   ├── plots/                    # Plotting utilities
│   ├── experiment_status.py      # Show experiment arm status (DONE/RUN/MISS)
│   ├── analyze_trajectories.py   # JSONL trajectory analysis CLI (--diversity, --mixture)
│   ├── analyze_game.py           # Game-specific trajectory analysis
│   ├── generate_report.py        # Automated experiment reporting
│   ├── analyze_within_batch.py   # Within-batch diversity analysis
│   └── parse_game_logs.py        # Convert slurm game logs to trajectory JSONL
│
├── src/unexploitable_search/   # Main Python package
│   ├── config/                 #   Hydra config dataclasses
│   ├── wandb_utils.py          #   W&B helpers (sync, artifact upload)
│   ├── settings/               #   Task domains
│   │   ├── setting.py          #     Base Setting class
│   │   ├── quest.py            #     Base Quest class (+ NoneSideQuest placeholder)
│   │   ├── apps/               #     APPS (code generation, sandboxed exec)
│   │   ├── toy/                #     Toy (5 Python tasks, diversity experiments)
│   │   │   ├── tasks.py        #       Task definitions + test cases
│   │   │   ├── properties.py   #       20 structural/naming property detectors
│   │   │   ├── metrics.py      #       Diversity metrics (unique ASTs, Self-BLEU, etc.)
│   │   │   └── quests/         #       exec()-based evaluation (no sandbox)
│   │   └── alpaca/             #     Alpaca (text generation)
│   ├── train/
│   │   ├── utils.py            #   Shared training utilities
│   │   ├── trainers/           #   Trainer implementations
│   │   │   ├── grpo_trainer.py         # Base GRPO
│   │   │   ├── maxent_trainer.py       # + entropy bonus
│   │   │   ├── kl_from_base_trainer.py # + KL penalty
│   │   │   └── game_trainer/           # Two-player game variants (+ ibr_controller.py)
│   │   ├── callbacks/          #   TrainerCallbacks
│   │   │   ├── trajectory_logger.py    # Unified trajectory logging
│   │   │   ├── best_reward.py          # Save best-reward checkpoint
│   │   │   ├── highest_accuracy.py     # Save highest-accuracy checkpoint
│   │   │   └── clean_sandbox_storage.py
│   │   └── mitigation_strats/  #   Training orchestration
│   │       ├── train.py                # Main entry: train_with_mitigation_strat()
│   │       └── strats/                 # Reward wrappers (length_penalty)
│   ├── eval/                   #   Evaluation pipeline
│   ├── judges/                 #   API-based judges (Claude, GPT)
│   ├── logging/                #   Completion logger + heuristics
│   └── tests/                  #   Test suite
│
├── pyproject.toml
├── setup.sh
└── uv.lock

---

Concepts

Settings

A Setting defines a task domain. Three are available:

The toy setting is a lightweight alternative to APPS for fast iteration and diversity experiments. It defines 5 deterministic Python function tasks (remove duplicates, flatten, char frequency, second largest, run-length encoding), each solvable in 1–6 lines with multiple valid algorithmic approaches. Evaluation uses exec() directly — no sandbox needed. Includes 20 structural/naming properties and diversity metrics (unique ASTs, Self-BLEU, BoN coverage) for measuring output diversity.

Note: side_quest is a vestigial run-name label whose sole valid value is none; it carries no behaviour.

Starting Models

The model_organism config selects how the starting policy is loaded before RL. Three loaders:

Training Strategies

Training applies RL (GRPO) to the starting model, rewarding correct task completion while encouraging a semantically diverse set of solutions. Six strategies:

The game / hard_game strategies are the GAP method that is the focus of this repo; the others are diversity baselines.

Trainers

All trainers extend a custom fork of TRL's GRPO implementation:

• GRPOTrainer — standard Group Relative Policy Optimization
• MaxEntGRPOTrainer — adds an entropy bonus to the reward to encourage diverse outputs
• KLFromBaseGRPOTrainer — adds a KL-divergence penalty from the base (pre-LoRA) model
• GRPOGameTrainer — two-player adversarial game; Alice (model) tries to solve the task while an adversary (Charlie) proposes alternative solutions and properties to catch Alice
• GRPOHardGameTrainer — harder variant with a separate Bob agent that generates properties

Reward Functions

Reward functions are composed at runtime:

1. Main quest reward — always present (e.g. unit-test pass rate for APPS)
2. Length penalty — optional, for the length_penalty strategy
3. Game reward — added automatically by game trainers (Alice-vs-Charlie outcome)

Trajectory Logging

The TrajectoryLoggerCallback captures per-completion data during training:

• Prompts, completions, per-function rewards, advantages
• Heuristic enrichment (syntax validity, tmp-variable usage, code length, etc.)
• Game trace data (for game trainers)
• Strategy metrics (KL, entropy, entropy bonus)

Outputs to wandb Tables, local JSONL files, or both. Configured under train.trajectory_logging in the mitigation config.

---

Usage

1. Train (RL)

uv run scripts/train/train_with_mitigation_strat.py \
  setting=apps \
  side_quest=none \
  model_organism=prompted \
  it_model=qwen3_8b \
  strat=game

Key overrides:

• strat=none / entropy_bonus / kl_from_base_penalty / length_penalty / hard_game — choose the strategy (game/hard_game are GAP; the rest are diversity baselines)
• model_organism=trained model_organism.wandb_artifact_path=<path> — start from a trained checkpoint instead of a fresh LoRA
• seed=42 — set random seed
• strat.trainer_config.max_steps=500 — override training length
• train.trajectory_logging.enabled=true — enable trajectory logging (on by default)
• train.trajectory_logging.backend=local — JSONL only (no wandb tables)

Sweep across strategies:

uv run scripts/train/train_with_mitigation_strat.py --multirun \
  setting=apps \
  side_quest=none \
  model_organism=prompted \
  it_model=qwen3_8b \
  strat=none,entropy_bonus,kl_from_base_penalty,length_penalty,game

2. Evaluate

Benign evaluation (task-correctness on the held-out test split):

uv run scripts/eval/run_benign_eval.py setting=apps

For semantic-diversity evaluation on the toy setting, see Diversity Eval below.

3. Analyze Trajectories

After a training run with trajectory logging enabled:

python scripts/analyze_trajectories.py outputs/.../trajectories/trajectories_*.jsonl

Options:

• --filter 100-300 — restrict to a step range
• --heuristic num_functions — show trend for a specific structural heuristic
• --game — show game-specific metrics (game_reward, alice/charlie flagged rates, property diversity)
• --diversity — compute per-step diversity metrics (unique ASTs, Self-BLEU, BoN coverage; toy setting only)
• --bon-reference path.jsonl — compute BoN coverage against a reference set
• --mixture 0.3 — pool completions from the trailing 30% of training to compute mixture-policy diversity (requires --diversity)

---

Configuration Reference

The project uses Hydra for composable configuration. All configs live under conf/. Values marked ??? are required and must be provided on the command line. Any config value can be overridden from the CLI.

How Hydra Composition Works

Each entry point has a main config that pulls in config groups via a defaults list:

conf/mitigation_strats/config.yaml     <- main config
  defaults:
    - it_model: ???                    <- REQUIRED (8 options)
    - adapter: lora                    <- defaults to LoRA
    - model_organism: ???              <- REQUIRED (3 options)
    - strat: ???                       <- REQUIRED (6 options)
    - setting: ???                     <- REQUIRED (3 options)

Shared config groups live in conf/_shared/ and are available to all entry points via Hydra's searchpath. Strategy configs (conf/mitigation_strats/strat/) nest further, each selecting a trainer_config sub-group.

Override any value on the CLI with dot-notation:

uv run scripts/train/train_with_mitigation_strat.py \
  setting=apps side_quest=none model_organism=prompted \
  it_model=qwen3_8b strat=game \
  strat.trainer_config.max_steps=200 \
  strat.trainer_config.learning_rate=1e-5 \
  train.trajectory_logging.backend=local \
  seed=123

Hydra's --multirun flag submits a grid of jobs to SLURM (configured in conf/_shared/hydra/launcher/slurm.yaml).

Variable Interpolation

Configs use three interpolation types:

---

Top-Level Parameters — Mitigation Training

Source: conf/mitigation_strats/config.yaml

---

GRPO Trainer Config (RL)

Source: conf/mitigation_strats/strat/trainer_config/base_grpo.yaml. All strategy trainer configs inherit from this.

TRL algorithm defaults (not in config files, but active at runtime via TRL's GRPOConfig):

Override any of these on the CLI, e.g. strat.trainer_config.num_generations=16.

Prompt budget: max_length (4096, set at top level) minus max_completion_length (768) = 3328 tokens available for prompts. Note: 21% of APPS samples with 4 references exceed this budget.

LoRA Adapter Config

Source: conf/_shared/adapter/lora.yaml

---

Strategy-Specific Parameters

Each strategy selects a trainer config (inheriting from base_grpo) and may add its own parameters. Source: conf/mitigation_strats/strat/.

strat=none — Vanilla GRPO (baseline)

No additional parameters. Uses trainer_config: vanilla_grpo.

strat=game — Two-Player Game

strat=hard_game — Three-Agent Game

Same parameters as game, including property_checker_model_id (adds a separate Bob agent in Python code; config is identical). Also supports alice_turns_per_cycle and adversary_turns_per_cycle for frozen best-response alternation.

strat=entropy_bonus — Entropy Bonus

strat=kl_from_base_penalty — KL-from-Base Penalty

strat=length_penalty — Length Penalty

---

Sandbox Config

Source: conf/mitigation_strats/config.yaml under train.sandbox

Trajectory Logging Config

Source: conf/mitigation_strats/config.yaml under train.trajectory_logging

---

Evaluation Config

Source: conf/evals/_shared/inspect/inspect.yaml

---

Available Base Models

Configured under conf/_shared/it_model/:

The base model path is used by kl_from_base_penalty strategy to compute KL divergence.

---

SLURM Launcher Config

Source: conf/_shared/hydra/launcher/slurm.yaml. Used when --multirun is passed.

The launcher also sets up cache directories (HF_HOME, TRANSFORMERS_CACHE, WANDB_DIR, UV_CACHE_DIR) under $PROJECT_SHARED/.cache/.

---

Tests

# All non-GPU, non-sandbox tests
uv run pytest src/unexploitable_search/tests/ -v -m "not gpu and not sandbox and not slow"

# GPU tests
uv run pytest src/unexploitable_search/tests/ -v -m gpu

# Sandbox tests (requires podman)
uv run pytest src/unexploitable_search/tests/ -v -m sandbox

Test markers: sandbox (requires podman), gpu (requires GPU), slow (multi-step training).

On the cluster, use srun — see CLAUDE.md for srun patterns.

Test Files

Tests are grouped into 13 categories. The E2E Plan column references the category number for each test.

CPU-only (no markers)

GPU-required (@pytest.mark.gpu)

Sandbox-required (@pytest.mark.sandbox)

Manual scripts (not part of automated suite)

---

Toy Experiments

The toy setting is designed for fast diversity experiments comparing training strategies.

Toy 0.6B Experiment

Compares 7 training arms on Qwen2.5-0.5B-Instruct across 5 simple Python tasks, measuring whether GAP (game-based adversarial training) produces more diverse solutions than entropy/KL/length baselines.

Arms: vanilla GRPO, entropy bonus (0.01/0.05/0.1), length penalty, KL-from-base, GAP (game)

bash scripts/experiments/toy_0.6b/run_all.sh

Toy 1.7B Experiment

Same arms on Qwen3-1.7B. Includes a relaunch variant using Qwen3-8B as the property checker judge.

# Full batch (baseline + 7 arms)
bash scripts/experiments/toy_1.7b/run_all.sh

# GAP with 8B property judge (simultaneous + 3:1 alternation)
bash scripts/experiments/toy_1.7b/relaunch_gap_8b_judge.sh

Diversity Eval

uv run python scripts/eval/run_toy_diversity_eval.py \
  --base_model Qwen/Qwen3-1.7B \
  --adapter_path <output_dir>/final_model \
  --ref_freqs <baseline_dir>/ref_freqs.json \
  --num_completions 100

Outputs (on /projects/):

experiments/toy_0.6b/
├── env.sh              # Shared environment (TRANSFORMERS_CACHE, etc.)
├── ref_freqs.json      # Baseline property frequencies
├── logs/               # Slurm .out/.err per job
└── outputs/
    ├── vanilla_s0/     # Each arm has:
    │   ├── checkpoint-50/  ... checkpoint-500/
    │   ├── best_model/
    │   └── final_model/
    │       ├── adapter_model.safetensors
    │       └── adapter_config.json
    ├── entropy_0.01_s0/
    ├── entropy_0.05_s0/
    ├── entropy_0.1_s0/
    ├── length_s0/
    ├── kl_s0/
    └── gap_s0/         # Also contains charlie/ and property_checker/ subdirs

---

APPS Experiments

Full-scale diversity benchmark on Qwen3-8B-Base with APPS code generation, comparing GAP (game) against the vanilla / entropy / KL / length baselines across seeds.

bash scripts/experiments/qwen3_8b_base/run_diversity_benchmark.sh

Experiment Directory Structure

Experiment outputs follow a naming convention that enables auto-discovery:

experiments/qwen3_8b_base/
├── outputs/
│   ├── game_none_s0/                # {strat}_{side_quest}_s{seed}; side_quest is always `none`
│   │   ├── checkpoint-50/
│   │   ├── checkpoint-100/
│   │   ├── final_model/             # Present when training completes
│   │   └── trajectories/
│   │       ├── trajectories_20260215_1234.jsonl
│   │       └── trajectories_latest.jsonl  -> trajectories_20260215_1234.jsonl
│   ├── game_none_s1/
│   ├── entropy_bonus_none_s0/
│   └── ...
├── manifest.json                    # Launch manifest (job IDs, arms, output dirs)
└── logs/                            # Slurm .out/.err files

• APPS convention: {strat}_{side_quest}_s{seed} — side_quest is now always none (e.g. game_none_s0). The discovery parser still recognises the old side-quest names so historical experiment dirs keep parsing.
• Toy convention: {arm_name}_s{seed} (e.g. gap_8b_eps_sim_s0)
• trajectories_latest.jsonl: Symlink to the most recent trajectory file (auto-created)
• manifest.json: Written by launch scripts, records Slurm job IDs per arm

Experiment Status

Check on all arms in an experiment:

python scripts/experiment_status.py /path/to/experiment
python scripts/experiment_status.py /path/to/experiment --side-quest none
python scripts/experiment_status.py /path/to/experiment --strat game

Shows DONE/RUN/MISS status, checkpoint progress, and trajectory line counts for each arm.

Trajectory Analysis

# Direct file paths
python scripts/analyze_trajectories.py outputs/.../trajectories/*.jsonl
python scripts/analyze_trajectories.py --game --filter 100-300 outputs/.../trajectories/*.jsonl

# Auto-discover from experiment directory
python scripts/analyze_trajectories.py --experiment-dir /path/to/experiment --strat game
python scripts/analyze_trajectories.py --experiment-dir /path/to/experiment --side-quest none --heuristic num_functions

Game Trajectory Analysis

# Direct file paths
python scripts/analyze_game.py outputs/.../trajectories/*.jsonl

# Auto-discover from experiment directory
python scripts/analyze_game.py --experiment-dir /path/to/experiment --strat game --side-quest none

# From slurm log files (when trajectory logging is disabled)
python scripts/parse_game_logs.py <logfile>.out -o traj.jsonl
python scripts/analyze_game.py traj.jsonl

Analysis modes: --summary (default), --branches, --properties, --traces N, --plot, --heuristics, --phase-split.

---

License

This repository is released under the Apache License 2.0. See NOTICE for required attribution.

The trainer files under src/unexploitable_search/train/trainers/ are copies of TRL's GRPOTrainer (Copyright The HuggingFace Team, Apache 2.0), modified for this project. Each carries an Apache header noting the modifications.

Third-party data & models

The Apache 2.0 license covers this repository's code only. Datasets and model weights are downloaded at runtime, are not redistributed here, and remain governed by their own licenses:

Note in particular that the Alpaca data is non-commercial: the Alpaca domain is suitable for academic / non-commercial research only, regardless of this repository's code license. Users are responsible for complying with each asset's license.

Citation

If you use this code, please cite the paper:

@misc{garl_without_demonstrations,
  title  = {GARL without demonstrations},
  author = {Jacob Pfau and Arathi Mani and Alejandro Aristizabal and Stefan Jones and Xanthe Spence and Jazon Szabo},
  year   = {2026},
  note   = {https://github.com/AI-Safety-Institute/unexploitable-search}
}