CREBench

CREBench is a benchmark for evaluating large language models (LLMs) on cryptographic binary reverse engineering.

• Paper: arXiv:2604.03750
• Code: wangyu-ovo/CREBench
• Project Page: CREBench Homepage

Dataset Description

CREBench measures reverse-engineering performance on cryptographic binaries across four evaluation levels:

The main benchmark corpus contains:

• 48 cryptographic algorithms
• 3 key embedding modes: hardcode_plain, fragmented_build, weak_prng_seeded
• 3 compilation / obfuscation difficulties: O0, O3, constxor
• 432 challenge instances in total (48 × 3 × 3)

Algorithms

3-Way, A5-1, A5-2, AES-128-CBC, ARIA-128-CBC, Anubis-128-CBC, BF-CBC-Official, CAMELLIA-128, CAST5, ChaCha20, Clefia, Crypto-1, DES, DESX, E0, GOST-28147-89, IDEA, KHAZAD-64, Kalyna-128, Kasumi, Kuznyechik-128-ECB, LEA, LOKI97, Lucifer-128-ECB, MAGENTA-128, MARS, MISTY1-64, NOEKEON, RC2-CBC-Official, RC4, RC5-CBC, RC6, SAFER, SC2000, SEED, SHACAL-2, SHARK, SKIPJACK, SM4-CBC-Official, Serpent, Simon, Speck, Square, TEA, Threefish, Unicorn-A, XTEA, XXTEA

Dataset Structure

Each algorithm directory (e.g. CREBench/AES-128-CBC/) contains:

CREBench/<algorithm>/
  config.yaml                          # Challenge metadata and evaluation config
  challenge_src/                       # Source templates and generated key material
    generated/<key_mode>/metadata.json # Ground-truth key, IV, flag, and ciphertext
  public-<key_mode>-<difficulty>/      # Public release artifacts for one instance
    challenge                          # Stripped ELF binary
    address.json                       # Ghidra function address map
    decompile/                         # Ghidra decompiled function dumps
  test_vectors-<key_mode>.json         # Mode-specific verification vectors
  src/                                 # Reference cryptographic implementation
  python_implementation/               # Python reference (when available)

Instance naming convention

Public instance directories follow the pattern:

public-<key_mode>-<difficulty>

Examples:

• public-hardcode_plain-O0
• public-fragmented_build-O3
• public-weak_prng_seeded-constxor

Ground-truth labels

Files such as config.yaml and challenge_src/generated/*/metadata.json contain evaluation labels (keys, IVs, flags, and target ciphertexts). These are intentional ground-truth annotations required for automated scoring in the official evaluation harness.

Usage

Download the full corpus

from huggingface_hub import snapshot_download

local_dir = snapshot_download("Danny-1223/CREBench", repo_type="dataset")
print(local_dir)

Or with the Hugging Face CLI:

hf download Danny-1223/CREBench --repo-type dataset --local-dir ./CREBench-data

Browse the instance index

from datasets import load_dataset

dataset = load_dataset("Danny-1223/CREBench", split="train")
print(dataset[0])

Each row in manifest.jsonl indexes one challenge instance and points to the corresponding binary, decompilation, config, and metadata paths inside the repository.

Run official evaluation

Clone the evaluation code repository and point the runner at the downloaded corpus:

git clone https://github.com/wangyu-ovo/CREBench.git
cd CREBench
pip install -r requirements.txt

# Single challenge
python3 scripts/run_passk_eval.py \
  --model gpt-5.4 \
  --challenge AES-128-CBC \
  --difficulty O0 \
  --key-mode weak_prng_seeded \
  --pass-k 3

# Full benchmark matrix (48 × 3 × 3)
python3 scripts/run_passk_eval.py \
  --model gpt-5.4 \
  --all-c-all \
  --all-key-modes \
  --difficulty ALL \
  --pass-k 3 \
  --eval-mode full \
  --jobs 4

For Docker-based sandbox setup, Ghidra packaging, and provider configuration, see the GitHub README.

Evaluation Levels

The official harness scores each run on four levels:

1. L1 — Algorithm identification: recover the underlying cipher / mode
2. L2 — Key extraction: recover secret key material (and IV when applicable)
3. L3 — Code reimplementation: produce a working wrapper-level reimplementation
4. L4 — Flag recovery: decrypt the target ciphertext and recover the embedded flag

Supported Models (paper experiments)

The paper reports results with:

• gpt-5.4, gpt-5.4-mini, gpt-5.2, o4-mini
• gemini-2.5-pro
• claude-sonnet-4-6
• doubao-seed-1-8-251228
• mimo-v2-pro

Citation

If you use CREBench, please cite:

@article{chen2026crebench,
  title={CREBench: Evaluating Large Language Models in Cryptographic Binary Reverse Engineering},
  author={Chen, Baicheng and Wang, Yu and Zhou, Ziheng and Liu, Xiangru and Li, Juanru and Chen, Yilei and He, Tianxing},
  journal={arXiv preprint arXiv:2604.03750},
  year={2026}
}

License

Please refer to the GitHub repository for the latest license terms governing the benchmark corpus and evaluation code.