Key features

A comprehensive evaluation framework for Large Language Models (LLMs), providing extensive assessments across three key dimensions: general capabilities, safety, and robustness. The framework includes 14 diverse benchmarks and supports both API-based and local models with distributed evaluation capabilities.

Key Features:

• General Capability Assessment: Integrates Alpaca_eval and Simple_Evals covering reasoning, math, coding, and knowledge testing
• Safety Evaluation: Custom benchmarks for testing model safety (StrongReject, XSTest, WildChat)
• Robustness Testing: Advanced evaluation sets (AdvGLUE, AdvInstruction, DecodingTrust-OOD) for model stability
• Flexible Deployment: Supports both API models (GPT-4, Claude-3) and local models
• Distributed Processing: Optimized with Accelerate for efficient multi-GPU evaluation
• Comprehensive Metrics: Detailed reporting and analysis for each benchmark

Supported Evaluations:

1. General Capabilities

   • Knowledge & Reasoning: MMLU, GPQA, DROP, Alpaca Eval
   • Mathematics: Math, GSM8K, AIME
   • Coding: HumanEval

2. Safety Testing

   • Harmful Content Rejection: StrongReject, WildChat
   • Harmful Content Rejection & Over-refused Testing : XSTest

3. Robustness

   • Adversarial Robustness: AdvGLUE, AdvInstruction
   • Out-of-Distribution: DecodingTrust-OOD (Style)

Benchmark Locations:

1. SIMPLE EVAL
   • code_benchmark: human-eval;
   • math_benchmark: math, gsm8k, aime;
   • general_comprehensive_benchmark: mmlu, gpqa, drop,
2. ALPACA EVAL
   • general_comprehensive_benchmark: alpaca_eval;
3. CUSTOM BENCHMARK
   • safety_benchmark: xstest, strongreject, wildchat;
   • robustness_benchmark: advglue, advinstruction, decodingtrust_ood(style);

RUN

source /data/zhangyichi/fangzhengwei/o1/o1_evals/env.sh

model_name=meta-llama/Llama-3.1-8B-Instruct
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7

# robustness
accelerate launch --num_processes 8 -m evals.custom_benchmark.robustness_benchmark.advglue --model_name $model_name --result_dir results
accelerate launch --num_processes 8 -m evals.custom_benchmark.robustness_benchmark.advinstruction --model_name $model_name --result_dir results
accelerate launch --num_processes 8 -m evals.custom_benchmark.robustness_benchmark.decodingtrust_ood --model_name $model_name --result_dir results

# safety
accelerate launch --num_processes 8 -m evals.custom_benchmark.safety_benchmark.strongreject --model_name $model_name --result_dir results
accelerate launch --num_processes 8 -m evals.custom_benchmark.safety_benchmark.xstest --model_name $model_name --result_dir results
accelerate launch --num_processes 8 -m evals.custom_benchmark.safety_benchmark.wildchat --model_name $model_name --result_dir results

# general - alpaca
accelerate launch --num_processes 8 -m evals.alpaca_eval.src.alpaca_eval.demo_ddp --model_name $model_name --result_dir results

# general - simple_evals
accelerate launch --num_processes 8 -m evals.simple_evals.demo_ddp --model_name $model_name --result_dir results --eval_name mmlu
accelerate launch --num_processes 8 -m evals.simple_evals.demo_ddp --model_name $model_name --result_dir results --eval_name math
accelerate launch --num_processes 8 -m evals.simple_evals.demo_ddp --model_name $model_name --result_dir results --eval_name gpqa
accelerate launch --num_processes 8 -m evals.simple_evals.demo_ddp --model_name $model_name --result_dir results --eval_name drop
accelerate launch --num_processes 8 -m evals.simple_evals.demo_ddp --model_name $model_name --result_dir results --eval_name humaneval
accelerate launch --num_processes 8 -m evals.simple_evals.demo_ddp --model_name $model_name --result_dir results --eval_name gsm8k
accelerate launch --num_processes 8 -m evals.simple_evals.demo_ddp --model_name $model_name --result_dir results --eval_name aime

Model Inference Modes

1. API Model Inference (Single Process with Multi-threading)

For API-based models (e.g., OpenAI API), we recommend using single process with multi-threading to avoid duplicate requests and unnecessary costs.

Key Features:

• Single process to prevent duplicate API calls
• Multi-threading for parallel request handling
• Suitable for API-based models (e.g., GPT-3, GPT-4)

Example Usage:

strongreject.run(
    generate_kwargs={
        "jailbreaks": ["none", "pair", "pap_misrepresentation"],
        "num_proc": 20,  # Number of threads
    },
    batched=False,  # Disable batching for API calls
)

Launch Command:

CUDA_VISIBLE_DEVICES=0 accelerate launch --num_processes 1 -m evals.custom_benchmark.safety_benchmark.strongreject

2. Local DDP Model Inference (Multi-process with Single Threading)

For local models, we use DistributedDataParallel (DDP) to maximize parallel efficiency across multiple GPUs.

Key Features:

• Multi-process parallel execution
• Each process handles a separate data chunk
• Optimized for local GPU models

Implementation Details:

1. Jailbreaks run sequentially in the main process for maximum efficiency
2. batched=True enables LocalDDPModel mode:
   • Chunks data in batch_generate
   • Parallel processing across multiple GPUs
3. Response evaluation:
   • Runs in the main process
   • Uses multi-threading for faster evaluation

Example Usage:

strongreject.run(
    generate_kwargs={
        "jailbreaks": ["pair"],
        "max_batch_size": 32,  # Batch size for each GPU
    },
    batched=True  # Enable batched processing
)

Launch Command:

# Using 8 GPUs
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 accelerate launch --num_processes 8 -m evals.custom_benchmark.safety_benchmark.strongreject

API key

put your API settings into ./env.sh

export OPENAI_API_KEY=[YOUR OPENAI_API_KEY]
export OPENAI_BASE_URL=[YOUR OPENAI_BASE_URL]
export HF_TOKEN=[YOUR HUGGINGFACE_LOGIN_TOKEN]

ENV

pip install openai

pip install anthropic

pip install blobfile

pip install google-api-python-client

# latest version of sk-learn is required
pip install [-U] scikit-learn 

pip install accelerate