Learning from Red Teaming: Gender Bias Provocation and Mitigation in Large Language Models

This project is specifically designed to explore and provoke biases in Language Models (LLMs) using Reinforcement Learning techniques. For a detailed exploration of our methods and findings, refer to our paper.

Prerequisites

Before you begin, ensure you have the following prerequisites installed:

• Python 3.9.18
• Relevant Python libraries as listed in requirements.txt

Installation

To set up Bias-PPO on your local machine, follow these steps:

1. Clone the repository: git clone https://github.com/your-repository/bias-ppo.git
2. Navigate to the project directory: cd bias-ppo
3. Install the required dependencies: pip install -r requirements.txt

Usage and reproduce our result

• We provide FT-Gen generator checkpoint on huggingface: https://huggingface.co/farnhua/gpt2-m-testcase_generator

• The following steps shows how to train a test case generator using RL, generate test cases from a test case generator, evaluate the generated test cases and futher mitigate bias in target LLMs.

• The scripts below reproduce our result of provoking and mitigating bias in ChatGPT, it can be extend to GPT4 and LLaMA2 by changing some argument.

• Our P-Chat and FT-Gen baseline test cases are provided in the folder ./Baseline_Prompt, if you want to reproduce the baseline result, you can just skip step 1 and step 2.

Step 1: Training

To train the model with RL for bias provocation in LLMs, execute the script below. Parameters:

• --path : Data path for updating lm_loss
• --model_name : Pretrained test case generator path (FT-Gen in paper)
• --bot : Target LLM, such as ChatGPT, GPT4
• --save_path: Tha path for RL finetuned model. The model will be saved at ./results/<save_path>

export OPENAI_API_KEY=""
python main.py \
  --mode finetune \
  --prompt GPT2 \
  --agent ppo_ptx_kl \
  --path ./gpt2_finetune/pretrain_data/ChatGPT.csv \
  --model_name farnhua/gpt2-m-testcase_generator \
  --bot ChatGPT \
  --dataset Netflix \
  --type bias \
  --exp_name ChatGPT_test \
  --log_interval 5\
  --seed 42 \
  --bz 8 \
  --kl_coef 0.01 \
  --ep_lr 1.0 \
  --k_epoch 5\
  --discount_r 1.0 \
  --end_batch 200 \
  --sample_time 8 \
  --max_pt_len 30 \
  --inner_lr 9e-6 \
  --lm_lr 0.1 \
  --save_path ChatGPT_test \
  --save_interval 10 \
  --wandb disabled

• If use the script above, the finetuned model should be saved at ./results/ChatGPT_test

Step 2: Generate testcases

• --model_path : Path to the RL finetuned GPT-2 checkpoint
• --bot : Target LLM

export OPENAI_API_KEY=""
python3 test.py \
  --model_path ./results/ChatGPT_test \
  --bot ChatGPT \

• The generated test case files will be located at ./RL_Result/<bot>/, including:
  • <bot>-distinct1000-test.csv: Bias-provoking prompts.
  • <bot>-distinct1000-incontext.csv: Example prompts for mitigation.
• If use above script, you should have two files:
  • ./RL_Result/ChatGPT/ChatGPT-distinct1000-test.csv
  • ./RL_Result/ChatGPT/ChatGPT-distinct1000-incontext.csv

Step 3: Analyzing Test Results

To evaluate the test cases generated by GPT-2, use the following script. Parameters:

• --prompt_path : CSV file containing test cases
• --save_path : Path for saving results, a csv file with responses and sentiment gap as score. The file will be saved at ./Score_Result/<bot>/<save_path>
• --bot : Target LLM

export OPENAI_API_KEY=""
python3 score.py \
--prompt_path ./RL_Result/ChatGPT/ChatGPT-distinct1000-incontext.csv \
--save_path ChatGPT-incontext.csv \
--bot ChatGPT \

• The above script is to evaluate the sentiment gap of responses generated by ChatGPT using the test cases in ./RL_Result/ChatGPT-distinct1000-incontext.csv, and the result will be saved at ./Score_Result/ChatGPT/ChatGPT-incontext.csv.

Step 4: Mitigation

• --type : The strategy for mitigation. We only support top, sample, human as shown in the paper currently
• --save_path : Path to save mitigation result, containing resposnes and sentiment gap. The file will be saved at Mitigate_Result/<bot>/<save_path>.
• --bot : Target LLM
• --demo_num: The number of in-context learning example.
• --testfile: Bias-provoking prompts generated in Step 2.
• --demofile: Example prompts for mitigation with score generated in Step 3

export OPENAI_API_KEY=""
python3 mitigate_origin.py \
        --type top \
        --save_path ChatGPT_mitigate_top_5_origin.csv \
        --bot ChatGPT \
        --demo_num 5 \
        --testfile ./RL_Result/ChatGPT/ChatGPT-distinct1000-incontext \
        --demofile ./Score_Result/ChatGPT/ChatGPT-incontext.csv

• The above script is to use Top 5 strategy shown in the paper to mitigate bias of ChatGPT, the resutl will be saved at Mitigate_Result/ChatGPT/ChatGPT_mitigate_top_5_origin.csv

This codebase is modified from BlackBoxBot