AdvGLUE++ Dataset Generation

Setup

1. Install conda or miniconda.

2. Optional: Install mamba. Mamba is much faster than Conda at resolving dependencies.

   conda install mamba -n base -c conda-forge
   

3. Create an environment and install the dependencies. If you went with conda, run:

   conda env create --file alpaca.yml
   

   Otherwise, if you went with mamba, run:

   mamba env create --file alpaca.yml
   

4. Activate the virtual environment.

   conda activate alpaca

5. Upgrade pip:

   pip install --upgrade pip

6. Manually install transformers. This needs to be done manually!

   pip install git+https://github.com/huggingface/transformers   
   

Attack

SemAttack

1. Construct contextualized embedding space by running preprocessing.slurm. We follow the process in paper "Visualizing and Measuring the Geometry of BERT" (GitHub) to calculate word embeddings.
2. Data preprocessing. Please use get_FC.py, get_FT.py, and get_FK.py to calculate candidate perturbations generated by different semantic perturbation functions. We include a script (get_perturbation_candidates.slurm) to process for all models.
3. Use attacks/attack.py to perform SemAttack. We include a script attack.slurm to perform attack for all models.

Note: We also provide the processed perturbation candidates. You can download them at https://drive.google.com/file/d/103cRN_15sY246LSnqaG0nD3aO_p9VkaE/view?usp=sharing

OpenAttack

Use atatcks/open_attack.py to perform other word-level attacks.

You may check the code for more details. You may also try different semantic perturbation functions and different attack parameters.