This repository provides the code for the paper Understanding Jailbreak Success: A Study of Latent Space Dynamics in Large Language Models.
Conversational Large Language Models are trained to refuse to answer harmful questions. However, emergent jailbreaking techniques can still elicit unsafe outputs, presenting an ongoing challenge for model alignment. To better under- stand how different jailbreak types circumvent safeguards, this paper analyses model activations on different jailbreak inputs. We find that it is possible to extract a jailbreak vector from a single class of jailbreaks that works to mitigate jailbreak effectiveness from other classes. This may in- dicate that different kinds of effective jailbreaks operate via similar internal mechanisms. We in- vestigate a potential common mechanism of harm- fulness feature suppression, and provide evidence for its existence by looking at the harmfulness vector component. These findings offer action- able insights for developing more robust jailbreak countermeasures and lay the groundwork for a deeper, mechanistic understanding of jailbreak dynamics in language models. Disclaimer: This paper includes disturbing language in some examples.
This repository provides the code for the paper Understanding Jailbreak Success: A Study of Latent Space Dynamics in Large Language Models.
Conversational Large Language Models are trained to refuse to answer harmful questions. However, emergent jailbreaking techniques can still elicit unsafe outputs, presenting an ongoing challenge for model alignment. To better under- stand how different jailbreak types circumvent safeguards, this paper analyses model activations on different jailbreak inputs. We find that it is possible to extract a jailbreak vector from a single class of jailbreaks that works to mitigate jailbreak effectiveness from other classes. This may in- dicate that different kinds of effective jailbreaks operate via similar internal mechanisms. We in- vestigate a potential common mechanism of harm- fulness feature suppression, and provide evidence for its existence by looking at the harmfulness vector component. These findings offer action- able insights for developing more robust jailbreak countermeasures and lay the groundwork for a deeper, mechanistic understanding of jailbreak dynamics in language models. Disclaimer: This paper includes disturbing language in some examples.
