audio enabled LLM vulnerabilities through code-switching
Python
0
5 commits
updated Aug 21, 2026
Code for SpeechJBB: Probing Safety Alignment and Comprehension in Large Audio Language Models under Code-Switched Speech (EMNLP Findings 2026).
Dataset: https://huggingface.co/datasets/McGill-NLP/SpeechJBB
Paper: https://arxiv.org/abs/2606.06037
pip install -r requirements.txt
export OPENAI_API_KEY=... # code-switching, pseudo-words, judging
export GOOGLE_API_KEY=... # Gemini inference
SALMoNN additionally needs SALMONN_REPO, SALMONN_CKPT,
SALMONN_BEATS_PATH, and SALMONN_WHISPER_PATH.
| Step | Script |
|---|---|
| Translate JailbreakBench prompts into DE/ES/FR/IT | translate.py |
| Generate EN--X code-switched text | code_switch.py |
| Generate X--Y code-switched text | code_switch_txt.py |
| Insert pseudo-words (10/30/50%) | augment_code_switch.py |
| Synthesize speech with XTTS | audio_xtts.py |
| Run the nine LALMs | query_LLMs.py |
| Judge responses (Refusal / Deflection / Jailbroken) | judge.py |
| Aggregate judged outputs into per-condition rates | analyze_judge_summary.py |
| Analysis | Script |
|---|---|
| Pseudo-word detection and meaning attribution | pseudo_words_meaning.py, judge_pseudo_words.py |
| MGSM, FLEURS, FLEURS-SLU comprehension | judge_comprehension.py, audio_xtts_mgsm.py, audio_gemini_mgsm.py, asr_transcribe_mgsm.py |
| WER / CER for synthesized audio | wer_cer.py |
| Figures | plots.py, plots_gibberish.py, plots_mgsm.py |
@inproceedings{ceccatelli2026speechjbb,
title = {SpeechJBB: Probing Safety Alignment and Comprehension in Large
Audio Language Models under Code-Switched Speech},
author = {Ceccatelli, Virginia and Jeon, Yejin and Adelani, David Ifeoluwa},
booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2026},
year = {2026},
publisher = {Association for Computational Linguistics}
}
audio enabled LLM vulnerabilities through code-switching
Python
0
5 commits
updated Aug 21, 2026
Code for SpeechJBB: Probing Safety Alignment and Comprehension in Large Audio Language Models under Code-Switched Speech (EMNLP Findings 2026).
Dataset: https://huggingface.co/datasets/McGill-NLP/SpeechJBB
Paper: https://arxiv.org/abs/2606.06037
pip install -r requirements.txt
export OPENAI_API_KEY=... # code-switching, pseudo-words, judging
export GOOGLE_API_KEY=... # Gemini inference
SALMoNN additionally needs SALMONN_REPO, SALMONN_CKPT,
SALMONN_BEATS_PATH, and SALMONN_WHISPER_PATH.
| Step | Script |
|---|---|
| Translate JailbreakBench prompts into DE/ES/FR/IT | translate.py |
| Generate EN--X code-switched text | code_switch.py |
| Generate X--Y code-switched text | code_switch_txt.py |
| Insert pseudo-words (10/30/50%) | augment_code_switch.py |
| Synthesize speech with XTTS | audio_xtts.py |
| Run the nine LALMs | query_LLMs.py |
| Judge responses (Refusal / Deflection / Jailbroken) | judge.py |
| Aggregate judged outputs into per-condition rates | analyze_judge_summary.py |
| Analysis | Script |
|---|---|
| Pseudo-word detection and meaning attribution | pseudo_words_meaning.py, judge_pseudo_words.py |
| MGSM, FLEURS, FLEURS-SLU comprehension | judge_comprehension.py, audio_xtts_mgsm.py, audio_gemini_mgsm.py, asr_transcribe_mgsm.py |
| WER / CER for synthesized audio | wer_cer.py |
| Figures | plots.py, plots_gibberish.py, plots_mgsm.py |
@inproceedings{ceccatelli2026speechjbb,
title = {SpeechJBB: Probing Safety Alignment and Comprehension in Large
Audio Language Models under Code-Switched Speech},
author = {Ceccatelli, Virginia and Jeon, Yejin and Adelani, David Ifeoluwa},
booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2026},
year = {2026},
publisher = {Association for Computational Linguistics}
}